BitTorrent is Hella Awesome
I think it's kind of a shame that torrenting is basically synonymous with piracy and gets a bad rap, because the BitTorrent protocol is so amazing. I feel like understanding how it works makes the piracy make more sense and also is a cool introduction to peer-to-peer systems. And it can also be helpful to know how it's a great option for people who use it for legit purposes.
Before I get into explaining things I should say that I'm not an expert on any of this, but rather a passionate nerd. So I guess take things with a grain of salt. If I got any of this wrong, and actual experts read this, I'd love to get any corrections.
So direct downloading doesn't need an introduction: server has file, you download file, you got the file. This has been around forever, it's how most of the internet works. You direct downloaded this webpage the same way you direct download the videos you watch on whatever platform. However, it wasn't always perfect.
Back in the day internet speeds were slow, there weren't always enough servers to handle all the people downloading, and content delivery networks (CDNs) that served popular files/media from your nearest internet hub were still being developed. Over time, all these problems have been ironed out and today direct downloading is good enough that even most media piracy these days happens through direct downloads. But, in those rougher times "peer-to-peer" or "P2P" file sharing systems were the best option not only for getting free shit, but were just the best option regardless of whether the file was legal or not.
This also might not need an introduction, but just in case an explanation is needed: with P2P file sharing the "server" you download from is just other ordinary people's computers. And you can serve files from your own computer, whatever they are. This P2P craze kicked off with Napster back in 1999, which was not only a great option for downloading free music, but just downloading music off the internet in general. There were basically no legal options for downloading music off the internet then. If you wanted one song legally, you'd have to buy a CD, a whole album with songs you might not want to listen to.
And, importantly, Napster solved the prior music sharing option which was slow direct downloads. If lots of people wanted one song or whatever on one server with a relatively slow connection, it would just take forever to download, and the server owner might not like paying to have people overload their server and decide to turn it off. With a P2P system, anyone could be the server, and if there were enough people (peers) with the file, the download could be fast.
Another important point, Napster also made the process of finding and downloading a file much easier. Before, the process of finding a file and initiating a download was often not as seamless as downloading one program, searching for a file in said program, and then having the file download with the program.
Napster had a problem though, the place to discover files and other people with the files was on a central server, which made it a target for lawsuits from media companies trying to shut down the music piracy that happened. That led to less centralized options like LimeWire (Gnutella) and Kazaa (FastTrack) where the file and peer discovery was also P2P. Your download client found files and people with the files by basically just asking around. Since there was no server to shut down other than the servers which helped you find peers rather than files, these were harder to sue out of existence.
Small tangent, this P2P innovation led to the creation of Skype. Sending everyone's video and voice calls through central servers would have been very expensive back in the day, but Skype used P2P to connect people directly to make those calls. All their servers had to do was help make the initial connection, which was very cheap compared to passing through the call data. But over time, systems with servers which piped the video calls won out, and there's not many P2P voice call options being used today. Even Skype eventually turned theirs off.
So previously I had thought that these P2P systems had a lot of problems which had were solved by BitTorrent, but actually many problems were fixed before torrenting. File downloads resuming after the connection broke was solved. Downloading a file from multiple peers at the same time so the download is faster and redundant was solved. Verifying that you actually got the correct file and not some virus or corrupted file was solved. But still BitTorrent did make a few key innovations.
Before I go on, super important thing to understand: file hashing. This is what you (or your P2P client) use to make sure you have a correct file. The best way to understand hashing is it's a random number generator that spits out a particular "random" number with a particular input. And it's very very hard to trick it into giving a certain output with the wrong input. And a hash is short compared to an entire file. If I send you a file, and you want to verify that I didn't send you malware, and you know from a trusted source the file's hash, you can hash the file yourself and compare it to the trusted source's hash to verify that I did send you the right file.
Okay finally, let's get to BitTorrent. From what I understand there are 3 key innovations it brought back in 2001: 1) it didn't care about file searching, it only cared about peer discovery and downloading, 2) it formally split files into pieces and hashed every single piece individually, and 3) it made P2P downloads as fast as possible by incentivizing clients to upload.
The first point is kind of interesting. Basically, a problem with the prior systems was that the file search was part of the P2P system, and anyone could just post whatever file they wanted. There was trolling, people uploading malware, and anti-piracy groups uploading fake/bad files. BitTorrent did not care about the file discovery, and you found torrents outside the software. Those places outside could also be abused, but that's not BitTorrent's problem.
But the third point is probably why it took off so much. Remember the part where direct downloads were slow because there were a lot of people fighting over one server? This same "tragedy of the commons" problem kind of happened with the earliest P2P systems. Whoever was uploading files from their computer was only doing it out of the kindness of their heart. There was zero incentive to upload. As soon as you got your file, you could stop uploading and go on. And this limited download speeds in a similar way to direct downloads from servers.
What BitTorrent did was clever af: it has a tit-for-tat system. Basically, your client prefers to upload to clients that it downloads from. As soon as a client got one piece (lol) of a file and made sure it was the right piece by hashing, it could immediately start uploading it to other people. Everyone downloading had random pieces of a partially downloaded file and could trade pieces with each other. And your client liked to trade with people who were willing to trade. Another big thing was that the torrent client preferred to download the rarest pieces of a file first, so downloaders weren't fighting over a small amount of peers with the remaining 1% of a file. This was what made the downloads so fast.
When torrenting was new, to download a file, what you did was find a relatively small [filename].torrent file from somewhere. This .torrent contained the metadata of the file that will be downloaded: the file name, file size, the hashes of each of the file's pieces, etc. And also it contained links to "trackers", the place your torrent client goes to find peers to download the file from. Your client would ask the tracker(s) for the peers, and as soon as you got some, you connected with them, and you started the download.
And when you finished the download your client would start "seeding", essentially the term for uploading a finished file or being the initial uploader. And torrent clients would show your "ratio", the amount you uploaded divided by the amount you downloaded. A sort of seeding culture emerged in the torrenting community where it is considered noble to have a ratio greater than 1 on a torrent because you at least contributed back as much as you've taken.
Since torrenting is decentralized, anyone could make a torrent tracker and trackers were cheap to host compared to hosting files, but trackers were still a point of failure that could be targeted by lawyers trying to stop piracy, or could just stop working because the owner wanted to move on. This could essentially kill whatever file a .torrent file was pointing to. But smart people kept innovating, and eventually around 2005, the end game was reached: you neither needed a tracker, nor a .torrent file to be able to download a file and find people who have the file.
There are three parts to this: peer exchange (PeX), local peer discovery (LPD), and the almighty distributed hash table (DHT). The first two are the easiest to explain. PeX is dead simple, you basically ask other people you are downloading and uploading from for the peers they know who also have the file. And those other people could have been people that were discovered through LPD, some other tracker your client doesn't know about, or the almighty DHT.
LPD is literally just asking people on the same internet connection as you (your LAN) if they have the torrent you are looking for, which could find people with a torrent without asking a tracker. LPD also made downloading faster because you could download a file locally without it going over the internet. This is useful in cases where multiple people were torrenting the same file in a dorm or something.
Okay, now we get to the almighty DHT. This thing, this... thing has frustrated me for many years. I knew that it existed and what it did, but I could never find an explanation for how it worked that wasn't overly technical or didn't gloss over things. I had no choice but to ask Gemini for a dumbed down explanation, and wow, it's the greatest thing ever. Why did it take until this for me to get even a rough idea of what is going on under the hood? My information comes mostly from Gemini, but these are my own words and interpretation.
I need to start off with a weird metaphor. Imagine... dragons... no, imagine a big library with around a hundred thousand books, and in that library there are a thousand people. Every book has a unique number, but the library is a mess. The books are not sorted or organized in any way, they're just put on whatever shelf. However, every person in that library knows the exact location of about a hundred different books. And they also know people who might know the location of certain books.
So you want to find book #12345, you go up to one person and ask if they know where the book is. They say they don't know where that book is, but they point you towards someone who knows books that start with a 1 (book numbers #1XXXX). You ask that person, they also don't know, but they point you to someone who knows books that start with 12 (numbers #12XXX). The #12XXX person also doesn't know, and points you to the #123XX person. Finally, book #12345 is among the hundred books that #123XX person knows the location of, and they show you where it is. Even though there are 100000 books, and 1000 people, you only had to talk to 4 people to find the book you are looking for.
Inversely, let's say you want to add a book to the library and it has the number #23456. You put it wherever because the library is like that, and you do the same thing like in the last paragraph and go through 4 people to find the person keeping track of the #234XX books and tell them the location. Then people looking for the book you added can find that person to find the book.
What I didn't mention is that in this metaphor you are also part of the thousand people remembering of the location of certain books. Let's say you know the #345XX books. People come to ask you for the books with those numbers, and tell you the locations of new books with those numbers. Also, you tell a bunch of people (but not everyone) what books you keep track of, and also you remember a bunch of other people's knowledge of book locations. So if someone asks you where a book is and you don't know, you can at least point to someone who might know.
Obviously this would be silly in real life, but I hope this metaphor will be useful for you understand what's really going on. With the actual DHT, there could be billions of files, and millions of people keeping track of files, but everything still scales nicely, and importantly, not linearly. You wouldn't be going though thousands of people asking for the location of a file. It's more like a double-digit quantity with the DHT.
Every torrent has what is called an infohash. This is the hash not of the file the torrent will get you, but rather the hash of the file metadata inside the .torrent file. The infohash is the unique fingerprint of a torrent and is used to indentify it. When your client contacts trackers to look for peers, it's giving the infohash and the tracker gives back a list of peers who also have or want that infohash. Meanwhile the "distributed hash table" is the torrent infohashes distributed around to everyone.
How are the hashes distributed though? When you start a fresh torrent client for the first time, it creates a long random "node number" that will never change with the same length as the length of the infohash numbers (remember, hashes are just "random" numbers). Next, you join the DHT network as a DHT node, your client starts collecting a list of other clients and their node numbers, and you tell others your own node number.
Then, what happens is other clients start contacting yours and asking: "hey, I have this infohash number I want to find peers for, do you know peers who have this torrent or do you know any node who has a node number that is closer to this infohash number?" and your client tells them who it knows is closer, or if it has a list of peers who have that torrent's infohash. And when your client is discovered to be the closest node number to the infohash number, your client becomes the rendezvous point for people looking to seed or download a torrent. Basically it becomes like a mini tracker for a set of infohashes whose number is closest to your node number.
If you have your own infohash you want to find peers for, your client starts traversing the DHT just like the one who was asking yours. It first asks the nodes it knows who have the closest node numbers to the infohash, then those nodes return the closest node numbers they know, then your client asks the new closest nodes it learned of what they know, and this keeps repeating. Your client keeps finding and going down the list of the closest nodes it can find until it reaches a node that has the contact info for the peers that have the infohash. Once your client finds just one peer through the DHT, it can ask for more through PeX, and you can immediately end up with many peers for the torrent.
In the case where you are seeding a torrent, your client won't just tell one DHT node its contact info, it will tell a bunch that are close to the infohash number. Clients pop in and out all the time because of people turning off their computer, and having multiple nodes know your info is the best option for redundancy.
Probably what might make the DHT so effective is not just the whole looking for the closer number thing, but rather how your client picks which nodes to keep track of. Remember the library metaphor and how you slowly got closer to the person who knows a book's location digit-by-digit? Your client traverses the nodes of the DHT in a similar manner, slowly zooming in on who has the closest number. However, it could not do this so quickly if every node just selected a random list of nodes. There is particular math that is being used by every node to collect a list of nodes that are various distances away so anyone traversing the DHT through them can find what they are looking for as quickly as possible.
The nice thing about all this is that finding peers is not an either-or situation. As soon as you start up a new torrent, your client uses all available options to find peers. Trackers are not needed but they are not obsolete. They can return a list of peers in a second, whereas if you are using the DHT, it could take a bit longer to find any peers. But while your torrent can get started quickly with peers that came from a tracker, higher quality peers could be found through the DHT and PeX while it's downloading.
One more DHT thing I should talk about is: how are DHT nodes found? The easiest option is just asking other peers in a torrent you have for their node number and the nodes they know. Another option is your peer cache. Whenever you shut down your torrent client, and then start it up again later, it tries pinging the nodes it knew before to see if they're still around, and usually many are.
But in both cases you need either nodes you find outside the DHT or knew before. What if your client knows zero DHT nodes and can't find any? In that case there are "bootstrap" nodes which are similar to trackers. Bootstrap nodes have a link your client connects to to get a list of other DHT nodes. You could say this is also a point of failure, but if all the bootstrap nodes your torrent client knows about die, and you know the link to at least one working bootstrap node, you can add it to the list and get back into the DHT.
Remember when I mentioned that the .torrent file is also no longer needed? All you need for a torrent is the infohash to start downloading it! When your client finds a peer who has the infohash of your torrent, it asks for the torrent metadata from that peer (the same metadata that is inside the .torrent), hashes it to make sure that it creates the same infohash, and then starts the download like normal. What's so cool about not needing the .torrent file anymore is that all the information can be held inside of a relatively short link called a "magnet". This example won't work if you don't have a torrent client installed, but if you click this magnet link, it'll open up an Ubuntu ISO torrent.
That's kind of most of the details about the interactions between torrent clients I know about. I hope at this point it is clear why torrenting is so popular for piracy. It's literally people sharing files with each other in a way that cannot be disrupted by lawyers. Torrenting is also part of the reason why things like Spotify and Netflix exist, because people liked getting their media off the internet rather than physically. But also, torrenting is literally just file sharing. The files being shared don't need to be pirated.
If you want to share something you made, you can just share it as a torrent, and if it becomes popular you don't need to worry about the hosting because everyone's sharing the file collectively. What's neat is theoretically a torrent can be around forever (great for archiving), and will never be at the whims of a file host, video platform, etc. But also, if no one is seeding the torrent, the torrent is gone. So YouTube can get points for keeping some forgotten video from two decades ago up in this kind of case. But still, in certain cases, particularly cases where you want to share a very large and/or highly-demanded file, making a torrent can actually be a better option than finding a file host that can handle it.
Another neat thing is that if you know of a dead torrent with no seeders but you somehow have the file it shared, you can resurrect a torrent. I don't know how it's done in other clients, but in qbittorrent you just put the file in the download spot for it, start the torrent as you would normally, and it like "downloads" from your computer quickly. Boom, torrent resurrected. Anyone who was waiting for someone who has the file to appear will start downloading right away as well.
There is one more kind of issue with torrenting: your IP address can be found out basically anyone. For many people this is not an issue and they just torrent without worry, but in some countries like America or Germany there are hardcore anti-piracy groups that can figure out who is downloading what and you could face consequences for pirating. This anti-piracy stuff is usually why you might hear "get a VPN" in the context of torrenting. You get a proxy IP address that hides your real IP address so it's harder for the lawyers to find out who is torrenting what.
One of the most noteworthy examples of torrenting being used for legit purposes was Blizzard pushing out World of Warcraft updates over a built-in torrent client inside their updater. Their servers couldn't handle everyone downloading new content updates at the same time, so they used torrenting to solve this.
Similarly Linux distro makers also had issues with everyone rushing to download a new update or release when it came out, and also started to share new releases as torrents. Even today you can still get torrents of the latest Linux distro releases. The fact that this has been a legit use case for almost as long as torrenting has been around is probably why a popular euphemism for pirated files is "Linux ISOs".
There are a few other places that have legit torrents. If you want to download Wikipedia, you can download it as a torrent. OpenStreetMap, an open (like "open source") alternative to Google Maps and other mapping software, releases their huge data sets as torrents. Internet Archive also has much of what they host available as torrents.
I also want to bring up cool functionality that's inside of torrent clients, mainly the one I use: qbittorrent. If you're downloading a video file and want to start watching it before it finishes downloading, there's an option to have the file be downloaded in sequential order. And you can just open the partially finished file in VLC and start watching it. Though you might also need to select the "first and last pieces first" option because some video files have important info at the end of the file.
Another nice thing is the RSS feed thing. Basically, qbit grabs a list of the latest torrents from a site you choose, and just shows you what's available to download. You can open the feed and select what you want to download in the client without opening a site for torrents. And, you can set it up so it automatically downloads certain files that match a certain search term you want. So neat.
I don't need this anymore because I have such a fast internet connection now, but the ability to limit the amount of torrents simultaneously downloading was super helpful when I had a slow internet connection (like barely enough for YouTube 1080p slow). I could set the limit to like 2 torrents at once, and every one that came after was queued to download in the order you started them after one of the active downloads finished. This way I wouldn't need to wait for 10 different torrents to finish when they could and could prioritize what I wanted done first.
A kind of unintuitive thing I've seen is that you can use torrenting to make a video streaming service. I mentioned that you can download files in sequential order, but what if you want to skip ahead to the middle of the video file before it's downloaded? There's a solution for that too. It can't be done with a typical torrent client as far as I know, but there's a way for a piece of software with a video player combined with a torrent library to look at where you want the video to play from, calculate which piece of the torrent file has that part of the video, and then make the torrent library ask peers for that particular part of the file. This seeking is not as instant as it would be on YouTube, but it does work.
One thing I've been wondering about is why isn't there any sort of torrenting but for live streaming. I've vaguely heard something like this existed, but I've never seen it. I wonder why it isn't a thing, because I think it could be just as useful as plain torrents.
That's all I have to say. Hope I've adequately glazed torrenting. And hopefully I didn't miss anything important or got something wrong. Have fun.