How to Create and Seed a Torrent Download on Amazon S3
Posted by Carlton Bale on September 16th, 2007
I recently needed to share some open source files via BitTorrent and wanted to host them on my Amazon S3 account. For those of you familiar with S3, here is the short-answer: add ?torrent to the end of the URL of a public-shared file to get the *.torrent file, so the link would be http://s3.amazonaws.com/your_bucket_name/your_file_name?torrent
If you're new to Amazon S3, read on for more info and detailed instructions.
Why would you want to distribute a file using BitTorrent?
I wanted to do this because, when sharing a file via a torrent, you use less of your Amazon S3 bandwidth and lower your monthly expense. To be more accurate, according the S3 documentation:
The difference is that if a lot of clients are requesting the same object simultaneously via BitTorrent, then the amount of data S3 must serve to satisfy those clients will be lower than with client/server delivery.
I knew it was possible to host a torrent on S3, but wasn't quite sure how to do it. After quite a bit of searching and reading, I found that it was actually pretty easy to do.
Here are the details on how to copy a file to your Amazon S3 account, make it public, create the torrent seed / tracker, and have Amazon S3 share both the .torrent file and seed the download.
Preliminary Steps for new Amazon S3 Users:
- You must first have an Amazon S3 account
- Secondly, you need a way to create/manage Amazon S3 buckets, so you'll need to install a client on your PC.
- I recommend using Bucket Explorer, which is a full-featured and easy-to-use client that runs on Windows and Linux; (a Mac version is in private beta and should be available Oct 2007). A free, less-featured alternative is the S3 Organizer add-on for the Mozilla Firefox web browser.
- Install your application of choice and either:
- Open Bucket Explorer -or-
- Open Firefox and go to Tools menu -> S3 organizer, and click the Manage Accounts button
- Enter your AmazonAWS Access Key and Secret Key
- These are available by going to http://aws.amazon.com, mousing-over the "Your Web Services Account" in the upper right-hand corner, and selecting "AWS Access Identifiers"
How to Create and Seed a Torrent Download on Amazon S3:
- Create a new "bucket" (a.k.a. folder) by clicking the "create folder" icon in the right-hand panel of S3 Organizer. Note: you must use a unique bucket name; you won't be able to create bucket if the name is already being used by someone else.
- Upload the file which will be shared into the desired bucket using S3 Organizer
- Make the bucket public read by right-clicking on the bucket name and selecting "Edit ACL". Select "Read" access for "Everyone". Note: you may also have to repeat the read access step for each file that is uploaded to the bucket.
- Now that the file is uploaded and shared, it can be downloaded via standard http from the following location: http://s3.amazonaws.com/your_bucket_name/your_file_name .
- Note 1: You can also find the URL by right-clicking the file in S3 organizer and selecting Copy URL to Clipboard.
- Note 2: Unfortunately, the file will always be available via http download, which can be faster than BitTorrent download, but will use more of your S3 bandwidth.)
- Now you want to create the torrent file itself. Just enter the following URL: http://s3.amazonaws.com/your_bucket_name/your_file_name?torrent and the torrent file will be created and you will be able to download it the *.torrent file to your computer.
- You can share this downloaded *.torrent file with others via e-mail, webpage download, etc.
According to this article, you can use Amazon S3 as the torrent tracker only by following the steps above, opening the torrent file on your local PC, start seeding from your local PC, and then remove the original file. S3 will continue to be the (ultra reliable) tracker for the file but you don't have to use S3 bandwidth to share the file. To save bandwidth usage and prevent someone from going directly to the http download of the file instead of using the torrent download, it's probably best to download the small *.torrent file and share that with others instead publishing the URL to the torrent file on the Amazon S3 server.
Assuming you have the file being shared stored on your local computer, you can download the *.torrent and start seeding from your local PC without re-downloading (the shared file) from S3, further reducing your S3 bandwidth.That's it, my start-to-finish guide on how to take a file and share/seed/tracker it via BitTorrent and Amazon S3. If I missed something or if something isn't clear, let me know in the comments and I'll fix it.
September 24th, 2007 at 12:21 am
Great post. Thank you.
September 30th, 2007 at 10:33 pm
I have a dedicated server with me with 1500GB transfer limit. If I want to seed some files, which one is cheaper, Amazon or my dedicated server. I have more than 1000 GB bandwidth unused every month on my dedicated.
October 1st, 2007 at 7:29 am
Kenney: Amazon S3 costs $0.20 per GB of data transferred. You have a huge amount of bandwidth; that would cost $300/month on S3. Most people don't have anywhere near that; I have 100 GB/month for example. The real advantage of S3 is that it has unlimited storage capacity and bandwidth. An overage would shut-down your hosted web server, but would not shut down S3. The other advantage is that S3 can track and seed a torrent, and most hosted web servers can't. The big advantage of S3 and torrents is that Amazon S3 is a super-reliable tracker, and you can unshare the file and use basically no bandwidth to share/seed the file (once other clients have it downloaded.)
For you, with all of that unused bandwidth, you'd have to have a bunch transfer before S3 would make sense for you. I'm not sure if you could use your provider to track/seed a torrent, but you could definitely use it to host the file.
November 1st, 2007 at 12:16 pm
Wow! Thanks a lot. this is what i am looking for
December 3rd, 2007 at 10:12 am
Hey Carlton,
I followed your directions last night and emailed my friend the torrent file. He downloaded it and opened it up just fine. I'm sharing it locally and its still on the server. But, nothing would download.
Is this service down right now?
Could you help me out?
Thanks… =/
December 3rd, 2007 at 11:35 am
Make sure you have the ACL setup properly. Can you download the file from S3 using a standard HTTP connection? If you un-shared the file on S3 so you could seed only with your local PC, maybe it's not connecting to the tracker on S3 properly.
December 3rd, 2007 at 11:52 am
I can download the file from S3 using a standard HTTP connection.
Can I send you the torrent to see if you can download it? Looking at the details in uTorrent, it seems okay.
Would you like a screenshot?
Thanks for replying so quickly… =)
January 14th, 2008 at 6:07 am
Thanks for this tutorial. I really need to know - is there a way of having your amazon s3 server seed a torrent for mininova/pirate bay? I don't know how to add trackers to a torrent downloaded from the amazon s3 server. When I try to upload the .torrent file from my amazon s3 server to mininova or piratebay it doesn't work. Is there a way to do it somehow?
thanks
January 14th, 2008 at 8:36 pm
ive been looking for something this and im happy i found this site. question though, i can seem to create the torrent file. i keep trying to add the ?torrent at the end of my file and its not working. i enter it into my bittorrent client and it doesnt work, i put the url in my browser and its saying it doesnt exist. can someone help me?
i just need to figure out how to get the torrent made then i can seed it thru bittorrent right?
sory, im a total newbie at this… forgive me
March 1st, 2008 at 11:11 pm
Hey Guys, Perhaps you guys can help. I regularly distribute content for people, like music videos of up and coming bands and singers and also low budget films or films that didnt quite make it to the cinema. I need to help these people get recognised by distributing their content. I already use myspace and youtube but the problem Im having is that is it such a pain to upload torrents to each site. I have maybe about 20 a day and submit to about 15 of the top sites. So you can imagine that its a very time consuming job.
I was wondering if anyone knows of a way I could write all the information 1 time and submit it to all the important sites that way?
Like a bulk torrent uploader or something like that?
Does anyone have any Ideas?
I'd really appreciate it!
Many Thanks Everyone,
Darren Starr
March 2nd, 2008 at 8:17 am
Darren: Instead of uploading the torrent file, could you instead provide a link to the torrent to be downloaded (such as the Amazon S3 link above.) That way, you won't have to upload anything. They download the torrent from S3, their bit torrent client launches, and the download will begin. Would this work or am I missing something?
May 18th, 2008 at 6:46 am
Hi,
I'm interested in this feature of S3. I've been doing some tests. For example, I want to collect stats from the torrent, so I tried editing the .torrent file, adding optional trackers to see if they can collect stats, but it doesn't seem to work.
Is there any method to collect stats from the S3 torrents?
Thanks in advance
May 18th, 2008 at 10:11 am
You can enable bucket logging. See this post: http://www.carltonbale.com/2008/05/9-hidden-features-of-amazon-s3/
May 18th, 2008 at 11:03 am
Oh, I see.
But I think that's not what I'm looking for. I intend to collect the stats through torrent scrape into the DB. Then I'd output the data (seeds/peers, etc) like this:
http://www.hg-subs.com/index.php?option=com_torrenttrader&Itemid=3
I've been able to collect some data though, using the method I wrote above (editing the torren to use one extra tracker). However, my script is not able to collect those stats.
Maybe I should try using another tracker.
Thanks anyway!