s3cmd failed too many times

Question

I used to be a happy s3cmd user. However recently when I try to transfer a large zip file (~7Gig) to Amazon S3, I am getting this error:

$> s3cmd put thef         


        

Answer 1

I had the same problem with ubuntu s3cmd.

s3cmd --guess-mime-type --acl-public put test.zip s3://www.jaumebarcelo.info/teaching/lxs/test.zip
test.zip -> s3://www.jaumebarcelo.info/teaching/lxs/test.zip  [1 of 1]
 13037568 of 14456364    90% in  730s    17.44 kB/s  failed
WARNING: Upload failed: /teaching/lxs/test.zip (timed out)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
test.zip -> s3://www.jaumebarcelo.info/teaching/lxs/test.zip  [1 of 1]
  2916352 of 14456364    20% in  182s    15.64 kB/s  failed
WARNING: Upload failed: /teaching/lxs/test.zip (timed out)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...

The solution was to update s3cmd with the instructions from s3tools.org:

Debian & Ubuntu

Our DEB repository has been carefully created in the most compatible way – it should work for Debian 5 (Lenny), Debian 6 (Squeeze), Ubuntu 10.04 LTS (Lucid Lynx) and for all newer and possibly for some older Ubuntu releases. Follow these steps from the command line:

• Import S3tools signing key:

  wget -O- -q http://s3tools.org/repo/deb-all/stable/s3tools.key | sudo apt-key add -

• Add the repo to sources.list:

  sudo wget -O/etc/apt/sources.list.d/s3tools.list http://s3tools.org/repo/deb-all/stable/s3tools.list

• Refresh package cache and install the newest s3cmd:

  sudo apt-get update && sudo apt-get install s3cmd

Answer 2

For me, the following worked:

In .s3cfg, I changed the host_bucket

host_bucket = %(bucket)s.s3-external-3.amazonaws.com

Answer 3

On my case, I've fixed this just adding right permissions.

Bucket > Properties > Permissions 
"Authenticated Users"
- List
- Upload/Delete
- Edit Permissions

Answer 4

And now in 2014, the aws cli has the ability to upload big files in lieu of s3cmd.

http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html has install / configure instructions, or often:

$ wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
$ unzip awscli-bundle.zip
$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
$ aws configure

followed by

$ aws s3 cp local_file.tgz s3://thereoncewasans3bucket

will get you satisfactory results.

Answer 5

s3cmd 1.0.0 does not support multi-part yet. I tried 1.1.0-beta and it works just fine. You can read about the new features here: http://s3tools.org/s3cmd-110b2-released

Answer 6

Search for .s3cfg file, generally in your Home Folder.

If you have it, you got the villain. Changing the following two parameters should help you.

socket_timeout = 1000
multipart_chunk_size_mb = 15