Problem with copying local data onto HDFS on a Hadoop cluster using Amazon EC2/ S3

Question

I have setup a Hadoop cluster containing 5 nodes on Amazon EC2. Now, when i login into the Master node and submit the following command

bin/hadoop jar 

        

Answer 1

Try using Amazon Elastic MapReduce. It removes the need for configuring the hadoop nodes, and you can just access objects in your s3 account in the way you expect.

Answer 2

Use

-Dfs.s3n.awsAccessKeyId=<your-key> -Dfs.s3n.awsSecretAccessKey=<your-secret-key>

e.g.

hadoop distcp -Dfs.s3n.awsAccessKeyId=<your-key> -Dfs.s3n.awsSecretAccessKey=<your-secret-key> -<subsubcommand> <args>

or

hadoop fs -Dfs.s3n.awsAccessKeyId=<your-key> -Dfs.s3n.awsSecretAccessKey=<your-secret-key> -<subsubcommand> <args>

Answer 3

You probably want to use s3n:// urls, not s3:// urls. s3n:// means "A regular file, readable from the outside world, at this S3 url". s3:// refers to an HDFS file system mapped into an S3 bucket.

To avoid the URL escaping issue for the access key (and to make life much easier), put them into the /etc/hadoop/conf/core-site.xml file:

<property>
  <name>fs.s3.awsAccessKeyId</name>
  <value>0123458712355</value>
</property>
<property>
  <name>fs.s3.awsSecretAccessKey</name>
  <value>hi/momasgasfglskfghaslkfjg</value>
</property>
<property>
  <name>fs.s3n.awsAccessKeyId</name>
  <value>0123458712355</value>
</property>
<property>
  <name>fs.s3n.awsSecretAccessKey</name>
  <value>hi/momasgasfglskfghaslkfjg</value>
</property>

There was at one point an outstanding issue with secret keys that had a slash -- the URL was decoded in some contexts but not in others. I don't know if it's been fixed, but I do know that with the keys in the .conf this goes away.

Other quickies:

• You can most quickly debug your problem using the hadoop filesystem commands, which work just fine on s3n:// (and s3://) urls. Try hadoop fs -cp s3n://myhappybucket/ or hadoop fs -cp s3n://myhappybucket/happyfile.txt /tmp/dest1 and even hadoop fs -cp /tmp/some_hdfs_file s3n://myhappybucket/will_be_put_into_s3
• The distcp command runs a mapper-only command to copy a tree from there to here. Use it if you want to copy a very large number of files to the HDFS. (For everyday use, hadoop fs -cp src dest works just fine).
• You don't have to move the data to the HDFS if you don't want. You can pull all the source data straight from s3, do all further manipulations targeting either the HDFS or S3 as you see fit.
• Hadoop can become confused if there is a file s3n://myhappybucket/foo/bar and a "directory" (many files with keys s3n://myhappybucket/foo/bar/something). Some old versions of the s3sync command would leave just such 38-byte turds in the S3 tree.
• If you start seeing SocketTimeoutException's, apply the patch for HADOOP-6254. We were, and we did, and they went away.

Answer 4

You can also you Apache Whirr for this workflow. Check the Quick Start Guide and the 5 minutes guide for more info.

Disclaimer: I'm one of the committers.