elastic-map-reduce

i'm trying to learn how to run a java Map/Reduce (M/R) job on amazon's EMR. the documentation that i am following is here http://aws.amazon.com/articles/3938 . i am on a windows 7 computer. when i try to run this command, i am shown the help information. ./elasticmapreduce-client.rb RunJobFlow streaming_jobflow.json of course, since i am on a windows machine, i actually type in this command. i am not sure why, but for this particular command, there was not a windows version (all commands where shown in pairs, one for *nix and one for windows). ruby elastic-mapreduce RunJobFlow my_job.json my

I am trying to copy files from s3 to hdfs using workflow in EMR and when I run the below command the jobflow successfully starts but gives me an error when it tries to copy the file to HDFS .Do i need to set any input file permissions ? Command: ./elastic-mapreduce --jobflow j-35D6JOYEDCELA --jar s3://us-east-1.elasticmapreduce/libs/s3distcp/1.latest/s3distcp.jar --args '--src,s3://odsh/input/,--dest,hdfs:///Users Output Task TASKID="task_201301310606_0001_r_000000" TASK_TYPE="REDUCE" TASK_STATUS="FAILED" FINISH_TIME="1359612576612" ERROR="java.lang.RuntimeException: Reducer task failed to

There must be a way to change the ports 50070 and 50030 so that the following urls display the clustr statuses on the ports i pick NameNode - http://localhost:50070/ JobTracker - http://localhost:50030/ Define your choice of ports by setting properties dfs.http.address for Namenode and mapred.job.tracker.http.address for Jobtracker in conf/core-site.xml: <configuration> <property> <name>dfs.http.address</name> <value>50070</value> </property> <property> <name>mapred.job.tracker.http.address</name> <value>50030</value> </property> </configuration> This question is old but probably worth

I am getting an error "No space left on device" when I am running my Amazon EMR jobs using m1.large as the instance type for the hadoop instances to be created by the jobflow. The job generates approx. 10 GB of data at max and since the capacity of a m1.large instance is supposed to be 420GB*2 (according to: EC2 instance types ). I am confused how just 10GB of data could lead to a "disk space full" kind of a message. I am aware of the possibility that this kind of an error can also be generated if we have completely exhausted the total number of inodes allowed on the filesystem but that is

I have my mapper and reducers as follows. But I am getting some kind of strange exception. I can't figure out why is it throwing such kind of exception. public static class MyMapper implements Mapper<LongWritable, Text, Text, Info> { @Override public void map(LongWritable key, Text value, OutputCollector<Text, Info> output, Reporter reporter) throws IOException { Text text = new Text("someText") //process output.collect(text, infoObjeject); } } public static class MyReducer implements Reducer<Text, Info, Text, Text> { @Override public void reduce(Text key, Iterator<Info> values,

I have tried the following combinations of bootstrap actions to increase the heap size of my job but none of them seem to work: --mapred-key-value mapred.child.java.opts=-Xmx1024m --mapred-key-value mapred.child.ulimit=unlimited --mapred-key-value mapred.map.child.java.opts=-Xmx1024m --mapred-key-value mapred.map.child.ulimit=unlimited -m mapred.map.child.java.opts=-Xmx1024m -m mapred.map.child.ulimit=unlimited -m mapred.child.java.opts=-Xmx1024m -m mapred.child.ulimit=unlimited What is the right syntax? You have two options to achieve this: Custom JVM Settings In order to apply custom