hadoop-streaming

I am trying to execute NLTK in Hadoop environment. Following is the command which i used for execution. bin/hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-1.0.4.jar -input /user/nltk/input/ -output /user/nltk/output1/ -file /home/hduser/softwares/NLTK/unsupervised_sentiment-master.zip -mapper /home/hduser/softwares/NLTK/unsupervised_sentiment-master/sentiment.py unsupervised_sentiment-master.zip --- contains all the dependent files required for sentiment.py I am getting java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2 at org.apache.hadoop

For a python Hadoop streaming job, how do I pass a parameter to, for example, the reducer script so that it behaves different based on the parameter being passed in? I understand that streaming jobs are called in the format of: hadoop jar hadoop-streaming.jar -input -output -mapper mapper.py -reducer reducer.py ... I want to affect reducer.py. The argument to the command line option -reducer can be any command, so you can try: $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \ -input inputDirs \ -output outputDir \ -mapper myMapper.py \ -reducer 'myReducer.py 1 2 3' \ -file

I am trying to read(open) and write files in hdfs inside a python script. But having error. Can someone tell me what is wrong here. Code (full): sample.py #!/usr/bin/python from subprocess import Popen, PIPE print "Before Loop" cat = Popen(["hadoop", "fs", "-cat", "./sample.txt"], stdout=PIPE) print "After Loop 1" put = Popen(["hadoop", "fs", "-put", "-", "./modifiedfile.txt"], stdin=PIPE) print "After Loop 2" for line in cat.stdout: line += "Blah" print line print "Inside Loop" put.stdin.write(line) cat.stdout.close() cat.wait() put.stdin.close() put.wait() When I execute : hadoop jar /usr

I am trying to run python script on Hadoop cluster using Hadoop Streaming for sentiment analysis. The Same script I am running on Local machine which is running Properly and giving output. to run on local machine I use this command. $ cat /home/MB/analytics/Data/input/* | ./new_mapper.py and to run on hadoop cluster I use below command $ hadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.5.0-mr1-cdh5.2.0.jar -mapper "python $PWD/new_mapper.py" -reducer "$PWD/new_reducer.py" -input /user/hduser/Test_04012015_Data/input/* -output /user/hduser/python-mr/out-mr-out The

This is what confuses me, when I use wordcount example, I keep code at master and let him do things with slaves and it runs fine But when I am running my code, it starts to fail on slaves giving weird errors like Traceback (most recent call last): File "/app/hadoop/tmp/mapred/local/taskTracker/hduser/jobcache/job_201110250901_0005/attempt_201110250901_0005_m_000001_1/work/./mapper.py", line 55, in <module> from src.utilities import utilities ImportError: No module named src.utilities java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop