Hadoop Streaming - Unable to find file error

问题

I am trying to run a hadoop-streaming python job.

bin/hadoop jar contrib/streaming/hadoop-0.20.1-streaming.jar 
-D stream.non.zero.exit.is.failure=true 
-input /ixml 
-output /oxml 
-mapper scripts/mapper.py 
-file scripts/mapper.py 
-inputreader "StreamXmlRecordReader,begin=channel,end=/channel" 
-jobconf mapred.reduce.tasks=0

I made sure mapper.py has all the permissions. It errors out saying

Caused by: java.io.IOException: Cannot run program "mapper.py":     
error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
... 19 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.(UNIXProcess.java:53)
    at java.lang.ProcessImpl.start(ProcessImpl.java:91)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)

I tried copying mapper.py to hdfs and give the same hdfs://localhost/mapper.py link, that does not work too! Any thoughts on how to fix this bug?.

回答1:

Looking at the example on the HadoopStreaming wiki page, it seems that you should change

-mapper scripts/mapper.py 
-file scripts/mapper.py

-mapper mapper.py 
-file scripts/mapper.py

since "shipped files go to the working directory". You might also need to specify the python interpreter directly:

-mapper /path/to/python mapper.py 
-file scripts/mapper.py

回答2:

Your problem most likely is that python executable does not exist on the slaves (where TaskTracker is running). Java will give the same error message.

Install it everywhere where it's used. Un your file you can use shebang as you probably already do:

#!/usr/bin/python -O
rest
of
the
code

Make sure that the path after the shebang is the same where python is installed on the TaskTrackers.

回答3:

One other sneaky thing can cause this. If your line-endings on the script are DOS-style, then your first line (the "shebang line") may look like this to the naked eye:

#!/usr/bin/python

...my code here...

but its bytes look like this to the kernel when it tries to execute your script:

% od -a myScript.py
0000000   #   !   /   u   s   r   /   b   i   n   /   p   y   t   h   o
0000020   n  cr  nl  cr  nl   .   .   .   m   y  sp   c   o   d   e  sp
0000040   h   e   r   e   .   .   .  cr  nl

It's looking for an executable called "/usr/bin/python\r", which it can't find, so it dies with "No such file or directory".

This bit me today, again, so I had to write it down somewhere on SO.

回答4:

I ran into the exact same issue on a CDH4 Hadoop cluster trying to run a streaming python job. The trick is to add in your mapper / reducer file as the first lines:

import sys
sys.path.append('.')

This will make python look in the current working directory and it should then be able to run, also make sure that your shebang is correct.

回答5:

I have faced same issue while running map reduce with python code. Solution is: We have to specify "-file" as well in front of mapper and reducer.

Here is the command:

hadoop jar /opt/cloudera/parcels/CDH-5.12.2-1.cdh5.12.2.p0.4/lib/hadoop-mapreduce/hadoop-streaming-2.6.0-cdh5.12.2.jar **-file /home/mapper.py** -mapper /home/mapper.py   **-file /home/reducer.py** -reducer /home/reducer.py  -input /system/mainstream/tmp/file.txt -output /system/mainstream/tmp/output

回答6:

Does your mapper.py have execute permission on it ? If not then you need it.

chmod a+x scripts/mapper.py

Hadoop forks and runs the the script before it writes/reads to std so you need to give it execute permission to run.

回答7:

I just received the same error when my mapper returns a null or empty string. So I had to do a check for the value:

try:
    # Skip over any errors

    word = words[18].strip()

        if (len(word) == 0):
            word = "UKNOWN"

    print '%s\t%s' % (word, 1)

except Value:
    pass

回答8:

File not found error sometimes does not means "File not found", instead it means "Cannot execute this script".

Knowing this I solved problems like this, when you are facing with issues ( no java ) on streaming I suggest you to follow this check list:

Does the scripts run? Don't start is using the interpreter i.e. python myScript.py make it executable at start it as ./myScript.py this is the way the streaming will call your script.
use -verbose to see what is going into the jar which will be deployed into the container, sometime this help.
Inside the containers scripts are symlink not real files.
Files which are moved using -file are not in folders. -mapper folder/script.py or -reducer folder/script.py are treat as script.py
Containers and anything inside them are deleted after the job completes, if you want to see what is happening into a container move it into HDFS, I.E: replacing the mapper or the reducer with a .sh script which does the work.

This checklist helped me a lot, I hope can be useful also for you.

Here follows the classic log with the ambiguous error message.

It's true, it cannot run the program.

Caused by: java.io.IOException: Cannot run program "/hadoop/yarn/local/usercache/root/appcache/application_1475243242823_0007/container_1475243242823_0007_01_000004/./reducer.py": 
error=2, No such file or directory

It's the reason the lie.

    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)
    ... 15 more

Read this:

Caused by: java.io.IOException: error=2, No such file or directory

It's a lie, file does exists if -verbose shows it into the packaging list.

    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
    at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)

来源：https://stackoverflow.com/questions/4339788/hadoop-streaming-unable-to-find-file-error

标签

python

streaming

Hadoop

MapReduce