问题
The stderr
logs the following while running a map-reduce job:
root@dbpc42:/tmp/mesos/slaves/20141201-225046-698725789-5050-19765-S24/frameworks/20141201-225046-698725789-5050-19765-0016/executors/executor_Task_Tracker_2/runs/latest# ls
hadoop-2.5.0-cdh5.2.0 hadoop-2.5.0-cdh5.2.0.tgz stderr stdout
Contents of stderr
:
WARNING: Logging before InitGoogleLogging() is written to STDERR I1202 19:41:40.323521 7223 fetcher.cpp:76] Fetching URI 'hdfs://dbpc41:9000/hadoop-2.5.0-cdh5.2.0.tgz' I1202 19:41:40.323685 7223 fetcher.cpp:105] Downloading resource from 'hdfs://dbpc41:9000/hadoop-2.5.0-cdh5.2.0.tgz' to '/tmp/mesos/slaves/20141201-225046-698725789-5050-19765-S24/frameworks/20141201-225046-698725789-5050-19765-0016/executors/executor_Task_Tracker_0/runs/59c9a5ac-c376-4e9e-b9b7-63a24ae84dcd/hadoop-2.5.0-cdh5.2.0.tgz'
I1202 19:41:48.779220 7223 fetcher.cpp:64] Extracted resource '/tmp/mesos/slaves/20141201-225046-698725789-5050-19765-S24/frameworks/20141201-225046-698725789-5050-19765-0016/executors/executor_Task_Tracker_0/runs/59c9a5ac-c376-4e9e-b9b7-63a24ae84dcd/hadoop-2.5.0-cdh5.2.0.tgz' into '/tmp/mesos/slaves/20141201-225046-698725789-5050-19765-S24/frameworks/20141201-225046-698725789-5050-19765-0016/executors/executor_Task_Tracker_0/runs/59c9a5ac-c376-4e9e-b9b7-63a24ae84dcd'
sh: 1: cd: can't cd to hadoop-2.5.0-cdh5.2.0
sh: 1: ./bin/hadoop: not found
The stdout
is empty.
The interesting part is, that when I run the slave on the same node where the master runs, the problem does not appear and tasks finish successfully. Only remote slaves introduce this problem. What could be the problem?
回答1:
The script placed by the JobTracker and ran by the executor was not able to access the directory. I've solved the problem by changing permissions recursively for directory /tmp
to "free for all", 777.
回答2:
What you need to do is to give +x access wherever your mesos-work-dir is located. You don't need 777 recursively.
来源:https://stackoverflow.com/questions/27254666/hadoop-2-5-0-on-mesos-0-21-0-with-library-0-0-8-executor-error