Hadoop jobs fail when submitted by users other than yarn (MRv2) or mapred (MRv1)

Question

I am running a test cluster running MRv1 (CDH5) paired with LocalFileSystem, and the only user I am able to run jobs as is mapred (as mapred is the user starting the jobtrac

Answer 1

You need to be setting up a staging directory for each user in the cluster. This is not as complicated as it sounds.

Check the following properties:

hadoop.tmp.dir
/tmp/hadoop-${user.name}
core-default.xml

This basically setups a tmp directory for each user.

Tie this to your staging directory :

mapreduce.jobtracker.staging.root.dir
${hadoop.tmp.dir}/mapred/staging
mapred-default.xml

Let me know if this works or if it already setup this way.

These properties should be in yarn-site.xml - if i remember correctly.