Set hadoop system user for client embedded in Java webapp

问题

I would like to submit MapReduce jobs from a java web application to a remote Hadoop cluster but am unable to specify which user the job should be submitted for. I would like to configure and use a system user which should be used for all MapReduce jobs.

Currently I am unable to specify any user and no matter what the hadoop job runs under the username of the currently logged in user of the client system. This causes an error with the message

Permission denied: user=alice, access=WRITE, inode=\"staging\":hduser:supergroup:rwxr-xr-x

... where \"alice\" is the local, logged in user on the client machine.

I have tried

various combinations of creating UserGroupInformation instances (both proxies and normal user) and
setting the Java System property with -Duser.name=hduser, changing the USER envar and as a hard coded System.setProperty(\"user.name\", \"hduser\") call.

... to no avail. Regarding 1) I admit to having no clue on how these classes are supposed to be used. Also please note that changing the Java System property is obviously not a real solution for use in the web application.

Does any body know how you specify which user Hadoop uses to connect to a remote system?

PS/ Hadoop is using the default configuration meaning that no authentication is used when connecting to the cluster and that Kerberos is not used in communicating with the remote machines.

回答1:

Finally I stumbled on the constant

static final String HADOOP_USER_NAME = "HADOOP_USER_NAME";`

in the UserGroupInformation class.

Setting this either as an environment variable, as a Java system property on startup (using -D) or programmatically with System.setProperty("HADOOP_USER_NAME", "hduser"); makes Hadoop use whatever username you want for connecting to the remote Hadoop cluster.

回答2:

The code below works for me the same as

System.setProperty("HADOOP_USER_NAME", "hduser")

UserGroupInformation ugi = UserGroupInformation.createRemoteUser("hduser"); 
ugi.doAs(new PrivilegedExceptionAction<Void>() {
    public Void run() throws Exception {
        Configuration configuration = new Configuration(); 
        configuration.set("hadoop.job.ugi", "hduser");
        int res = ToolRunner.run(configuration, new YourTool(), args);
        return null; 
    }
});

回答3:

I am able to resolve similar issue by using secure impersonation feature http://hadoop.apache.org/docs/stable1/Secure_Impersonation.html

following is code snippet

UserGroupInformation ugi = UserGroupInformation.createProxyUser("hduser", UserGroupInformation.getLoginUser()); 

ugi.doAs(new PrivilegedExceptionAction() { 
public Void run() throws Exception { 
  Configuration jobconf = new Configuration(); 
  jobconf.set("fs.default.name", "hdfs://server:hdfsport"); 
  jobconf.set("hadoop.job.ugi", "hduser"); 
  jobconf.set("mapred.job.tracker", "server:jobtracker port"); 
  String[] args = new String[] { "data/input", "data/output" }; 
  ToolRunner.run(jobconf, WordCount.class.newInstance(), args); 
  return null; 
} });

The remote (windows desktop host in my case) login user id should be added in core-site.xml as mentioned in above mentioned URL

来源：https://stackoverflow.com/questions/11041253/set-hadoop-system-user-for-client-embedded-in-java-webapp

标签

Hadoop

cluster-computing