How to pass system property to map function in hadoop

问题

is there a way how to pass system parameter (something like -Dmy_param=XXX) to map function in hadoop map reduce framework. Submission of job to hadoop cluster is done via .setJarByClass(). In mapper I have to create configuration so I would like to make it configerable so I thought that standard way via property file would be ok. Just struggling with passing parameter where the property is set. Another way would be to add property file to submitted jar. Does someone have an experience how solve that?

回答1:

If you haven't already used this in your job, you can try GenericOptionsParser, Tool, and ToolRunner for running Hadoop Job.

Note: the MyDriver extends Configured and implements Tool. And, to run you job use this

hadoop -jar somename.jar MyDriver -D your.property=value arg1 arg2

For more information, check this link.

Here's some sample code I prepared for you:

public class MyDriver extends Configured implements Tool {

  public static class MyDriverMapper extends Mapper<LongWritable, Text, LongWritable, NullWritable> {

    protected void map(LongWritable key, Text value, Context context)
      throws IOException, InterruptedException {
      // In the mapper you can retrieve any configuration you've set
      // while starting the job from the terminal as shown below

      Configuration conf = context.getConfiguration();
      String yourPropertyValue = conf.get("your.property");
    }
  }

  public static class MyDriverReducer extends Reducer<LongWritable, NullWritable, LongWritable, NullWritable> {

    protected void reduce(LongWritable key, Iterable<NullWritable> values, Context context) 
      throws IOException, InterruptedException {
      // --- some code ---
    }
  }

  public static void main(String[] args) throws Exception {
    int exitCode = ToolRunner.run(new MyDriver(), args);
    System.exit(exitCode);
  }

  @Override
  public int run(String[] args) throws Exception {
    Configuration conf = getConf();
    // if you want you can get/set to conf here too.
    // your.property can also be file location and after
    // you retrieve the properties and set them one by one to conf object.

    // --other code--//
    Job job = new Job(conf, "My Sample Job");
    // --- other code ---//
    return (job.waitForCompletion(true) ? 0 : 1);
  }
}

来源：https://stackoverflow.com/questions/17759826/how-to-pass-system-property-to-map-function-in-hadoop