How to chain mapper/reducer in Hadoop 1.0.4?

问题

I was using "new" API of Hadoop 1.0.4 (classes in package org.apache.hadoop.mapreduce). When I wanted to chain mapper/reducer, I found out that ChainMapper, ChainReducer are written for the "old" API (classes in package org.apache.hadoop.mapred). What should I do?

回答1:

I was also searching for the same. I did get the answer and even though its late I thought sharing this may help someone.

From Hadoop 2.0 onwards you can find ChainMapper and ChainReducer in the package org.apache.hadoop.mapreduce.lib.chain

ChainMapper usage pattern:

...
Job job = new Job(conf, "MyJob");

Configuration map1Conf = new Configuration(false); 
... ChainMapper.addMapper(job, AMap.class, LongWritable.class, Text.class, Text.class, Text.class, true, map1Conf);

Configuration map2Conf = new Configuration(false); 
... ChainMapper.addMapper(job, BMap.class, Text.class, Text.class, LongWritable.class, Text.class, false, map2Conf);

Configuration map3Conf = new Configuration(false); 
... ChainReducer.setReducer(job, CReducer.class, Text.class, Text.class, LongWritable.class, Text.class, false, map3Conf);
...

job.waitForComplettion(true);
... 

回答2:

Please read this post. This shows how to make use of two JobConf to enable Chaining of Map Reduce Jobs instead of using ChainMapper/ChainReducer.

来源：https://stackoverflow.com/questions/13499107/how-to-chain-mapper-reducer-in-hadoop-1-0-4