Driver class compilation error - hadoop Mapreduce

问题

I have 3 .java file

1) Mapper.java
2) Reducer.java
3) Driver.java

I am trying to compile hadoop mapreduce program at command line using Driver class but it is showing below error

Driver.java:39: error: cannot find symbol
        job.setMapperClass(Mapper.class);
                           ^
  symbol:   class Mapper
  location: class Driver
Driver.java:40: error: cannot find symbol
        job.setReducerClass(Reducer.class);

How can I solve above error.Below is run method in Driver class

public boolean runnerParsing(String inputPath, String outputPath) throws IOException, ClassNotFoundException, InterruptedException {
         Configuration conf = new Configuration();

        Job job = new Job(conf, "Parsing");
        job.setJarByClass(Driver.class);


        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);

        job.setMapperClass(Mapper.class);
        job.setReducerClass(Reducer.class);
        //job.setNumReduceTasks(0);

        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        FileInputFormat.addInputPath(job, new Path(inputPath));
        FileOutputFormat.setOutputPath(job, new Path(outputPath));

        return job.waitForCompletion(true);

     }

回答1:

You will need to compile all java files as below:

javac -classpath /usr/local/hadoop/hadoop-core-1.2.1.jar -d compiled_classes Driver.java Mapper.java Reducer.java

Please note that your classpath value may change slightly depending on how you install Hadoop.

If you need further help, please have a look at this article which might help you: http://www.bigdatatutes.com/getting-started-with-big-data/

回答2:

This is what i think:

You don't have Mapper/Reducer stored in Mapper.java/Reducer.java.
You have mapper/reducer class in another package

Can you add your mapper and reducer class as well to question?

回答3:

In Chapter 3 of “Big Data with Hadoop” byGarry Tukington and Gabriele Modena, there is the example of pattern Top N where, in the driver code of TopTenHashTag class, there is this instruction with a reference to another java class :

job.setJarByClass(HashTagCount.class);

In my PC, both files, HashTagCount.java and TopTenHashTag.java are in: /home/hduser/playground/src

My compilation command is the following (and has worked for me) :

javac -classpath $HADOOP_HOME/share/hadoop/common/lib/activation-1.1.jar:$HADOOP_HOME/share/hadoop/common/hadoop-common-2.7.1.jar:$HADOOP_HOME/share/hadoop/common/lib/:/usr/hadoop/hadoop-2.7.1/share/hadoop/mapreduce/ -d playground/classes7 playground/src/TopTenHashTag.java \ playground/src/HashTagCount.java

This is the command to create the .jar file : jar -cvf playground/TopTenHashTag.jar -C playground/classes7/ .

and finally this is the command to start the mapreduce job :

hadoop jar /home/hduser/playground/TopTenHashTag.jar com.learninghadoop2.mapreduce.TopTenHashTag /user/hduser/inxYZ/ outHashXYZ

来源：https://stackoverflow.com/questions/26570205/driver-class-compilation-error-hadoop-mapreduce

标签

Hadoop

command-line

MapReduce