Building a simple MapReduce project with gradle: Hadoop dependencies don't have Mapper and Reducer

问题

I'm trying to build a simple Hadoop mapreduce program and I chose Java for that job. I checked out the example codes around and tried to build myself. I created the following gradle script and when I looked at the installed dependencies, none had Mapper or Reducer. Not even org.apache.hadoop.mapreduce package.

group 'org.ardilgulez.demoprojects'
version '1.0-SNAPSHOT'

apply plugin: 'java'

repositories {
    mavenCentral()
}

dependencies {
    testCompile group: 'junit', name: 'junit', version: '4.11'
    compile group: 'org.apache.hadoop', name: 'hadoop-common', version:'2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-hdfs', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-yarn-common', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-minicluster', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-core', version:'2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-jobclient', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-app', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-shuffle', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-mapreduce-client-common', version: '2.7.3'
    compile group: 'org.apache.hadoop', name: 'hadoop-client', version: '2.7.3'
}

I know I'm not going to be needing at least 7 out of those 10 hadoop dependencies, but I don't know which of these dependencies have org.apache.hadoop.mapreduce package (I know these 11 don't though).

Which dependency/repository should I add so that I can actually build a mapreduce job?

Can I do that with raw org.apache.hadoop packages and not vendor packages (such as Cloudera)?

Thanks in advance for all the help.

回答1:

This should be the correct dependency:

compile 'org.apache.hadoop:hadoop-mapreduce-client-core:2.7.3'

Be sure to refresh your gradle project.

回答2:

Add

compile group: 'org.apache.hadoop', name:'hadoop-core', version: '2.7.3'

来源：https://stackoverflow.com/questions/40310852/building-a-simple-mapreduce-project-with-gradle-hadoop-dependencies-dont-have

标签

java

Hadoop

gradle

MapReduce