CDH4: Version conflict: Found interface org.apache.hadoop.mapreduce.Counter, but class was expected

Question

I'm trying to upgrade from CDH3 to CDH4 and am getting a version conflict from compile to run time. I'm getting this error: Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.Counter, but class was expected

From googling it seems that my code is being compiled against Hadoop 1.x and is running on Hadoop 2.0. I'm compiling and running the app on the same Hadoop client, so it should all be Hadoop 2.0. Here's what I get from running "hadoop version" on the client or any of the other nodes in this test cluster: Hadoop 2.0.0-cdh4.4.0 Subversion file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.4.0/src/hadoop-common-project/hadoop-common -r c0eba6cd38c984557e96a16ccd7356b7de835e79 Compiled by jenkins on Tue Sep 3 19:33:17 PDT 2013 From source with checksum ac7e170aa709b3ace13dc5f775487180 This command was run using /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.4.0.jar

Here's the relevant dependencies and repos from my pom.xml:

<dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>2.0.0-mr1-cdh4.4.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.0.0-mr1-cdh4.4.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.mrunit</groupId>
            <artifactId>mrunit</artifactId>
            <version>0.9.0-incubating</version>
            <classifier>hadoop1</classifier>
 </dependency>
...
 <repository>
            <id>cloudera</id>
            <name>cloudera</name>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
        </repository>

And after compiling my app, this directory exists in my maven repo (which I wouldn't expect and seems like my problem): .m2/repository/org/apache/hadoop/hadoop-core/0.20.2-cdh3u0

But I searched the apps maven dependency:tree and the only reference to hadoop-core is: org.apache.hadoop:hadoop-core:jar:2.0.0-mr1-cdh4.4.0:compile FYI - I've also tried excluding the MRunit dependency and all my test code but same results.

Can anyone advise how I'm getting my version crossed from compile to run time (why is hadoop-core/0.20.2-cdh3u0 getting pulled down when I build the app) ? I believe since I've installed hadoop MRv1, it should be Hadoop 2.0 code at compile and run time ... but I'm still somewhat confused by the versioning in MRv1.

Answer 1

Figured it out - mvn dependency:tree on it's own did not show me the dependency that was calling out for cdh3, but mvn dependency:tree -Dverbose revealed the conflicting dependency. From there I added an exclusion to the pom and now the job runs.