Provided dependency failing to provide

问题

I got a Scalding (a hadoop abstraction layer in Scala) project that I'm trying to build using Gradle.

Looks like Hadoop became a provided dependency in latest releases and it requires some workaround.

So I patched my build.gradle script like so:

apply plugin: 'scala'
apply plugin: 'idea'
configurations {
    provided
}
sourceSets {
    main { compileClasspath += configurations.provided }
}
repositories {
  mavenLocal()
  mavenCentral()
  maven{
    url 'http://conjars.org/repo/'
    }
}
ext.cascadingVersion = '2.1.6'
ext.hadoopVersion = '1.1.2'
dependencies {
  compile 'org.scala-lang:scala-compiler:2.9.2'
  compile 'org.scala-lang:scala-library:2.9.2'
  compile 'bixo:bixo-core:0.9.1'
  compile 'org.testng:testng:6.8.7'
  testCompile  'org.scala-tools.testing:specs:1.6.2.2_1.5.0'
  compile( 'com.twitter:scalding_2.9.2:0.8.1' )
  compile( group: 'cascading', name: 'cascading-core', version: cascadingVersion )
  compile( group: 'cascading', name: 'cascading-hadoop', version: cascadingVersion )
  provided "org.apache.hadoop:hadoop-client:${hadoopVersion}"
}
jar {
  description = "Assembles a Hadoop-ready JAR file"
  doFirst {
    into( 'lib' ) {
      from configurations.compile
    }
  }
  manifest {
    attributes( "Main-Class": "com.Crawler" )
  }
}

Which I thought would solve the problem. But I keep getting the following error at when trying to build:

[ant:scalac] Element '/Users/tkmafj4/Projects/AIT/Crawler/build/resources/main' does not exist.
[ant:scalac] scala.tools.nsc.symtab.Types$TypeError: class file needed by Source is missing.
[ant:scalac] reference value hadoop of package org.apache refers to nonexisting symbol.

Which looks a lot like there's something missing in my configuration.

How can I check that sources are being fetched?

What is the proper workaround to get this to compile?

回答1:

hadoop becoming a "provided" dependency of cascading means that depending on cascading will no longer pull in hadoop, because hadoop is meant to be provided by the target environment or by whoever creates the ultimate deployable archive. If the hadoop dependency needs to go into your fat Jar, you need to make it (at least) a runtime dependency. But since there appears to be some problem with compilation, I'd try to make it a compile dependency.

If the hadoop dependency should go on the compile class path but not into the fat Jar, you'll need to add something like sourceSets.main.compileClasspath += configurations.provided.

PS: Your fat Jar task needs to package configurations.runtime rather than configurations.compile, and the doFirst { line (as well as the corresponding closing brace) should be removed.

来源：https://stackoverflow.com/questions/23018342/provided-dependency-failing-to-provide

标签

scala

Hadoop

gradle

dependency-management