How to build Spark Mllib submodule individually

Question

I modified the mllib in Spark and want to use the customized mllib jar in other projects. It works when I build spark using:

build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

learned from Spark's document at http://spark.apache.org/docs/latest/building-spark.html#building-submodules-individually. But building the whole Spark package took quite long (about 7 minutes on my desktop) so I would like to just build the mllib alone. The instruction for building a submodule in Spark is also from the same link and I used:

build/mvn -pl :spark-mllib_2.10 clean install

to just build Mllib itself. It built successfully, however, I couldn't see the changes I made in the mllib when running other projects that use mllib. While this did work when I build the whole Spark from scratch, I am wondering how should I use maven in order to build the mllib individually?

Answer 1

I was suspecting that the compiled mllib jar is not really used when running the application. So I print out the location of the modified class when running the application by adding this line in the code:

logInfo(getClass.getProtectionDomain.getCodeSource.getLocation.getPath)

And it turned out that Spark is using the spark-assembly-1.6.0-hadoop2.4.0.jar, and it is still using the old mllib jar. So I instead compiled both mllib and assembly by using:

build/mvn -DskipTests -pl :spark-mllib_2.10 install
build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests -pl :spark-assembly_2.10 install

This reduced the whole compiling time to a little over 1 minutes on my machine. There must be better method to do incremental compiling than this that takes much shorter, I am still looking for such a solution. But at the moment, I will use this hot fix.