Unnecessary recompilations by SBT

Question

I have a question about seemingly unnecessary recompilations by SBT. I have the following scenario: I’m running SBT inside a docker container, by attaching the volume with m

Answer 1

Okay, after a day of debugging I've found the way around this problem.

SBT invalidates the compiled classes mainly based on the following rules:

• Canonical path
• Last modification date

That is, paths and modification dates have to be exactly same for

• Source code files
• Ivy dependencies jars
• JRE's jars

First 2 points are quite easy to achieve, because it's just mapping of docker volumes. The crucial thing is to map to exactly same path as on the host machine. For example, if you work on OS X as I do, path your project sources probably looks like this: /Users/<username>/projects/bla, so in your docker run command you have to do something like:

docker run ... -v /Users/<username>/projects/bla:/Users/<username>/projects/bla ...

You don't care about timestamps for sources and ivy jars, because they will be exactly same (it's the same files).

Where you have to care about timestamps is the JRE stuff. I build the docker image with JRE baked in (using sbt-docker plugin), so I ended up reading modification date of local JRE libs and setting same dates inside the image:

new mutable.Dockerfile {
  ...
  val hostJreTimestamp = new Date(new File(javaHome + "/jre/lib/rt.jar").lastModified()).toString
  val hostJceTimestamp = new Date(new File(javaHome + "/jre/lib/jce.jar").lastModified()).toString
  runRaw(s"""touch -d "$hostJreTimestamp" $javaHome/jre/lib/rt.jar""")
  runRaw(s"""touch -d "$hostJceTimestamp" $javaHome/jre/lib/jce.jar""")
  ...
}

And, of course, JRE should also be installed to exactly same path as on the host, which might be problematic if you used to install Java from RPM, for example. I ended up downloading server JRE (which is distributed as .tar.gz) and extracting it to the right path manually.

So, long story short, it worked in the end. No recompilation, no long waiting time. I was able to find the relevant information from 2 main sources: SBT source code, particularly this function: https://github.com/sbt/sbt/blob/0.13/compile/inc/src/main/scala/sbt/inc/IncrementalCommon.scala#L271, and enabling SBT debug output in build.sbt:

logLevel := Level.Debug
incOptions ~= { _.copy(apiDebug = true, relationsDebug = true) }

(prepare for a lot of output)

Answer 2

This is just a guess. As rumoku suggest it may be an issue with repositories but I think it's related with SBT itself. From SBT's point of view you are running two different machines and it must compile all when it thinks the files have changed.

I don't know how SBT or the compiler identifies the versions of Scala and Java but it may be the case that even though you have the exact same versions of Java and Scala in both environments SBT thinks they are different. Which makes sense as the are different OS.