How to work efficiently with SBT, Spark and “provided” dependencies?

Question

I\'m building an Apache Spark application in Scala and I\'m using SBT to build it. Here is the thing:

1. when I\'m developing under IntelliJ IDEA, I want Spark depend

Answer 1

You should be not looking at SBT for an IDEA specific setting. First of all, if the program is supposed to be run with spark-submit, how are you running it on IDEA ? I am guessing you'd be running as standalone in IDEA, while running it through spark-submit normally. If that's the case, add manually the spark libraries in IDEA, using File|Project Structure|Libraries. You'll see all dependencies listed from SBT, but you can add arbitrary jar/maven artifacts using the + (plus) sign. That should do the trick.