Storm command fails with NoClassDefFoundError after adding jsoup as provided dependency

Question

I\'m using JSoup in my project and I\'ve declared the dependency in my POM file. It compiles just fine and runs fine too, but only when I used the jar wit

Answer 1

This is as much a question about Maven as it is about Storm and its deployment model. You have to check out what the storm command actually does. First of all, it's actually a Python script that ultimately calls java.

If you look at the function get_classpath(extrajars), you'll note that it does not use the $CLASSPATH evironment variable at all. Rather, it loads the core Storm jars and any jars that you have under a directory lib/ relative to your working directory, as well as config files under ~/.storm

(You will find that ignoring $CLASSPATH is very common in many Java applications. Usually the first thing a "launch script" does is overwrite the CLASSPATH or not use it at all. This is to prevent unknown / unsupported / earlier versions of your jars from causing problems in your application).

As to your application fails when jsoup is declared "provided": when you declare the jar as a provided dependency, it will not be packaged in your "jar with dependencies" assembly. See this question for a good explanation: Difference between maven scope compile and provided for JAR packaging

The tl;dr explanation is that compile scope is shipped with your uber-jar, provided scope isn't, because it's expected to be "provided" by the container you are deploying to. Typically, the "container" is a Java web server, like Tomcat (hence, you should never have to ship JSP or Servlet jars with your web apps in Java). In this case, the "container" that you are expecting to "provide" classes is Storm. However, jsoup is not provided by Storm, hence your error.

Compile-scope classes still need to be shipped with your application because your application will be instantiating / using interfaces, enums, etc.

My recommendation is to just declare jsoup "compile" scope and move on. The alternative will be to write your own bespoke deployment script and/or assembly that puts jsoup under lib/ - essentially the same thing in the end.