Separating application logs in Logback from Spark Logs in log4j

Question

I have a Scala Maven project using that uses Spark, and I am trying implement logging using Logback. I am compiling my application to a jar, and deploying to an EC2 instance

Answer 1

After much struggle I've found another solution: library shading. After I've shaded org.slf4j, my application logs are separated from spark logs. Furthermore, logback.xml in my application jar is honored.

Here you can find information on library shading in sbt, in this case it comes down to putting:

assemblyShadeRules in assembly += ShadeRule.rename(s"org.slf4j.**" -> "your_favourite_prefix.@0").inAll

in your build.sbt settings.

Side note: If you are not sure whether shading actually happened, open your jar in some archive browser and check whether directory structure reflects shaded one, in this case your jar should contain path /your_favourite_prefix/org/slf4j, but not /org/slf4j