How do I unit test PySpark programs?

Question

My current Java/Spark Unit Test approach works (detailed here) by instantiating a SparkContext using \"local\" and running unit tests using JUnit.

The code has to be

Answer 1

Sometime ago I've also faced the same issue and after reading through several articles, forums and some StackOverflow answers I've ended with writing a small plugin for pytest: pytest-spark

I am already using it for few months and the general workflow looks good on Linux:

1. Install Apache Spark (setup JVM + unpack Spark's distribution to some directory)
2. Install "pytest" + plugin "pytest-spark"
3. Create "pytest.ini" in your project directory and specify Spark location there.
4. Run your tests by pytest as usual.
5. Optionally you can use fixture "spark_context" in your tests which is provided by plugin - it tries to minimize Spark's logs in the output.