How to write JDBC Sink for Spark Structured Streaming [SparkException: Task not serializable]?

Question

I need a JDBC sink for my spark structured streaming data frame. At the moment, as far as I know DataFrame’s API lacks writeStream to JDBC implementation (neith

Answer 1

In case somebody encounters this in an interactive workbook, this solution also works:

Instead of saving the JDBCSinkclass to a seperate file, you can also just declare it as a separate package ("Packaged cell") within the same workbook and import that package in the cell where you are using it. Well described here https://docs.databricks.com/user-guide/notebooks/package-cells.html