Why does starting a streaming query lead to “ExitCodeException exitCode=-1073741515”?

Question

Been trying to get used to the new structured streaming but it keeps giving me below error as soon as I start a .writeStream query.

Any idea what could

Answer 1

The problem I had occurred while converting a DataFrame to a Parquet file. It would create the directory, but fail with “ExitCodeException exitCode=-1073741515”.

I ran Spark from Intellij 2020.2.2 x64 on Windows 10 (version 2004). I have spark-3.0.1-bin-hadoop3.2 installed in GitBash on my C drive. I downloaded the winutils from this Github repo https://github.com/cdarlint/winutils after being redirected from this repo https://github.com/steveloughran/winutils.

I installed them in C:\winutils directory (top level), which contained one subdirectory named /bin/, which contained the winutil.exe and associated files. This top-level path was added as a Windows environment variable in System variables named HADOOP_HOME. I also have a variable for SPARK_HOME to C:\Users\name\spark-3.0.1-bin-hadoop3.2\

I was getting this error, until I found this SO post and Moises Trelles' answer above, and also this page https://answers.microsoft.com/en-us/insider/forum/insider_wintp-insider_repair/how-do-i-fix-this-error-msvcp100dll-is-missing/c167d686-044e-44ab-8e8f-968fac9525c5?auth=1

I have a 64-bit system, so installed both x86 and x64 versions of msvcp100.dll as recommended in the answers.microsoft.com answer. I did not reboot, but I did close Intellij and reload and upon rerunning, the correct output (parquet file) was generated. Good luck! I'm so thankful for stackoverflow/google/internet/community of helpful people.