Different outcomes for same function in Spark. How to troubleshoot?

Question

I\'m creating an ETL job to move several thousand files from S3 to Azure Data Lake using Spark (Databricks, Python, Spark 3). The files are zip compressed JSON and therefore