Azure Datalake gen 1 files not available immediately after write when read with sqlContext.read load in Azure Databricks

Question

I have a Databricks job that create files in an Azure Datalake Gen1 folder. It is scheduled to create one file pr hour. The job to create file starts at i.e. 10.35 and take