How to read multiple csv (leaving out specific ones) from a nested directory in pyspark?

Question

Lets say I have a directory called \'all_data\', and inside this, I have several other directories based on the date of the data that it contains. These directories are name