How to load directory of JSON files into Apache Spark in Python

Question

I\'m relatively new to Apache Spark, and I want to create a single RDD in Python from lists of dictionaries that are saved in multiple JSON files (each is gzipped and contai

Answer 1

You can load a directory of files into a single RDD using textFile and it also supports wildcards. That wouldn't give you file names, but you don't seem to need them.

You can use Spark SQL while using basic transformations like map, filter etc. SchemaRDD is also an RDD (in Python, as well as Scala)