I need to read contiguous files in pySpark. The following works for me.
from pyspark.sql import SQLContext
file = "events.parquet/exportDay=2015090[1-7]"
df = sqlContext.read.load(file)
How do I read files 8-14?
kathleen
Use curly braces.
file = "events.parquet/exportDay=201509{08,09,10,11,12,13,14}"
Here's a similar question on stack overflow: Pyspark select subset of files using regex glob. They suggest either using curly braces, OR performing multiple reads and then unioning the objects (whether they are RDDs or data frames or whatever, there should be some way).
Barry Loper
It uses shell globbing, I believe.
The post: How to read multiple text files into a single RDD?
Seems to suggest the below should work.
"events.parquet/exportDay=2015090[89],events.parquet/exportDay=2015091[0-4]"
来源：https://stackoverflow.com/questions/37732766/read-range-of-files-in-pyspark