Reading DataFrame from partitioned parquet file

Question

How to read partitioned parquet with condition as dataframe,

this works fine,

val dataframe = sqlContext.read.parquet(\"file:///home/msoproj/dev_data         


        

Answer 1

If you want to read for multiple days, for example day = 5 and day = 6 and want to mention the range in the path itself, wildcards can be used:

val dataframe = sqlContext
  .read
  .parquet("file:///your/path/data=jDD/year=2015/month=10/day={5,6}/*")

Wildcards can also be used to specify a range of days:

val dataframe = sqlContext
  .read
  .parquet("file:///your/path/data=jDD/year=2015/month=10/day=[5-10]/*")

This matches all days from 5 to 10.