How to read a list of parquet files from S3 as a pandas dataframe using pyarrow?

Question

I have a hacky way of achieving this using boto3 (1.4.4), pyarrow (0.4.1) and pandas (0.20.3).

First, I can read a single parq

Answer 1

Probably the easiest way to read parquet data on the cloud into dataframes is to use dask.dataframe in this way:

import dask.dataframe as dd
df = dd.read_parquet('s3://bucket/path/to/data-*.parq')

dask.dataframe can read from Google Cloud Storage, Amazon S3, Hadoop file system and more!