How to read a list of parquet files from S3 as a pandas dataframe using pyarrow?

Question

I have a hacky way of achieving this using boto3 (1.4.4), pyarrow (0.4.1) and pandas (0.20.3).

First, I can read a single parq

Answer 1

If you are open to also use AWS Data Wrangler.

import awswrangler as wr

df = wr.s3.read_parquet(path="s3://...")