How to read a list of parquet files from S3 as a pandas dataframe using pyarrow?

Question

I have a hacky way of achieving this using boto3 (1.4.4), pyarrow (0.4.1) and pandas (0.20.3).

First, I can read a single parq

Answer 1

Provided you have the right package setup

$ pip install pandas==1.1.0 pyarrow==1.0.0 s3fs==0.4.2

and your AWS shared config and credentials files configured appropriately

you can use pandas right away:

import pandas as pd

df = pd.read_parquet("s3://bucket/key.parquet")

---

In case of having multiple AWS profiles you may also need to set

$ export AWS_DEFAULT_PROFILE=profile_under_which_the_bucket_is_accessible

so you can access your bucket.