How to read a list of parquet files from S3 as a pandas dataframe using pyarrow?

问题:

I have a hacky way of achieving this using boto3 (1.4.4), pyarrow (0.4.1) and pandas (0.20.3).

First, I can read a single parquet file locally like this:

import pyarrow.parquet as pq  path = 'parquet/part-r-00000-1e638be4-e31f-498a-a359-47d017a0059c.gz.parquet' table = pq.read_table(path) df = table.to_pandas() 

I can also read a directory of parquet files locally like this:

import pyarrow.parquet as pq  dataset = pq.ParquetDataset('parquet/') table = dataset.read() df = table.to_pandas() 

Both work like a charm. Now I want to achieve the same remotely with files stored in a S3 bucket. I was hoping that something like this would work:

dataset = pq.ParquetDataset('s3n://dsn/to/my/bucket') 

But it does not:

OSError: Passed non-file path: s3n://dsn/to/my/bucket

After reading pyarrow's documentation thoroughly, this does not seem possible at the moment. So I came out with the following solution:

Reading a single file from S3 and getting a pandas dataframe:

import io import boto3 import pyarrow.parquet as pq  buffer = io.BytesIO() s3 = boto3.resource('s3') s3_object = s3.Object('bucket-name', 'key/to/parquet/file.gz.parquet') s3_object.download_fileobj(buffer) table = pq.read_table(buffer) df = table.to_pandas() 

And here my hacky, not-so-optimized, solution to create a pandas dataframe from a S3 folder path:

import io import boto3 import pandas as pd import pyarrow.parquet as pq  bucket_name = 'bucket-name' def download_s3_parquet_file(s3, bucket, key):     buffer = io.BytesIO()     s3.Object(bucket, key).download_fileobj(buffer)     return buffer  client = boto3.client('s3') s3 = boto3.resource('s3') objects_dict = client.list_objects_v2(Bucket=bucket_name, Prefix='my/folder/prefix') s3_keys = [item['Key'] for item in objects_dict['Contents'] if item['Key'].endswith('.parquet')] buffers = [download_s3_parquet_file(s3, bucket_name, key) for key in s3_keys] dfs = [pq.read_table(buffer).to_pandas() for buffer in buffers] df = pd.concat(dfs, ignore_index=True) 

Is there a better way to achieve this? Maybe some kind of connector for pandas using pyarrow? I would like to avoid using pyspark, but if there is no other solution, then I would take it.

回答1:

You should use the s3fs module as proposed by yjk21. However as result of calling ParquetDataset you'll get a pyarrow.parquet.ParquetDataset object. To get the Pandas DataFrame you'll rather want to apply .read_pandas().to_pandas() to it:

import pyarrow.parquet as pq import s3fs s3 = s3fs.S3FileSystem()  pandas_dataframe = pq.ParquetDataset('s3://your-bucket/', filesystem=s3).read_pandas().to_pandas() 

回答2:

You can use s3fs from dask which implements a filesystem interface for s3. Then you can use the filesystem argument of ParquetDataset like so:

import s3fs s3 = s3fs.S3FileSystem() dataset = pq.ParquetDataset('s3n://dsn/to/my/bucket', filesystem=s3) 

文章来源: How to read a list of parquet files from S3 as a pandas dataframe using pyarrow?