Apache Spark reads for S3: can't pickle thread.lock objects

问题:

So I want my Spark App to read some text from Amazon's S3. I Wrote the following simple script:

import boto3 s3_client = boto3.client('s3') text_keys = ["key1.txt", "key2.txt"] data = sc.parallelize(text_keys).flatMap(lambda key: s3_client.get_object(Bucket="my_bucket", Key=key)['Body'].read().decode('utf-8'))

When I do data.collect I get the following error:

TypeError: can't pickle thread.lock objects

and I don't seem to find any help online. Have perhaps someone managed to solve the above?

回答1:

Your s3_client isn't serialisable.

Instead of flatMap use mapPartitions, and initialise s3_client inside the lambda body to avoid overhead. That will:

1. init s3_client on each worker
2. reduce initialisation overhead

转载请标明出处:Apache Spark reads for S3: can't pickle thread.lock objects

文章来源: Apache Spark reads for S3: can't pickle thread.lock objects