Using AWS Glue to convert very big csv.gz (30-40 gb each) to parquet

Question

There are lots of such questions but nothing seems to help. I am trying to covert quite large csv.gz files to parquet and keep on getting various errors like

\'C         


        

Answer 1

How many DPUs you are using. This article gives a nice overview of DPU capacity planning. Hope that helps. There is no definite rulebook from AWS stating how much DPU you need to process a particular size.