aws-glue

问题 I've created an AWS Glue crawler to gather information on my Redshift Database. Is there a way I can customize this crawler to update the "comment" field in Glue with a field that all my tables have? This field would be the comment or description field that all Redshift tables have. Any help would be appreciated. Thanks 来源： https://stackoverflow.com/questions/59200724/aws-glue-customized-crawler

问题 I haven't been able to figure this out, but I'm trying to use a direct output committer with AWS Glue: spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 Is it possible to use this configuration with AWS Glue? 回答1: Option 1 : Glue uses spark context you can set hadoop configuration to aws glue as well. since internally dynamic frame is kind of dataframe. sc._jsc.hadoopConfiguration().set("mykey","myvalue") I think you neeed to add the correspodning class also like this sc._jsc

问题 As of January 19, 2018 updates, Athena can skip the header row of files, Support for ignoring headers. You can use the skip.header.line.count property when defining tables, to allow Athena to ignore headers. I use AWS Glue in Cloudformation to manage my Athena tables. Using the Glue Table Input, how can I tell Athena to skip the header row? 回答1: Basing off the full template for AWS::Glue::Table here, making the change from, Resources: ... MyGlueTable: ... Properties: ... TableInput: ...

问题 Part One : I tried glue crawler to run on dummy csv loaded in s3 it created a table but when I try view table in athena and query it it shows Zero Records returned. But the demo data of ELB in Athena works fine. Part Two (Scenario:) Suppose I Have a excel file and data dictionary of how and what format data is stored in that file , I want that data to be dumped in AWS Redshift What would be best way to achieve this ? 回答1: I experienced the same issue. You need to give the folder path instead

问题 Part One : I tried glue crawler to run on dummy csv loaded in s3 it created a table but when I try view table in athena and query it it shows Zero Records returned. But the demo data of ELB in Athena works fine. Part Two (Scenario:) Suppose I Have a excel file and data dictionary of how and what format data is stored in that file , I want that data to be dumped in AWS Redshift What would be best way to achieve this ? 回答1: I experienced the same issue. You need to give the folder path instead

问题 I am relatively new to AWS and this may be a bit less technical question, but at present AWS Glue notes a maximum of 25 jobs permitted to be created. We are loading in a series of tables that each have their own job that subsequently appends audit columns. Each job is very similar, but simply changes the connection string source and target. Is there a way to parameterize these jobs to allow for reuse and simply pass the proper connection strings to them? Or even possibly loop through a set

问题 I'm working on an ETL job that will ingest JSON files into a RDS staging table. The crawler I've configured classifies JSON files without issue as long as they are under 1MB in size. If I minify a file (instead of pretty print) it will classify the file without issue if the result is under 1MB. I'm having trouble coming up with a workaround. I tried converting the JSON to BSON or GZIPing the JSON file but it is still classified as UNKNOWN. Has anyone else run into this issue? Is there a

问题 I have partitioned data in CSV files on S3: s3://bucket/dataset/p=1/*.csv (partition #1) ... s3://bucket/dataset/p=100/*.csv (partition #100) I run a classifier over s3://bucket/dataset/ and the result looks very much promising as it detects 150 columns (c1,...,c150) and assigns various data types. Loading the resulting table in Athena and querying ( select * from dataset limit 10 ) it though will yield the error message: HIVE_PARTITION_SCHEMA_MISMATCH: There is a mismatch between the table

问题 I'm trying to figure out how to automatically kick off an AWS Glue Job when an AWS Glue Crawler completes. I see that the Crawlers send events when they complete, but I'm struggling to parse through the documentation to figure out how to listen to that event and then launch the AWS Glue Job. This seems like a fairly simple question, but I haven't been able to find any leads so far. I'd appreciate some help. Thanks in advance! 回答1: You can create a CloudWatch event, choose Glue Crawler state

问题 I have some json files stored in s3, and I need to convert them, at the folder folder they are, to csv format. Currently I'm using glue to map them to athena, but, as I said, now I need to map them to csv. Is it possible to use a Glue JOB to do that? I trying to understand if a glue job can crawl into my s3 folder directories, converting all json files it finds to csv (as new files). If not possible, is there any aws service that could help me do that? EDIT1: Here's the current code i'm