AWS Glue to Redshift: Is it possible to replace, update or delete data?

后端 未结 6 1925
执念已碎
执念已碎 2020-12-25 12:33

Here are some bullet points in terms of how I have things setup:

  • I have CSV files uploaded to S3 and a Glue crawler setup to create the table and schema.
6条回答
  •  情深已故
    2020-12-25 13:19

    Job bookmarks are the key. Just edit the job and enable "Job bookmarks" and it won't process already processed data. Note that the job has to rerun once before it will detect it does not have to reprocess the old data again.

    For more info see: http://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html

    The name "bookmark" is a bit far fetched in my opinion. I would have never looked at it if I did not coincidentally stumble upon it during my search.

提交回复
热议问题