aws-glue

I need to copy a zipped file from one AWS S3 folder to another and would like to make that a scheduled AWS Glue job. I cannot find an example for such a simple task. Please help if you know the answer. May be the answer is in AWS Lambda, or other AWS tools. Thank you very much! You can do this, and there may be a reason to use AWS Glue: if you have chained Glue jobs and glue_job_#2 is triggered on the successful completion of glue_job_#1 . The simple Python script below moves a file from one S3 folder ( source ) to another folder ( target ) using the boto3 library, and optionally deletes the

Consider scala code: import com.amazonaws.services.glue.GlueContext import com.amazonaws.services.glue.util.{GlueArgParser, Job, JsonOptions} import org.apache.spark.SparkContext import scala.collection.JavaConverters.mapAsJavaMapConverter object MyGlueJob { def main(sysArgs: Array[String]) { val spark: SparkContext = SparkContext.getOrCreate() val glueContext: GlueContext = new GlueContext(spark) val args = GlueArgParser.getResolvedOptions(sysArgs, Seq("JOB_NAME").toArray) Job.init(args("JOB_NAME"), glueContext, args.asJava) val input = glueContext .getCatalogSource(database = "my_data_base",

According to AWS Athena limitations you can submit up to 20 queries of the same type at a time, but it is a soft limit and can be increased on request. I use boto3 to interact with Athena and my script submits 16 CTAS queries each of which takes about 2 minutes to finish. In a AWS account, it is only me who is using Athena service. However, when I look at the state of queries through console I see that only a few of queries (5 on average) are actually being executed despite all of them being in state Running . Here is what would normally see in Athena hisotry tab: I understand that, after I

I would like to know if it is possible to add a timestamp column in a table when it is loaded by an AWS Glue Job. First Scenario: Column A | Column B| TimeStamp A|2|2018-06-03 23:59:00.0 When a Crawler updates the table in the data catalog and run the job again, the table will add the new data in the table with a new time stamp.. Column A | Column B| TimeStamp A|4|2018-06-04 05:01:31.0 B|8|2018-06-04 06:02:31.0 import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue

I am having an AWS EMR cluster (v5.11.1) with Spark(v2.2.1) and trying to use AWS Glue Data Catalog as its metastore. As per guidelines provided in official AWS documentation (reference link below), I have followed the steps but I am facing some discrepancy with regards to accessing the Glue Catalog DB/Tables. Both EMR Cluster & AWS Glue are in the same account and appropriate IAM permissions have been provided. AWS Documentation : https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-glue.html Observations: - Using spark-shell (From EMR Master Node): Works . Able to access Glue DB

I use dynamic frames to write a parquet file in S3 but if a file already exists my program append a new file instead of replace it. The sentence that I use is this: glueContext.write_dynamic_frame.from_options(frame = table, connection_type = "s3", connection_options = {"path": output_dir, "partitionKeys": ["var1","var2"]}, format = "parquet") Is there anything like "mode":"overwrite" that replace my parquet files? Currently AWS Glue doesn't support 'overwrite' mode but they are working on this feature. As a workaround you can convert DynamicFrame object to spark's DataFrame and write it using