How can I use a Lambda function to call a Glue function (ETL) when a text file is loaded to an S3 bucket

问题

I am trying to set up a lambda function that activates a Glue function when a .txt file is uploaded to an S3 bucket, I am using python 3.7

So far I have this:

from __future__ import print_function

import json
import boto3
import urllib

print('Loading function')

s3 = boto3.client('s3') 

def lambda_handler(event, context): # handler
    source_bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.quote_plus(event['Records'][0]['s3']['object']['key'].encode('utf8'))
    try:
        # what to put here
    except Exception as e:
        print(e)
        print('Error')
        raise e

But I don't understand how can I call the glue function

回答1:

I manage to do it like this:

from __future__ import print_function
import json
import boto3

client = boto3.client('glue')

def lambda_handler(event, context):
    response = client.start_job_run(JobName = 'GLUE_CODE_NAME')

Later I will post the S3 event

回答2:

You can configure an S3 Event Notification that will trigger this Lambda function when PUT object actions is called on an S3 prefix.

https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-event-notifications.html

This lambda function can then trigger the StartJobRun action of Glue API.

https://docs.aws.amazon.com/glue/latest/webapi/API_StartJobRun.html

来源：https://stackoverflow.com/questions/55367322/how-can-i-use-a-lambda-function-to-call-a-glue-function-etl-when-a-text-file-i