arguments error while calling an AWS Glue Pythonshell job from boto3

问题

Based on the previous post, I have an AWS Glue Pythonshell job that needs to retrieve some information from the arguments that are passed to it through a boto3 call.

My Glue job name is test_metrics

The Glue pythonshell code looks like below

import sys
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv,
                          ['test_metrics',
                           's3_target_path_key',
                           's3_target_path_value'])

print ("Target path key is: ", args['s3_target_path_key'])
print ("Target Path value is: ", args['s3_target_path_value'])

The boto3 code that calls this job is below:

glue = boto3.client('glue')

response = glue.start_job_run(
    JobName = 'test_metrics',
    Arguments = {
        '--s3_target_path_key': 's3://my_target',
        '--s3_target_path_value': 's3://my_target_value'
    } 
)

print(response)

I see a 200 response after I run the boto3 code in my local machine, but Glue error log tells me:

test_metrics.py: error: the following arguments are required: --test_metrics

What am I missing?

回答1:

Which job you are trying to launch? Spark Job or Python shell job?

If spark job, JOB_NAME is mandatory parameter. In Python shell job, it is not needed at all.

So in your python shell job, replace

args = getResolvedOptions(sys.argv,
                          ['test_metrics',
                           's3_target_path_key',
                           's3_target_path_value'])

with

args = getResolvedOptions(sys.argv,
                           ['s3_target_path_key',
                           's3_target_path_value'])

回答2:

Seems like the documentation is kinda broken.

I had to update the boto3 code like below to make it work

glue = boto3.client('glue')
response = glue.start_job_run(
               JobName = 'test_metrics',
               Arguments = {
                 '--test_metrics': 'test_metrics',
                 '--s3_target_path_key':   's3://my_target',
                 '--s3_target_path_value':   's3://my_target_value'} )

来源：https://stackoverflow.com/questions/58046474/arguments-error-while-calling-an-aws-glue-pythonshell-job-from-boto3