boto3

Wait until AWS Glue crawler has finished running

℡╲_俬逩灬. 提交于 2021-02-19 05:30:38
问题 In the documentation, I cannot find any way of checking the run status of a crawler. The only way I am doing it currently is constantly checking AWS to check if the file/table has been created. Is there a better way to block until crawler finishes its run? 回答1: You can use boto3 (or similar) to do it. There is the get_crawler method. You will find needed information in "LastCrawl" section https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get

Wait until AWS Glue crawler has finished running

ぐ巨炮叔叔 提交于 2021-02-19 05:30:17
问题 In the documentation, I cannot find any way of checking the run status of a crawler. The only way I am doing it currently is constantly checking AWS to check if the file/table has been created. Is there a better way to block until crawler finishes its run? 回答1: You can use boto3 (or similar) to do it. There is the get_crawler method. You will find needed information in "LastCrawl" section https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get

How can I use versioning in S3 with boto3?

假装没事ソ 提交于 2021-02-18 11:44:08
问题 I am using a versioned S3 bucket, with boto3. How can I retrieve all versions for a given key (or even all versions for all keys) ? I can do this: for os in b.objects.filter(Prefix=pref): print("os.key") but that gives me only the most recent version for each key. Many thanks, 回答1: import boto3 bucket = 'bucket name' key = 'key' s3 = boto3.resource('s3') versions = s3.Bucket(bucket).object_versions.filter(Prefix=key) for version in versions: obj = version.get() print(obj.get('VersionId'), obj

Cannot revoke_ingress for non-default VPC with boto3

坚强是说给别人听的谎言 提交于 2021-02-11 17:19:27
问题 AWS Lambda / python 2.7 / boto3 I'm trying to revoke one rule out of many in a security group ( SG_we_are_working_with ) but receive error An error occurred (InvalidGroup.NotFound) when calling the RevokeSecurityGroupIngress operation: The security group 'sg-xxxxx' does not exist in default VPC 'none' The SG is really not in the default VPC but custom one, but I mention VPC id explicitly! SG_we_are_working_with = 'sg-xxxxx' SG_which_is_the_source_of_the_traffic = 'sg-11111111' VpcId = 'vpc

DynamoDB Update operation without using the Key attribute

北城以北 提交于 2021-02-11 14:58:16
问题 The use case: I have a bid table which holds the bid on Loades. One Load can have multiple Bids. The Bid status is new for every bid. Once the bid is accepted by the Admin(The person who put that Load up for bidding) then I need to change the status for that particular bid as "Accepted" and for other bids on the same Load the status should be "rejected". Table Definition: Bid_id(Which is unique for every record) and Load_id(multiple entries) is my primary and sort key respectively. How do I

aioboto3 speedup not as expected

…衆ロ難τιáo~ 提交于 2021-02-10 17:59:50
问题 I'm trying the aioboto3 lib, which looks extremely promising for speeding certain tasks up. For example, I need to find tags for all S3 objects within a particular bucket and prefix. But unfortunately the speed increase is not what I had hoped for. With 1000 objects, it's about half the time. With 8000 objects, it was about the same time! This was run on a c3.8xlarge EC2 instance. Code: import asyncio import aioboto3 from boto3.dynamodb.conditions import Key import boto3 import logging import

How to speed up processing time of AWS Transcribe?

拈花ヽ惹草 提交于 2021-02-10 16:17:27
问题 I have 6 second audio recording( ar-01.wav ) in wav format. I want to transcribe the audio file to text using amazon services. For that purpose I created a bucket by name test-voip and uploaded the audio file to bucket. When I try to convert the speech to text, a 6 second audio is taking 13.12 seconds. Here is my code snippet session = boto3.Session(aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key) transcribe = session.client('transcribe', region_name='us-east

How to fix “MissingHeaders” Error while appending where clause with s3 select

假如想象 提交于 2021-02-10 05:41:30
问题 I have a csv file in the format IDATE_TIMESTAMP,OPEN,HIGH,LOW,CLOSE,VOLUME 1535535060,94.36,94.36,94.36,94.36,1 1535535120,94.36,94.36,93.8,93.8,1 1535535180,93.8,93.8,93.8,93.8,0 1535535240,93.8,93.8,93.74,93.74,1 1535535300,93.74,93.74,93.74,93.74,0 1535535360,93.74,93.74,93.74,93.74,0 1535535420,93.74,93.74,93.74,93.74,0 1535535480,93.74,93.74,93.74,93.74,0 1535535540,93.74,93.74,93.74,93.74,0 . . . . I have to and from timestamp which will filter out the data from the file and return the

Boto3 - botocore.errorfactory.NotAuthorizedException

旧时模样 提交于 2021-02-09 07:31:38
问题 Using boto3 and warrant in python3. Headless system. Trying to login using a python script using ASWSRP from warrant. With user pool there is the problem of changing the temporary password. So... I have the code trying the temp password first and when the fails it switches over to trying the permanent password. I am running into a couple different issues: 1) After I get the temporary password changed and I run the script again I get the exception: botocore.errorfactory.NotAuthorizedException

python (boto3) program to delete old snapshots in aws

谁说我不能喝 提交于 2021-02-08 15:00:28
问题 I have already written a program to delete old snapshots.But the problem for me now is if the snapshot is attached with an ami then it doesn't get deleted and the program also stops.It displays the following message : botocore.exceptions.ClientError: An error occurred (InvalidSnapshot.InUse) when calling the DeleteSnapshot operation: The snapshot snap-12345678 is currently in use by ami-12345 I want the program to skip those snapshots alone and continue to delete other snapshots. here is my