google-cloud-storage

Why does gsutil cp require storage.objects.delete on versioned bucket?

喜夏-厌秋 提交于 2021-02-11 06:52:09
问题 I'm using a service account to upload a file to Google Cloud Storage bucket that has versioning. I want to keep the service account privileges minimal, it only ever needs to upload files so I don't want to give it permission to delete files, but the upload fails (only after streaming everything!) saying it requires delete permission. Shouldn't it be creating a new version instead of deleting? Here's the command: cmd-that-streams | gsutil cp -v - gs://my-bucket/${FILE}

Why does gsutil cp require storage.objects.delete on versioned bucket?

孤街浪徒 提交于 2021-02-11 06:52:04
问题 I'm using a service account to upload a file to Google Cloud Storage bucket that has versioning. I want to keep the service account privileges minimal, it only ever needs to upload files so I don't want to give it permission to delete files, but the upload fails (only after streaming everything!) saying it requires delete permission. Shouldn't it be creating a new version instead of deleting? Here's the command: cmd-that-streams | gsutil cp -v - gs://my-bucket/${FILE}

Too many open files: '/home/USER/PATH/SERVICE_ACCOUNT.json' when calling Google's Natural Language API

无人久伴 提交于 2021-02-11 06:27:39
问题 I'm working on a Sentiment Analysis project using the Google Cloud Natural Language API and Python, this question might be similar to this other question, what I'm doing is the following: Reads a CSV file from Google Cloud Storage, file has approximately 7000 records. Converts the CSV into a Pandas DataFrame. Iterates over the dataframe and calls the Natural Language API to perform sentiment analysis on one of the dataframe's columns, on the same for loop I extract the score and magnitude

Google App Engine deleted default GCS bucket. Is there a way to restore it?

淺唱寂寞╮ 提交于 2021-02-10 20:45:52
问题 I was deleting some files from the default bucket of my application and I accidentally deleted also the bucket itself using gsutil rm command. Is there a way to restore/recreate/bring it back ? I would like to continue to use the free quota and not enable billing. What I tried to do so far is that (but I'm not sure it is correct). I first enable billing just to re-create a bucket with the same name: xxx.appspot.com. In order to be able to do that I verified that I'm the owner of the domain. I

Google bigquery export table to multiple files in Google Cloud storage and sometimes one single file

泄露秘密 提交于 2021-02-10 20:25:46
问题 I am using Bigquery python libraries to export data from Bigquery tables into GCS in csv format. I have given a wildcard pattern assuming some tables can be more than 1 GB Sometimes even though table is few MB it creates multiple files and sometimes just it creates just 1 file. Is there a logic behind this? My export workflow is the following: project = bq_project dataset_id = bq_dataset_id table_id = bq_table_id bucket_name =bq_bucket_name workflow_name=workflow_nm csv_file_nm=workflow_nm+"/

Deleting Buckets in Google Cloud Storage

醉酒当歌 提交于 2021-02-10 17:26:13
问题 I have followed the directions provided by Google to delete some old buckets in the in my account ... it is very straight forward process listed however after confirming the deletion to occur the "Preparing to Delete" pops up on the bottom left, but the system never deletes the files and bucket ? I have posted this several times but no one have suggested a solution or a reason why the process does not work. 回答1: If you have a lot of files in your bucket, it might simply take a long time to

Deleting Buckets in Google Cloud Storage

孤者浪人 提交于 2021-02-10 17:26:03
问题 I have followed the directions provided by Google to delete some old buckets in the in my account ... it is very straight forward process listed however after confirming the deletion to occur the "Preparing to Delete" pops up on the bottom left, but the system never deletes the files and bucket ? I have posted this several times but no one have suggested a solution or a reason why the process does not work. 回答1: If you have a lot of files in your bucket, it might simply take a long time to

Deleting Buckets in Google Cloud Storage

纵然是瞬间 提交于 2021-02-10 17:24:52
问题 I have followed the directions provided by Google to delete some old buckets in the in my account ... it is very straight forward process listed however after confirming the deletion to occur the "Preparing to Delete" pops up on the bottom left, but the system never deletes the files and bucket ? I have posted this several times but no one have suggested a solution or a reason why the process does not work. 回答1: If you have a lot of files in your bucket, it might simply take a long time to

App Engine Fails to Download Greater Than 33MB

青春壹個敷衍的年華 提交于 2021-02-10 16:50:01
问题 The code below is straight from google apis github for downloading a file from GCS. I changed it to be able to download a file to my local computer. If the file is larger than 33MB the app will stop attempting after 2 min or less. I think I read that app engine's limit is 32 or 33. I don't see anything in quotas to increase though. Can someone point me in the right direction as to how to manage this? if (blob.getSize() < 1_000_000) { // Blob is small read all its content in one request byte[]

App Engine Fails to Download Greater Than 33MB

拥有回忆 提交于 2021-02-10 16:49:28
问题 The code below is straight from google apis github for downloading a file from GCS. I changed it to be able to download a file to my local computer. If the file is larger than 33MB the app will stop attempting after 2 min or less. I think I read that app engine's limit is 32 or 33. I don't see anything in quotas to increase though. Can someone point me in the right direction as to how to manage this? if (blob.getSize() < 1_000_000) { // Blob is small read all its content in one request byte[]