gsutil

问题 I can't seem to find the documentation about how to upload files that are greater than 30MB to google cloud storage. I have just been using: gsutil cp test.csv gs://example Any help would be awesome. Thanks 回答1: If you are using the CSV format you can also upload the gzip version. Just create a gzip file and upload on GCS using below: gsutil cp <localgzipfile> gs://location/of/gzipfile.csv.gz I am using this strategy to upload more than 200MB flat file whose gzip version is mere 40 MB. Hope

问题 One of our collaborators has made some data available on AWS and I was trying to get it into our google cloud bucket using gsutil (only some of the files are of use to us, so I don't want to use the GUI provided on GCS). The collaborators have provided us with the AWS bucket ID, the aws access key id, and aws secret access key id. I looked through the documentation on GCE and editied the ~/.botu file such that the access keys are incorporated. I restarted my terminal and tried to do an 'ls'

问题 I'm new with google cloud dataflow and I need to copy files in GCS (Google Cloud Storage) from one bucket to another and rename. Answer with example would highly appreciated. 回答1: Technically yes you can do this, but it would be better if you used Dataflow to just pick up the files and move them over to the new bucket instead of as a wrapper around gustil. The class you need should be beam.io.gcsio.GcsIO() and that should be able to pick up and put down files where you need it too. 来源： https:

问题 So when I bring up a GCE instance using the standard debian 7 image, and issue a "gsutil config" command, it fails with the following message: jcortez@master:~$ gsutil config Failure: No handler was ready to authenticate. 4 handlers were checked. ['ComputeAuth', 'OAuth2Auth', 'OAuth2ServiceAccountAuth', 'HmacAuthV1Handler'] Check your credentials. I've tried it on the debian 6 and centos instances and had the same results. Issuing "gcutil config" works fine however. I gather I need to set up

问题 I was trying to change storage class of a set of existing objects (over 300 GBs) as advised in this post: I tried it on one file first: fyn@pod-arch:~$ gsutil ls -L gs://some-bucket/sub-dir/audioArch.mp3 gs://some-bucket/sub-dir/audioArch.mp3: Creation time: Fri, 29 Jul 2016 00:52:51 GMT Update time: Fri, 05 Aug 2016 15:40:51 GMT Storage class: DURABLE_REDUCED_AVAILABILITY Content-Language: en Content-Length: 43033404 Content-Type: audio/mpeg ... ... fyn@pod-arch:~$ gsutil -m rewrite -s

问题 gsutil use boto configuration file for authenticate at Google Cloud Storage. I generate ClientID in console.developers.google.com - that allow put files to Google Drive by python script: #!/usr/bin/env python from apiclient.discovery import build from apiclient.http import MediaFileUpload import httplib2 from oauth2client.client import SignedJwtAssertionCredentials credentials = SignedJwtAssertionCredentials( service_account_name='301714561983-nkd8kalz1op3bdjn75ija1b7ef1sslpr@developer

问题 backup_to_gcs.sh is a backup script that uses gsutil rsync to copy files to Google Cloud Storage. The backup script runs when called by user wolfv (who installed the google-cloud-sdk): $ /home/wolfv/scripts/backup_to_gcs/backup_to_gcs.sh backup_to_gcs.sh in progress ... backup_to_gcs.sh pass So far so good. But root needs to run gsutil so that the backup script can be called from Anacron. So lets have root run the script. It's safe for root and wolfv to share credentials because it's the same

问题 We've been using gsutil -m rsync -r to keep dev and deploy boxes in sync with a GCS bucket for nearly 2 years without any problem. There are about 85k objects in the bucket. Until recently, this worked perfectly: we'd run a deploy-box -> GCS rsync every 15 mins or so, to keep all new uploaded resource backed up, and then a GCS -> dev box rsync whenever we wanted to refresh the local dev data (running on OSX El Capitan). Within the last couple of months, though, the GCS->dev rsync has started

问题 Greetings StackOverflow, I'm working on a small project on Windows which needs to read the output of GSUTIL's copy function. Problem is, the output of the copy function doesn't seem to work via the Standard Output. Also, the behavior of GSUTIL is inconsistent: piping output doesn't work with the copy function but using the list function is does work. When I use the following command in my command prompt the output is displayed in the command prompt but not redirected to the text file. This

问题 I am trying to download a folder which is inside my Google Cloud Bucket , I read from google docs gsutil/commands/cp and executed below the line. gsutil cp -r appengine.googleapis.com gs://my-bucket But i am getting the error CommandException: No URLs matched: appengine.googleapis.com Edit By running below command gsutil cp -r gs://logsnotimelimit . I am getting Error IOError: [Errno 22] invalid mode ('ab') or filename: u'.\logsnotimelimit\appengine.googleapis.com\nginx.request\2018\03\14\14