google-cloud-pubsub

问题 My test pipeline is really simple. It attempts to read from a topic created via the console. public static void main(String[] args) throws IOException { Options options = PipelineOptionsFactory.fromArgs(args). withValidation().as(Options.class); options.setStreaming(true); Pipeline pipeline = Pipeline.create(options); PCollection<String> input = pipeline .apply(PubsubIO.Read.topic(options.getPubsubTopic())) .apply(ParDo.of(new ExtractEvents())); pipeline.run(); } When I attempt to execute

问题 My code is basically following the official tutorials and the main purpose is to collect all messages from one subscription (Constants.UNFINISHEDSUBID) and republish them on another. But currently I'm facing a problem, that i can't solve. In my implementation calling subscriber.stopAsync() results in the following exception: Mai 04, 2017 4:59:25 PM com.google.common.util.concurrent.AbstractFuture executeListener SCHWERWIEGEND: RuntimeException while executing runnable com.google.common.util

问题 I am trying to setup a Dataflow job using the google provided template PubSub to BigQuery . I see an option to specify the Cloud Pub/Sub input topic but I don't see any option to specify Pub/Sub input subscription in GCP console UI. If I provide the topic, job would automatically create a subscription to read the messages from the provided topic. Problem with this is, the job will see only messages published to the topic after the Dataflow job has started. Anything published before to the

问题 The docs for PubSub state that the max payload after decoding is 10MB. My question is whether or not it is advantageous to compress the payload at the publisher before publishing to increase data throughput? This especially can be helpful if the payload has a high compression ratio like a json formatted payload. 回答1: If you are looking for efficiency on PubSub I would first concentrate on using the best API, and that's the gRPC one. If are using the client libraries then the chance is high

问题 Integration with Cloud Pub/Sub APIs from App Engine Standard I am working on developing a Google app engine app in standard Python environment. For some portions of the code, I need to integrate with Google Cloud pub/sub APIs. As mentioned here, Pub/Sub can only be integrated in the App Engine flexible environment (BTW it is also only in alpha). Can someone please describe how to integrate with Pub/Sub in the App Engine Standard environment? My use case description I am trying to integrate

问题 What I would like to do in the pipeline: Read from pub/sub (done) Transform this data to dictionary (done) Take the value of a specified key from the dict (done) Run a parametrized/dynamic query from BigQuery in which the where part should be like this: SELECT field1 FROM Table where field2 = @valueFromP/S The pipeline | 'Read from PubSub' >> beam.io.ReadFromPubSub(subscription='') | 'String to dictionary' >> beam.Map(lambda s:data_ingestion.parse_method(s)) | 'BigQuery' >> <Here is where I'm

问题 I am trying to make a watch request using python as referred to in the google APIs but it does not work. request = { 'labelIds': ['INBOX'], 'topicName': 'projects/myproject/topics/mytopic' } gmail.users().watch(userId='me', body=request).execute() I could not find a library or a package to use gmail.users() function. How do I make a watch request using an access token? 回答1: Do it in gmail python client(provide by google).under main function request = { 'labelIds': ['INBOX'], 'topicName':

问题 I'm relatively new to GCP and just starting to setup/evaluate my organizations architecture on GCP. Scenario: Data will flow into a pub/sub topic (high frequency, low amount of data). The goal is to move that data into Big Table. From my understanding you can do that either with a having a cloud function triggering on the topic or with Dataflow. Now I have previous experience with cloud functions which I am satisfied with, so that would be my pick. I fail to see the benefit of choosing one

问题 I need to import Pubsub_v1 and bigquery from google.cloud module. I have installed it and pip freeze shows below : gapic-google-cloud-pubsub-v1==0.15.4 google-cloud-bigquery==0.26.0 google-cloud-pubsub==0.27.0 proto-google-cloud-pubsub-v1==0.15.4 In my python script, i am importing the modules as below: import os from google.cloud import pubsub_v1 import time import json from google.cloud import bigquery The script is throwing error as : ImportError: cannot import name pubsub_v1 If i run

问题 We are using a cloud function triggered by Pub/Sub to ensure delivery of an e-mail. Sometimes the e-mail service takes a long time to respond and our cloud function terminates before we get an error back. Since the message has already been acknowledged our e-mail gets lost. The cloud function appears to be sending an ACK the Pub/Sub message automatically when we are called. Is there a way to delay the ACK until the successful completion of our code? Alternatively is there a way to catch