google-cloud-pubsub

问题 I'm trying to build Google DataFlow pipeline, which has these steps: Read from pub/sub topic a message which contains filename. Find in the google bucket file from filename read each line from the file send each line with filename as a single message to another topic My problem is that I can't add filename to the final output message. Current implementation: ConnectorOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(ConnectorOptions.class); Pipeline p = Pipeline

问题 I would like to perform an action once all the messages from a pubsub topic have been acknowledged. I tried using Stackdriver monitoring API for the metric "Number of unacknowledged messages broken down by a cloud region" but did not understand the filter for region and why it is required. Where can I see what region my topics use? And for some unknown reason, for some of the topics, the API call fails for no reason at all. What is the best way of knowing if all messages have been

问题 I am using Google IOT core with mongoose os. I wanted to update device connection status to firestore. But i am unable to find event which reports mqtt connection status to pub/sub like when device disconnects or reconnect i.e if device is offline or not. I am stuck on this problem for days.Any help will be appreciated 回答1: Update As @devunwired mentioned in this response it is now possible to monitor Stackdriver logs for disconnect events. You must have at a minimum enabled INFO level

问题 I am trying to stream some data from google PubSub into BigQuery using a python dataflow. For testing purposes I have adapted the following code https://github.com/GoogleCloudPlatform/DataflowSDK-examples/blob/master/python/dataflow_examples/cookbook/bigquery_schema.py into a streaming pipeline by setting options.view_as(StandardOptions).streaming = True So then I changed the record_ids pipeline to read from Pub/Sub # ADDED THIS lines = p | 'Read PubSub' >> beam.io.ReadStringsFromPubSub(INPUT

问题 I am trying to stream some data from google PubSub into BigQuery using a python dataflow. For testing purposes I have adapted the following code https://github.com/GoogleCloudPlatform/DataflowSDK-examples/blob/master/python/dataflow_examples/cookbook/bigquery_schema.py into a streaming pipeline by setting options.view_as(StandardOptions).streaming = True So then I changed the record_ids pipeline to read from Pub/Sub # ADDED THIS lines = p | 'Read PubSub' >> beam.io.ReadStringsFromPubSub(INPUT

问题 I have implemented a model using google pubsub where the producer sends in the message and the subscriber processes the message and sends the response to the subscription. But how do I map the response to the publisher which sent the request? Are there any filters that can be put on the subscription so that the response can be tracked? or is there another way of implementing this? 回答1: There is no way in Cloud Pub/Sub for the publisher to know that the subscriber processed the message. One of

问题 This question already has answers here : Are global variables thread safe in flask? How do I share data between requests? (2 answers) Store large data or a service connection per Flask session (1 answer) Concurrency-safe way to initialize global data connections in Flask (1 answer) Closed 10 months ago . I have a Web server that runs a Flask app. I also need to implement a Google pub/sub subscriber, using a "Pull" strategy. This basically means I have to instanciate a subscriber, which will

问题 I'm trying to deploy Google pubsub java samples - appengine push application by referring Google pubsub java samples. I am deploying using, mvn gcloud:deploy -Dgcloud.version=1 -DskipTests=true Below is the complete output with errors. Help me to fix the errors and deploy the app. E:\java\cloud-pubsub-samples-java-master\appengine-push>mvn gcloud:deploy -Dgcloud.version=1 -DskipTests=true [INFO] Scanning for projects... [WARNING] The POM for com.google.appengine:appengine-maven-plugin:jar:2.0

问题 i have two question 1) I want to use Kafka with Google cloud Dataflow Pipeline program. in my pipeline program I want to read data from kafka is it possible? 2) I created Instance with BigQuery enabled now i want to enable Pubsub how can i do ? 回答1: (1) Ad mentioned by Raghu, support for writing to/reading from Kafka was added to Apache Beam in mid-2016 with the KafkaIO package. You can check the package's documentation[1] to see how to use it. (2) I'm not quite sure what you mean. Can you

问题 Basically what I understood from few Azure topics is as below: 1) Azure Event Hub - where data is received initially and converted into events 2) Service Bus - acting as a queue 3) Azure Event Grid - where events converted in hub are transferred here. so the connection is like below: Hub -> Service Bus -> Event Grid -> Pub Sub -> Storage I understood this concept. My problem is I want data to be pushed from event grid to GCP (subscription / topics). So how can I establish this using PUSH