google-cloud-pubsub

问题 I'm currently attempting to use withIdAttribute with PubSubIO to deduplicate messages that come from PubSub (since PubSub only guarantees at least once delivery). My messages have four fields, label1 , label2 , timestamp , and value . A value is unique to the two labels at some timestamp. Therefore, I additionally set a uniqueID attribute before writing to PubSub equal to these three values joined as a string. For example, this is what I get from reading from a subscription using the gcp

问题 I am writing a program in Apache Beam using Python SDK to read from Pub/Sub the contents of a JSON file, and do some processing on the received string. This is the part in the program where I pull contents from Pub/Sub and do the processing: with beam.Pipeline(options=PipelineOptions()) as pipeline: lines = pipeline | beam.io.gcp.pubsub.ReadStringsFromPubSub(subscription=known_args.subscription) lines_decoded = lines | beam.Map(lambda x: x.decode("base64")) lines_split = lines_decoded | (beam

问题 When I try to send a message in the cloud platform GUI (i.e. topic -> publish message on the cloud platform topic page) my endpoint PHP script is triggered, but the POST data is empty. So all the permissions and domain verifications are in place. The topic and subscription both seem to be correct. I found this same question here but json_decode($HTTP_RAW_POST_DATA); did nothing. I also tried $content = null; foreach( $_POST as $k => $v ){ // Just to see what any possible data might be

问题 Can someone clarify what's the purpose for id_label argument in ReafFromPubSub transform? I'm using BigQuery sink, my understanding it acts like an insertId for BQ Streaming API, Tabledata: insertAll A unique ID for each row. BigQuery uses this property to detect duplicate insertion requests on a best-effort basis. For more information, see data consistency. However I don't see this expected behaviour. I'm publishing messages to Pub/Sub, each message with same attribute message_id value (this

问题 I'm trying to subscribe to push notifications from Google Classroom using this link and getting the following error. I have created the project, the topic and subscription under the same project and added the service account permissions as noted in the docs. Right now I'm working on this in the API Explorer on this page. I can verify that the topic lives under the developer console project. Are there any other issues that would cause the topic to not be visible in this project? { "error": {

问题 I am trying to get a JSON object and so to make sure I had it I wrote the code like this: var payload = JSON.parse( JSON.stringify(Buffer.from(pubsubMessage.data, 'base64').toString()) ); console.log(payload); // { timestamp: '1533564208', device_id: '2nd_test', temperature: '20.0' } console.log(typeof payload); // string EDIT - Based on your feedback, I will elaborate further on the code and the type. Below is the full code and what it returns: var payload = Buffer.from(pubsubMessage.data,

问题 Currently, we're working on a spring boot application in a GCP project, which connects to a PubSub endpoint in the same GCP project, but also to a PubSub endpoint in another GCP project. I want to use plain spring cloud GCP PubSub components, but with those, I have no chance to set a second PubSub connection to a second GCP project. Also, if I would have one service account, with PubSubTemplate object I have no possibility to target a topic in another project than the current from the service

问题 While running PubSub Kafka connect using the command: .\bin\windows\connect-standalone.bat .\etc\kafka\WorkerConfig.properties .\etc\kafka\configSink.properties .\etc\kafka\configSource.properties I get this error: Sending metadata request {topics=[test]} to node -1 could not scan file META-INF/MANIFEST.MF in url file:/C:/confluent-3.3.0/bin/../share/java/kafka-serde-tools/commons-compress-1.8.1.jar with scanner SubTypesScanner could not scan file META-INF/MANIFEST.MF in url file:/C:

问题 In GCP Pub/Sub. I have a topic, and created Subscription1 and started publishing messages. Can I add another subscription Subscription2, and can I replay old messages that were published before the Subscription2 is created, would it allow it? At what point would I loose access to messages (with in the retention period)?, deleting all the Subscriptions? Keeping at least one active Subscription would allow PubSub to add new Subscriptions and replay old messages? Can I also increase the

问题 In Google Pub/Sub, is it possible to use an HTTPS server running on a Compute Engine VM instance as an endpoint for a push subscription? I'm a bit stumped, as in the documentation it says the server must be reachable by a DNS name. Any advice appreciated - cheers! 回答1: It is possible, but you'll need configure a domain name to point to your GCE instance. That means you need: A domain name you can assign to the GCE instance. An SSL certificate signed by a well-known authority. (It has been