Batching PubSub requests

问题

The NODEJS example code for batching pubsub requests looks like this:

// Imports the Google Cloud client library
const PubSub = require(`@google-cloud/pubsub`);

// Creates a client
const pubsub = new PubSub();

/**
 * TODO(developer): Uncomment the following lines to run the sample.
 */
// const topicName = 'your-topic';
// const data = JSON.stringify({ foo: 'bar' });
// const maxMessages = 10;
// const maxWaitTime = 10000;

// Publishes the message as a string, e.g. "Hello, world!" or JSON.stringify(someObject)
const dataBuffer = Buffer.from(data);

pubsub
  .topic(topicName)
  .publisher({
    batching: {
      maxMessages: maxMessages,
      maxMilliseconds: maxWaitTime,
    },
  })
  .publish(dataBuffer)
  .then(results => {
    const messageId = results[0];
    console.log(`Message ${messageId} published.`);
  })
  .catch(err => {
    console.error('ERROR:', err);
  });

For me it is not clear how to publish multiple messages simultaneously using this example. Could someone explain how to adjust this code so it can be used to publish multiple messages simultaneously?

回答1:

If you wanted to batch messages, then you'd need to keep hold of the publisher and call publish on it multiple times. For example, you could change the code to something like this:

// Imports the Google Cloud client library
const PubSub = require(`@google-cloud/pubsub`);

// Creates a client
const pubsub = new PubSub();


const topicName = 'my-topic';
const maxMessages = 10;
const maxWaitTime = 10000;
const data1 = JSON.stringify({ foo: 'bar1' });
const data2 = JSON.stringify({ foo: 'bar2' });
const data3 = JSON.stringify({ foo: 'bar3' });

const publisher = pubsub.topic(topicName).publisher({
    batching: {
      maxMessages: maxMessages,
      maxMilliseconds: maxWaitTime,
    },
  })

function handleResult(p) {
  p.then(results => {
    console.log(`Message ${results} published.`);
  })
  .catch(err => {
    console.error('ERROR:', err);
  });
}

// Publish three messages
handleResult(publisher.publish(Buffer.from(data1)));
handleResult(publisher.publish(Buffer.from(data2)));
handleResult(publisher.publish(Buffer.from(data3)));

Batching of messages is handled by the maxMessages and maxMilliseconds properties. The former indicates the maximum number of messages to include in a batch. The latter indicates the maximum number of milliseconds to wait to publish a batch. These properties trade off larger batches (which can be more efficient) with publish latency. If you are publishing many messages rapidly, then the maxMilliseconds property won't have much effect; as soon as ten messages are ready to go, the client library will make a publish request to the Cloud Pub/Sub service. However, if publishing is sporadic or slow, then a batch of messages might be sent before there are ten messages.

In the example code above, we call publish on three messages. This is not enough to fill up a batch and send it. Therefore, 10,000 milliseconds after the first call to publish, the three messages will be sent as a batch to Cloud Pub/Sub.

来源：https://stackoverflow.com/questions/49070836/batching-pubsub-requests

标签

google-cloud-pubsub