Logging Error message while reading or writing to Topics

问题

How do you log error messages while reading or writing to topic. We would be using Apache Beam API to read or write to topic. So I any exception is generated how do we log it. Can I send my data to other topic?

PubsubIO.writeMessages() PubsubIO.readMessages()

Can I write this DoFn and add debug logs log.debug("Publishing json message to pubsub topic"); PubsubIO.Write message = PubsubIO.writeMessages().to(pipelineOptions.getPubsubEnpEventTopic()); log.debug("Message published to pubsub");

回答1:

There are two methods 'withCoderAndParseFn' while reading from pubsub and 'withFormatFn' while writing to pubsub. These function are useful to parse messages while reading or writing. May be you can use them by applying your own logic in there like logging into stackdriver or pushing error data to pubsub.

More you can find here https://www.codota.com/web/assistant/code/rs/5c6563f8138b620001f3aa7c#L470

PubsubIO.<t>read().withCoderAndParseFn(coder, new ParsePayloadUsingCoder<>(coder));

private static class ParsePayloadUsingCoder<t> extends SimpleFunction<pubsubmessage, t=""> {
    private Coder<t> coder;

    public ParsePayloadUsingCoder(Coder<t> coder) {
      this.coder = coder;
    }

    @Override
    public T apply(PubsubMessage input) {
      try {
        return CoderUtils.decodeFromByteArray(coder, input.getPayload());
      } catch (CoderException e) {
        throw new RuntimeException("Could not decode Pubsub message", e);
      }
    }
  }

来源：https://stackoverflow.com/questions/58733142/logging-error-message-while-reading-or-writing-to-topics