What is the passthrough used for in alpakka-kafka connector while producing messages?

问题

This is the code for producing a single message to Kafka given in the doc https://doc.akka.io/docs/alpakka-kafka/current/producer.html

val single: ProducerMessage.Envelope[KeyType, ValueType, PassThroughType] =
  ProducerMessage.single(
    new ProducerRecord("topicName", key, value),
    passThrough
  )

Could you please explain what is passThrough used for?

回答1:

passThrough is an additional value which comes to be available for use in ProducerMessage.Results’s passThrough().

As the official documentation states, the best and the most widely used practice is to use this value as a transport to passing the CommittableOffset to a downstream Committer.Sink.

This seems convenient in cases when you are using a CommittableSource along with the enable.auto.commit Kafka consumer option set to false. In such cases you have to commit Kafka offsets manually as you processing them. I in particular use such approach when reading records from one Kafka topic, processing them and then sending them to another Kafka topic.

Here is a simple use case:

Consumer
      .committableSource(consumerSettings, Subscriptions.topics(topicIn))
      .mapAsync { msg ⇒
        process(msg.record.value).map {
          response ⇒
            ProducerMessage.single(
              new ProducerRecord(topicOut, msg.record.key, response),
              passThrough = msg.committableOffset
            )
        }
      }
      .via(Producer.flexiFlow(producerSettings))
      .map(_.passThrough) // extract the passThrough
      .toMat(Committer.sink(committerSettings))(Keep.both) // commiting offsets
      .mapMaterializedValue(DrainingControl.apply)
      .run()

来源：https://stackoverflow.com/questions/58577149/what-is-the-passthrough-used-for-in-alpakka-kafka-connector-while-producing-mess

标签

scala

apache-kafka

akka-stream

reactive

alpakka