Can Kafka Streams output topic be on a separate cluster?

问题

I have a topic where all logs are pushed to centralized topic but I would like to filter out some of those records to a separate topic and cluster if possible.

Thanks

回答1:

Kafka streams not allow to create stream with source and output topics from different Kafka clusters. So the following code will not work for you

streamsBuilder.stream(sourceTopicName).filter(..).to(outputTopicName)

in this case it expects that outputTopicName is from the same cluster as topic sourceTopicName.

As a workaround, in order to send messages into output topic from another cluster, you could use additionally created KafkaProducer with property bootstrap.servers that will point to external cluster and KStream.foreach() method.

streamsBuilder.stream(sourceTopicName)
    .filter((key, value) -> ..)
    .foreach((key, value) -> 
        sendMessage(kafkaProducerFromAnotherCluster, destinationTopicName, key, value);


public static void sendMessage(KafkaProducer<String, String> kafkaProducer, 
                               String destinationTopicName, String key, String value) {
    try {
        kafkaProducer.send(new ProducerRecord(destinationTopicName, key, value));
    } catch (RuntimeException ex) {
        log.error(errorMessage, ex);
    }
}

Another option is to create output topic in your Kafka cluster that will have filtered messages and setup Kafka Mirroring between two clusters (so messages will be copied from one topic to second from another cluster).

来源：https://stackoverflow.com/questions/53034610/can-kafka-streams-output-topic-be-on-a-separate-cluster