How to make restart-able producer?

Question

Latest version of kafka support exactly-once-semantics (EoS). To support this notion, extra details are added to each message. This means that at your consumer; if you print

Answer 1

In my case, multiple producers push data to one big topic. Therefore, reading entire topic would be nightmare.

The solution that I found is to maintain another topic i.e. "P1_Track" where producer can store metadata. Within a transaction a producer will send data to one big topic and P1_Track.

When I restart a producer, it will read P1_Track and figure out where to start from.

Thinking about storing last committed message in a database and using it when producer process restarts.