Apache Kafka order of messages with multiple partitions

Question

As per Apache Kafka documentation, the order of the messages can be achieved within the partition or one partition in a topic. In this case, what is the parallelism benefit

Answer 1

You may consider having a field which has the Timestamp/Date at the time of creation of the dataset at the source.

Once, the data is consumed you can load the data into database. The data needs to be sorted at the database level before using the dataset for any usecase. Well, this is an attempt to help you think in multiple ways.

Let's consider we have a message key as the timestamp which is generated at the time of creation of the data and the value is the actual message string.

As and when a message is picked up by the consumer, the message is written into HBase with the RowKey as the kafka key and value as the kafka value.

Since, HBase is a sorted map having timestamp as a key will automatically sorts the data in order. Then you can serve the data from HBase for the downstream apps.

In this way you are not loosing the parallelism of kafka. You also have the privilege of processing sorting and performing multiple processing logics on the data at the database level.

Note: Any distributed message broker does not guarantee overall ordering. If you are insisting for that you may need to rethink using another message broker or you need to have single partition in kafka which is not a good idea. Kafka is all about parallelism by increasing partitions or increasing consumer groups.