set spark.streaming.kafka.maxRatePerPartition for createDirectStream

Question

I need to increase the input rate per partition for my application and I have use .set(\"spark.streaming.kafka.maxRatePerPartition\",100) for the config. The st

Answer 1

Property fetches N messages from a partition per seconds. If I have M partition and batch interval is B, then total messages I can see in batch is N * M * B.

There are few things you should verify

1. Is your input rate is >500 for 10s.
2. Is kafka topic is properly partitioned.