问题
Whenever my consumer requests a new batch from Kafka, it is requesting always 1MB of data, then it seems to request the next 1MB, and so forth. Does anybody know what the configuration and programming steps are to to receive batches of 20MB?
回答1:
You can set the property max.partition.fetch.bytes in the consumer properties to the value you desire (default is 1MB).
Also, this value must be equal or greater than max.message.size property in the broker configuration to be sure that your consumers will be able to read all messages accepted by the broker.
Finally, if processing 20MB takes too long, you may want to increase your session.timeout.ms setting at the consumer (defaults to 3 seconds), to avoid the broker thinking your consumer is dead and triggering a rebalance.
来源：https://stackoverflow.com/questions/40139635/how-can-i-make-consumer-request-more-than-1mb-records-from-kafka