Kafka Consumer - JMX Properties

问题

I enabled JMX on the kafka brokers on port 8081. When I view the MBean properties in jConsole, I only see the following for kafka.consumer-

kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchRequestRateAndTimeMs,clientId=ReplicaFetcherThread-2-413

kafka.consumer:type=FetchRequestAndResponseMetrics,name=FetchResponseSize,clientId=ReplicaFetcherThread-0-413

But none of the other ones that are identified in here under Kafka Consumer Metrics are emitted by JMX.

Kafka Version # 0.8.2.1

I am specifically interested in - kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=([-.\w]+)

Any thoughts?

回答1:

The JMX PORT you are listening is the broker port. But the Mbean of kafka.consumer: is the consumer jvm metrics. So if you have another JVM that is consume a topic, you can see kafka.consumer Mbeans.

回答2:

ConsumerLag is an overloaded term in Kafka, it can stand for:

Consumer's metric: Calculated difference between a consumer's current log offset and a producer’s current log offset. You can find it under JMX bean, if you're using Java/Scala based consumer (e.g. pykafka consumer doesn't export metrics):
- kafka v0.8.2.x:
```
kafka.consumer:type= ConsumerFetcherManager, name=MaxLag, clientId=([-.\w]+) 
```
- kafka v0.9+:
```
kafka.consumer:type=consumer-fetch-manager-metrics,client-id=([-.w]+)
```
Consumer lag used to be stored in ZooKeeper (Kafka <= v0.8), newer versions of Kafka have special topic __consumer_offsets that stores each consumer's lag. There are tools (e.g. kafka-manager) that can compute lag by consuming messages from this topic and calculating lag. In kafka-manager you have to enable this feature for each cluster:

[ ] Poll consumer information (Not recommended for large # of consumers)
Broker's metric: Represent the offset differences between partition leaders and their followers. You can find this metric under JMX bean:
```
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+),topic=([-.\w]+),partition=([0-9]+)
```

回答3:

This may help to find it for 0.8, but I am currently running a Kafka 0.10 broker and consumer. When using a console consumer, I pointed jconsole to this consumer and found on the MBeans TAB: kafka.consumer-> consumer-fetcher-manager-metric -> consumer-1 -> Attributes -> records-max-lag.

来源：https://stackoverflow.com/questions/37799564/kafka-consumer-jmx-properties

标签

apache-kafka

jmx