Structured Streaming Kafka Source Offset Storage
问题 I am using the Structured Streaming source for Kafka (Integration guide), which as stated does not commit any offset. One of my goals is to monitor it (check if its lagging behind etc). Even though it does not commit the offsets it handles them by querying kafka from time to time and checking which is the next one to process. According to the documentation the offsets are written to HDFS so in case of failure it can be recovered, but the question is: Where are they being stored? Is there any