Same Kinesis Consumer running on multiple EC2 instances

时光毁灭记忆、已成空白 提交于 2021-02-10 19:09:05

问题


I have multiple instances of EC2 running for a same microservice, which has a Kinesis consumer running(with KCL).

My question is, when Kinesis stream gets a new event, since all consumers are polling, will the same event be consumed by consumers of all instances?


回答1:


The event will be consumed only by one consumer




回答2:


KCL is designed so that each shard is processed by only one worker - the built-in lease mechanism is the key to providing this functionality.

While under normal circumstances each consumer will only receive messages for the shards it owns the lease for, there are some edge cases (typically caused by when connections timeout/break or hosts die/stop responding) worth considering that can result in double-processing of a message from the stream.

As a concrete example: If a worker fails to renew a lease (typically caused by a connectivity issue between the worker host and DynamoDB), it's possible another worker will acquire that lease and start processing while the original worker is still doing work



来源:https://stackoverflow.com/questions/50923206/same-kinesis-consumer-running-on-multiple-ec2-instances

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!