amazon-kcl

问题 I have multiple instances of EC2 running for a same microservice, which has a Kinesis consumer running(with KCL). My question is, when Kinesis stream gets a new event, since all consumers are polling, will the same event be consumed by consumers of all instances? 回答1: The event will be consumed only by one consumer 回答2: KCL is designed so that each shard is processed by only one worker - the built-in lease mechanism is the key to providing this functionality. While under normal circumstances

问题 I have multiple instances of EC2 running for a same microservice, which has a Kinesis consumer running(with KCL). My question is, when Kinesis stream gets a new event, since all consumers are polling, will the same event be consumed by consumers of all instances? 回答1: The event will be consumed only by one consumer 回答2: KCL is designed so that each shard is processed by only one worker - the built-in lease mechanism is the key to providing this functionality. While under normal circumstances

问题 I have multiple instances of EC2 running for a same microservice, which has a Kinesis consumer running(with KCL). My question is, when Kinesis stream gets a new event, since all consumers are polling, will the same event be consumed by consumers of all instances? 回答1: The event will be consumed only by one consumer 回答2: KCL is designed so that each shard is processed by only one worker - the built-in lease mechanism is the key to providing this functionality. While under normal circumstances

问题 I have a streaming pipeline developed in Apache Beam (using Spark Runner) which reads from kinesis stream. I am looking out for options in Apache Beam to manage kinesis checkpointing (i.e. stores periodically the current position of kinesis stream) so as it allows the system to recover from failures and continue processing where the stream left off. Is there a provision available for Apache Beam to support kinesis checkpointing as similar to Spark Streaming (Reference link - https://spark

问题 I'm working with AWS DynamoStream which his API is based on the AWS KCL. In cases I received records which I failed to process and I want those records to be available later to allow reprocessing of them. For instance I'm trying to save them to a remote DB and I experience network issues sometime. My questions are: Can I use the Checkpointer in some way to indicate I Didn't handled the records? Should I just avoid executing Checkpointer.checkpoint()? will it have any effect if I still use it

问题 I am currently writing the simple Kinesis Client Library (KCL) in Golang version. One of the features that I want it for my simple KCL is load balancing shards across multiple record processors and EC2 instances. For example, I have two record processors (which will run in the separate EC2 instance) and four Kinesis shards. The load balancing feature will allow each record processors to process two Kinesis shards . I read that Java KCL implemented this but I can't find the implementation in

问题 I'm working on a micro-services monitoring app. My app supposes to update a GUI accordingly when receiving a new consumed record, meaning: When I receive a new record: 1)I check if the request it represents is a part of a legal flow, and if that flow already has representation in the GUI. By representation, I mean a set of circles that represent the full flow. For example, if I get a transaction (MS1 received request) a legal flow num 1: that is MS1 to MS2 to MS3, so my GUI will add a table

问题 I am trying to use the new feature of KCL library in Java for AWS Kinesis to do a graceful shutdown by registering with shutdown hook to stop all the record processors and then the worker gracefully. The new library provides a new interface which record processors needs to be implemented. But how does it get invoked? Tried invoking first the worker.requestShutdown() then worker.shutdown() and it works. But is it any intended way to use it. What is the use then to use both, and its benefit?

问题 I can't find in the formal documentation of AWS Kinesis any explicit reference between TRIM_HORIZON and the checkpoint, and also any reference between LATEST and the checkpoint. Can you confirm my theory: TRIM_HORIZON - In case the application-name is new, then I will read all the records available in the stream. Else, application-name was already used, then I will read from my last checkpoint. LATEST - In case the application-name is new, then I will read all the records in the stream which