Shard [shardId-000000000000] is not closed. This can happen if we constructed the list of shards while a reshard operation was in progress

Question

I am getting this error while fetching data from Amazon kinesis Stream. I am doing below steps

1. creating amazon kinesis Steam
2. put the data using putRecord api of AmazonKinesisClient.
3. Then using Worker Of KCL library to get the data from stream.

Answer 1

There are a few possibilities.

1. After you ordered to create the stream, did you wait long enough for completion? Sometimes, it may took 10 minutes for a shard to be created. Since you managed to use putRecord method, the stream and shard should be active.
2. Did you configure the DynamoDB correctly? I assume you are using it for your Kinesis Application synchronization. See inside the records of the DynamoDB table. Are all columns (ie. leaseKey, checkpoint, leaseOwner, parentShardId etc.) correctly filled? Maybe the data in the database is old, so you may be accessing a closed shard?

The exception you mention is here: https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/ShardSyncer.java#L193

Try deleting the DynamoDB table and let the Kinesis Application (KCL) create a brand new one. Link: https://console.aws.amazon.com/dynamodb/