Expected behavior for AWS Kinesis ShardIteratorType TRIM_HORIZON

Question

Context: I\'m not necessarily referring to a KCL-based application, just pure Kinesis API calls.

Does the using the TRIM_HORIZON shard iter

Answer 1

TRIM_HORIZON gives the oldest record in the stream.

Just that sometimes on giving TRIM_HORIZON as the shard_iterator_type :-

 Suppose the value of "millis_behind_latest" in the kinesis response is ~86399000 & your stream retention period is 24 hours(86400000) 

By the time you use the shard_iterator to retrieve the record, the record is no longer in the stream as the retention period of the record has been exceeded. Hence you get an empty result because the oldest record has expired and no longer there in the data stream. So the shard_iterator is now pointing to an empty space in the disk.

When such a thing happens take the value of "next_shard_iterator" and use get_records to once again get the kinesis data records.

Also another thing is we do not completely know how AWS manages each shard in the data stream. How data is erased and added into it. Maybe data is not stored in concurrent/contiguous memory memory blocks and hence we get empty results in between retrieval of data.

Keep taking the value of "next_shard_iterator" and use get_records until you get a value of 0 for "millis_behind_latest".

Hope this answer helps. :)