amazon-kinesis

问题 Context : I'm not necessarily referring to a KCL-based application, just pure Kinesis API calls. Does the using the TRIM_HORIZON shard iterator type immediately give you the earliest published record in the stream (ie earliest available within Kinesis' built-in 24hr window), or simply an iterator/cursor for some time period as much as 24 hours ago, that you must then use to advance along the stream until you hit the earliest published record? Put another way, in case that's not quite clear...

问题 Amazon claims their Kinesis streaming product guarantees record ordering. It provides ordering of records, as well as the ability to read and/or replay records in the same order (...) Kinesis is composed of Streams that are themselves composed of one or more Shards. Records are stored in these Shards. We can write consumer applications that connect to a Shard and read/replay records in the order they were stored. But can Kinesis guarantee, out of the box, ordering for the Stream itself

问题 The title might be a bit confusing. I'll try my best to make it clearer. Suppose I have a AWS Lambda function that has two different Kinesis streams A and B as input event sources. So, for the below, since a KinesisEvent instance contains a batch of records, will the batch contain records from a single stream, or essentially it contain records from both streams A and B? public class ProcessKinesisEvents { public void recordHandler(KinesisEvent event, Context context) { ... } } 回答1: Each

问题 I am trying to send a HTTP Post Request to put a record into Amazon Kinesis Stream. There are several ways (Kinesis Client, KPL, setting up AWS Gateway as Kinesis Proxy). I saw this document about Kinesis PutRecord API http://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecord.html POST / HTTP/1.1 Host: kinesis.<region>.<domain> Content-Length: <PayloadSizeBytes> User-Agent: <UserAgentString> Content-Type: application/x-amz-json-1.1 Authorization: <AuthParams> Connection: Keep-Alive

问题 I am trying to run the JavaKinesisWordCountASL example. The example seem to connect to my Kinesis Stream and gets data from the stream (as shown in the log below). However, Sparks does not invoke the call function passed to the unionStreams.flatMap method in the example and does not prints any wordcount. I have tried running using both Java 8 and Java 7. I am running it on an ubuntu instance. The same example works on my macbook. 14/11/15 01:59:42 INFO scheduler.ReceiverTracker: Stream 1

问题 I read you can have multiple consumer apps per kinesis stream. http://docs.aws.amazon.com/kinesis/latest/dev/developing-consumers-with-kcl.html however, I heard you can only have on consumer per shard. Is this true? I don't find any documentation to support this, and can't imagine how that could be if multiple consumers are reading from the same stream. Certainly, it doesn't mean the producer needs to repeat content in different shards for different consumers. 回答1: Kinesis Client Library

问题 I'm very new to Amazon Kinesis so maybe this is just a problem in my understanding but in the AWS Lambda FAQ it says: The Amazon Kinesis and DynamoDB Streams records sent to your AWS Lambda function are strictly serialized, per shard. This means that if you put two records in the same shard, Lambda guarantees that your Lambda function will be successfully invoked with the first record before it is invoked with the second record. If the invocation for one record times out, is throttled, or

问题 How to feed the audio stream output of the AWS Kinesis video stream (KVS) to AWS Transcribe service and get realtime audio to text conversion? I ingested audio via amazon connect to KVS(Kinesis video stream), My concern is, which format kinesis video stream (KVS) deliver (I hope its Payload format) and will Transcribe service able to support it or any way I can convert to Transcribe service understandable. 回答1: For me, I tinkered with the AWS CLI, two stage process although the output from

问题 I am currently writing the simple Kinesis Client Library (KCL) in Golang version. One of the features that I want it for my simple KCL is load balancing shards across multiple record processors and EC2 instances. For example, I have two record processors (which will run in the separate EC2 instance) and four Kinesis shards. The load balancing feature will allow each record processors to process two Kinesis shards . I read that Java KCL implemented this but I can't find the implementation in

问题 I have the following in the agent.json { "cloudwatch.emitMetrics": true, "kinesis.endpoint": "", "firehose.endpoint": "", "flows": [ { "filePattern": "/home/ec2-user/ETLdata/contracts/Delta.csv", "kinesisStream": "ETL-rawdata-stream", "partitionKeyOption": "RANDOM", "dataProcessingOptions": [ { "optionName": "CSVTOJSON", "customFieldNames": [ "field1", "field2"], "delimiter": "," } ] } ] } When I add the specified file to the folder, literally nothing happens. I only see the below in the logs