amazon-kinesis

问题 I am learning the AWS services for a use case. After going through the docs I came came up with the a simple flow. I want to ingest data into the Kinesis streams by using the Streams API and the KPL. I use the example putRecord method to ingest data to the streams. I am ingesting the this JSON to the stream - {"userid":1234,"username":"jDoe","firstname":"John","lastname":"Doe"} Once the data is ingested i get the following response in putRecordResult - Put Result :{ShardId: shardId

问题 So I have a stream going on in KVS. I am trying to save it (in chunks of 1 min for example) However, when saving the botocore.response.StreamingBody , I am getting only 1 sec video. Here is the code I am using: client = boto3.client('kinesis-video-media', endpoint_url=url) response = client.get_media( StreamARN=MyARN, StartSelector={ 'StartSelectorType': 'EARLIEST', }) with open('test.webm', 'w+') as f: chunk = response['Payload'].read(1024*8) while chunk: f.write(chunk) chunk = response[

问题 I am trying to connect to Amazon Kinesis Stream from Google Dataproc but am only getting Empty RDDs. Command: spark-submit --verbose --packages org.apache.spark:spark-streaming-kinesis-asl_2.10:1.6.2 demo_kinesis_streaming.py --awsAccessKeyId XXXXX --awsSecretKey XXXX Detailed Log: https://gist.github.com/sshrestha-datalicious/e3fc8ebb4916f27735a97e9fcc42136c More Details Spark 1.6.1 Hadoop 2.7.2 Assembly Used: /usr/lib/spark/lib/spark-assembly-1.6.1-hadoop2.7.2.jar Surprisingly that works

问题 I'm trying to use Apache Flink with AWS kinesis. The document says that I have to build the connector on my own. Therefore, I build the connector and added the jar file for my project and also, I put the dependency on my pom.xml file. <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kinesis_2.11</artifactId> <version>1.6.1</version> </dependency> However, when I tried to build using mvn clean package I got an error message like this [INFO] -----------------------<

问题 I have a kinesis data stream that delivers data in this format: created_at: timestamp payload: varchar(6000) a simplified example of a payload element { "version": 2.0, "data": { "whatever": "someString", "observations": [{ "obs_id": 1, "locaiton": { "lat": 10.000, "lng": 20.000 } }, { "obs_id": 2, "locaiton": { "lat": 10.0001, "lng": 20.0001 } }] } } In real time, the array data.observations in column payload is usually between 0 and 200 elements long. I'm trying to expand data in payload ,

问题 I know data records in Kinesis Stream will be deleted automatically in 24 hrs. But in my application when ever I write some data into stream, for the 2nd time if I want to write some other data, Data inserted first should be deleted. Please anyone help me since I am new to using AWS Kinesis Stream...I didn't get any help from Kinesis Service API... 回答1: You can not delete previously inserted data from stream, but you can read data using KCL. KCL will create checkpoint every after one data

问题 I am trying to run spark streaming job on EMR with Kinesis. Spark 1.6.1 with Kinesis ASL 1.6.1. Writing a plain sample wordcount example. <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming-kinesis-asl_2.10</artifactId> <version>1.6.1</version> </dependency> <dependency> <groupId>com.amazonaws</groupId> <artifactId>amazon-kinesis-client</artifactId> <version>1.6.3</version> </dependency> <dependency> <groupId>com.amazonaws</groupId> <artifactId>amazon-kinesis-producer

问题 I ingested data to (Kinesis Video Stream) KVS via AWS Connect service now using GetMedia API am able to extract the Payload but how can I convert this output to a mp3/wav ? I want to ingest this output to AWS Transcribe service to get text format of audio call ingested by AWS Connect service to KVS. Output of Payload for below code is like : 00#AWS_KINESISVIDEO_CONTINUATION_TOKEND\x87....\x1faudio/L16;rate=8000;channels=1;\x12T\xc......00"AWS_KINESISVIDEO_MILLIS_BEHIND_NOWD\x87\x10\x00\x00

问题 What I'm trying to do: Ultimately: I want to populate an AWS Kinesis stream from a browser extension (Safari, Chrome). I need to send the request to AWS using a signing process (v4); this involves setting headers and encrypting them (on a distant server with the aws secret key) to finally join those to the request. Amazon requests the header "Host" to be explicitly defined… However Javascript strictly disallow setting it (and a bunch of others, for good reasons) I must be missing something

问题 I want to consume POJOs arriving from Kinesis with Flink. Is there any standard for how to correctly send and deserialize the messages? Thanks 回答1: I resolved it with: DataStream<SamplePojo> kinesis = see.addSource(new FlinkKinesisConsumer<>( "my-stream", new POJODeserializationSchema(), kinesisConsumerConfig)); and public class POJODeserializationSchema extends AbstractDeserializationSchema<SamplePojo> { private ObjectMapper mapper; @Override public SamplePojo deserialize(byte[] message)