amazon-kinesis

问题 I am writing record to Kinesis Firehose stream that is eventually written to a S3 file by Amazon Kinesis Firehose. My record object looks like ItemPurchase { String personId, String itemId } The data is written to S3 looks like: {"personId":"p-111","itemId":"i-111"}{"personId":"p-222","itemId":"i-222"}{"personId":"p-333","itemId":"i-333"} NO COMMA SEPERATION. NO STARTING BRACKET as in a Json Array [ NO ENDING BRACKET as in a Json Array ] I want to read this data get a list of ItemPurchase

问题 I used python 3.6 and I want to post video stream to aws kinesis with API. I used python aws client to create stream and GetDataEndPoint but when I want to post my data with my custom request ( PutMedia doesn't include in python client actually), I get an error Unable to determine service/operation name to be authorized . I've follow the api doc of aws kinesis video media PutMedia and GetMedia. So I start by getting endpoint with GetDataEndPoint with client method: response = client.get_data

问题 I use aws api gateway integrated with aws lambda(java), but I'm seeing some serious problems in this approach. The concept of removing the server and having your app scaled out of the box is really nice but here are the problem I'm facing. My lambda is doing 2 simple things- validate the payload received from the client and then send it to a kinesis stream for further processing from another lambda(you will ask why I don't send directly to the stream and only use 1 lambda for all of the

问题 I use aws api gateway integrated with aws lambda(java), but I'm seeing some serious problems in this approach. The concept of removing the server and having your app scaled out of the box is really nice but here are the problem I'm facing. My lambda is doing 2 simple things- validate the payload received from the client and then send it to a kinesis stream for further processing from another lambda(you will ask why I don't send directly to the stream and only use 1 lambda for all of the

问题 from __future__ import print_function # Python 2/3 compatibility import boto3 import json import decimal #kinesis = boto3.resource('kinesis', region_name='eu-west-1') client = boto3.client('kinesis') with open("questions.json") as json_file: questions = json.load(json_file) Records = [] count = 0 for question in questions: value1 = question['value'] if value1 is None: value1 = '0' record = { 'StreamName':'LoadtestKinesis', 'Data':b'question','PartitionKey':'value1' } Records.append(record)

问题 I have Amazon Kinesis Streams, which has all the click stream data and we want to write an API to query Kinesis Streams. My plan is to create a API gateway which calls an AWS Lambda function that would query Kinesis Streams and return it back. Is it possible to use Lambda to query Kinesis Streams or should we use Kinesis Analytics and Lambda? 回答1: You can use AWS Kinesis with Lambda via the Stream-based model. AWS Lambda will itself poll the kinesis stream and fetch the number of records

问题 I'm new with Apache Spark Streaming. Trying to build Spark to read value from Kinesis Stream. This is my python script import settings from pyspark import SparkContext from pyspark.streaming import StreamingContext from pyspark.streaming.kinesis import KinesisUtils, InitialPositionInStream spark_context = SparkContext(master="local[2]", appName=settings.KINESIS_APP_NAME) streaming_context = StreamingContext(sparkContext=spark_context, batchDuration=settings.BATCH_DURATION) kinesis_good_stream

问题 I use jmeter to create HTTP request for my webservice REST. Now i want to send this request to Amazon kinesis using PutRecords method but i don't know hot to create the request, in particular how to set Headers fields for Signing and Authenticating in kinesis. Has someone used Rest request? thanks 回答1: As per PutRecords API reference sample request should look as POST / HTTP/1.1 Host: kinesis.<region>.<domain> x-amz-Date: <Date> Authorization: AWS4-HMAC-SHA256 Credential=<Credential>,

问题 I am trying to setup a sync between AWS Aurora and Redshift. What is the best way to achieve this sync? Possible ways to sync can be: - Query table to find changes in a table(since I am only doing inserts, updates don't matter), export these changes to a flat file in S3 bucket and use Redshift copy command to insert into Redshift. Use python publisher and Boto3 to publish changes into a Kinesis stream and then consume this stream in Firehose from where I can copy directly into Redshift. Use

问题 I am trying to use AWS Kinesis stream for one of our data streams. I would like to monitor pending messages on my stream for ops purposes(scale downstream according to backlog), but unable to find any API that gives (approx) pending messages in my stream. This looks strange as messages get expired after 7 days and if the producers and consumers are isolated and can't communicate, how do you know messages are expiring. How do you handle this problem? Thanks! 回答1: There is no such concept as