aws-msk | 易学教程

Problems with Amazon MSK default configuration and publishing with transactions

阅读更多关于 Problems with Amazon MSK default configuration and publishing with transactions

问题 Recently we have started doing some testing of our Kafka connectors to MSK, Amazon's managed Kafka service. Publishing records seem to work fine however not when transactions are enabled. Our cluster consists of 2 brokers (because we have 2 zones) using the default MSK configuration. We are creating our Java Kafka producer using the following properties: bootstrap.servers=x.us-east-1.amazonaws.com:9094,y.us-east-1.amazonaws.com:9094 client.id=kafkautil max.block.ms=5000 request.timeout.ms

Kafka Streams EOS Mode - Informed to Shutdown

阅读更多关于 Kafka Streams EOS Mode - Informed to Shutdown

问题 I have a Kafka Streams application, which shuts down without any proper logging even at the debug level - 2020-12-18 14:25:36:875 +0000 [Thread-7] INFO o.apache.kafka.streams.KafkaStreams:? - stream-client [trinity-client-pandprat-estestes5-null-b9346744-6bb4-464d-aeaa-9311ab16ce2c] State transition from REBALANCING to PENDING_SHUTDOWN 2020-12-18 14:25:36:973 +0000 [kafka-streams-close-thread] INFO o.a.k.s.p.internals.StreamThread:? - stream-thread [trinity-client-pandprat-estestes5-null

Kafka connect setup to send record from Aurora using AWS MSK

阅读更多关于 Kafka connect setup to send record from Aurora using AWS MSK

问题 I have to send records from Aurora/Mysql to MSK and from there to Elastic search service Aurora -->Kafka-connect--->AWS MSK--->kafka connect --->Elastic search The record in Aurora table structure is something like this I think record will go to AWS MSK in this format. "o36347-5d17-136a-9749-Oe46464",0,"NEW_CASE","WRLDCHK","o36347-5d17-136a-9749-Oe46464","<?xml version=""1.0"" encoding=""UTF-8"" standalone=""yes""?><caseCreatedPayload><batchDetails/>","CASE",08-JUL-17 10.02.32.217000000 PM,

Failed to find any class that implements Connector and which name matches io.confluent.connect.elasticsearch.ElasticsearchSinkConnector

阅读更多关于 Failed to find any class that implements Connector and which name matches io.confluent.connect.elasticsearch.ElasticsearchSinkConnector

问题 I have MSK running on aws and i am able to send records in and out from MSK . I just wanted to use Kafka connect so that records coming into MSK will go to Elastic Search . I have done below things but i am not sure if my connector is working properly or not ,Because i can not see records into Elastic Search This is the records that i am sending { "data": { "RequestID": 517082653, "ContentTypeID": 9, "OrgID": 16145, "UserID": 4, "PromotionStartDateTime": "2019-12-14T16:06:21Z",

Amazon Managed Streaming for Kafka- MSK features and performance

阅读更多关于 Amazon Managed Streaming for Kafka- MSK features and performance

I'm evaluating AWS Managed Service Kafka (MSK) and I know that currently, it is in preview mode, hence might not have all features or proper documentation. I tried setting up the msk cluster and was validating whether msk could fulfill all the use case/requirement of our company, but currently, it lacks documentation and example. https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html I have the following queries: i) How can I access AWS MSK with the Kafka clients running on my on-premise system? ii) Does MSK support schema evolution and exactly once semantics? iii) Will MSK

Amazon Managed Streaming for Kafka- MSK features and performance

阅读更多关于 Amazon Managed Streaming for Kafka- MSK features and performance

问题 I'm evaluating AWS Managed Service Kafka (MSK) and I know that currently, it is in preview mode, hence might not have all features or proper documentation. I tried setting up the msk cluster and was validating whether msk could fulfill all the use case/requirement of our company, but currently, it lacks documentation and example. https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html I have the following queries: i) How can I access AWS MSK with the Kafka clients running on my