apache-kafka-connect

问题 I am trying to read 2 kafka topics using JDBC sink connector and upsert into 2 Oracle tables which I manually created it. Each table has 1 primary key I want to use it in upsert mode. Connector works fine if I use only for 1 topic and only 1 field in pk.fields but if I enter multiple columns in pk.fields one from each table it fails to recognize the schema. Am I missing any thing please suggest. name=oracle_sink_prod connector.class=io.confluent.connect.jdbc.JdbcSinkConnector tasks.max=1

问题 I am trying to send the data in a batch to a NOSQL database using Kafka Sink Connector. I am following https://kafka.apache.org/documentation/#connect documentation and confused about where the logic of sending records has to be implemented. Please help me in understanding how the records are processed internally and what has to be used Put() or Flush() to process the records in a batch. 回答1: When a Kafka Connect worker is running a sink task, it will consume messages from the topic partition

问题 I am running Apache Kafka on my Windows machine with two Kafka-Connect-Workers(Port 8083, 8084) and three partitions(replication of one). My issue is that I am able to see the fail-over to other Kafka-Connect worker whenever I shutdown one of them, but load balancing is not happening because the number of tasks is always ONE. I am using Official MongoDB-Kafka-Connector as Source(ChangeStream) with tasks.max=6. I tried updating MongoDB with multiple threads so that it could push more data into

问题 Kafka JDBC Sink Connector gives a Null Pointer Exception for a message with schema having an optional field here 'parentId'. Have I missed anything? I am using out of the box JSONConverter and JDBC Sink Connector A message on Kafka topic is { "schema":{ "type":"struct", "fields":[ { "field":"id", "type":"string" }, { "field":"type", "type":"string" }, { "field":"eventId", "type":"string" }, { "field":"parentId", "type":"string", "optional":true }, { "field":"created", "type":"int64", "name":

问题 I need someone's help. I follow a tutorial Streaming Data from MySQL into Kafka with Kafka Connect and Debezium but I have the trouble connecting MySql to Kafka server using Debezium MySQL Connector. Here is my setup and other information. OS : Windows 10. Kafka Connect : Confluent 5.0. MySQL Connector : 0.8.1 Final. I keep files . I add the plugins path When I tried connecting to Kafka Connect, It returned me an error. "Failed to find any class that implements Connector and which name

问题 I'm trying to use the Confluent InfluxDB Sink Connector to get data from a kafka topic into my InfluxDB. Firstly, I transmit data to kafka topic from a log file by using nifi, and it works well. The kafka topic get the data, like below: { "topic": "testDB5", "key": null, "value": { "timestamp": "2019-03-20 01:24:29,461", "measurement": "INFO", "thread": "NiFi Web Server-795", "class": "org.apache.nifi.web.filter.RequestLogger", "message": "Attempting request for (anonymous) }, "partition": 0,

问题 Occasionally with Kafka Connect, I see my JdbcSourceConnector's task go up and down--that is, the REST interface sometimes reports one task that is RUNNING and sometimes reports no tasks (the connector remains RUNNING this whole time). During these times, the task seems to be working when its running. Then, if I delete and re-create the connector, the problem seems to go away. I suspect something is wrong--that tasks shouldn't churn like this, right? But the INFO/WARN logs on the server don't

问题 Occasionally with Kafka Connect, I see my JdbcSourceConnector's task go up and down--that is, the REST interface sometimes reports one task that is RUNNING and sometimes reports no tasks (the connector remains RUNNING this whole time). During these times, the task seems to be working when its running. Then, if I delete and re-create the connector, the problem seems to go away. I suspect something is wrong--that tasks shouldn't churn like this, right? But the INFO/WARN logs on the server don't

问题 I've been using KSQL and so far it's been working great. But now I'd like to sink the output to BigQuery via Kafka Connect, and need to attach a JSON schema. I'm having trouble figuring out how to do this. Here's my query: CREATE STREAM tweets_original ( CreatedAt BIGINT, Id BIGINT, Text VARCHAR, Source VARCHAR, GeoLocation VARCHAR, User STRUCT<Id BIGINT, Name VARCHAR, Description VARCHAR, ScreenName VARCHAR, URL VARCHAR, FollowersCount BIGINT, FriendsCount BIGINT> ) WITH (kafka_topic='tweets

问题 I am trying to setup kafka-connect-cassandra on an AWS instance. I have setup plugin.path in connect-avro-distributed.properties file: plugin.path=/home/ubuntu/kafka_2.11-1.0.0/plugins And I have kafka-connect-cassandra-1.0.0-1.0.0-all.jar in: /home/ubuntu/kafka_2.11-1.0.0/plugins/lib This is the traceback: [2018-02-18 10:28:33,268] INFO Kafka Connect distributed worker initializing ... (org.apache.kafka.connect.cli.ConnectDistributed:60) [2018-02-18 10:28:33,278] INFO WorkerInfo values: jvm