apache-kafka-connect

问题 Currently we are using a couple of custom connetor plugins for our confluent kafka connect distributed worker cluster. One thing that bothers me for a long time is that kafka connect writes all logs from all deployed connectors to one File/Stream. This makes debugging an absolute nightmare. Is there a way to let kafka connect log the connectors in different Files/Streams? Via the connect-log4j.properties I am able to let a specific class log to a different File/Stream. But this means that

问题 I am using kafka connect to source data from DB2 to kafka topic and i am configuring sql query to read the data from DB2 , below is query SELECT SEQ_I AS error_id, TRIM(SEND_I) AS sca , to_char(CREATE_TS,'YYYY-MM-DD HH24:MI:SS.FF3') AS create_timestamp, CREATE_TS, TRIM(ERR_MSG) AS error_message , CASE substr(ERR_MSG,1,locate('-',ERR_MSG)-1) WHEN 'WARNING' THEN 'W' WHEN 'SUSPENDED' THEN 'F' END ERROR_TYPE FROM INTCHG_ERROR_DIR WHERE TRAN_I ='503' AND PRCS_N = 'GLOBAL' am using setting

问题 I am using kafka connect to source data from DB2 to kafka topic and i am configuring sql query to read the data from DB2 , below is query SELECT SEQ_I AS error_id, TRIM(SEND_I) AS sca , to_char(CREATE_TS,'YYYY-MM-DD HH24:MI:SS.FF3') AS create_timestamp, CREATE_TS, TRIM(ERR_MSG) AS error_message , CASE substr(ERR_MSG,1,locate('-',ERR_MSG)-1) WHEN 'WARNING' THEN 'W' WHEN 'SUSPENDED' THEN 'F' END ERROR_TYPE FROM INTCHG_ERROR_DIR WHERE TRAN_I ='503' AND PRCS_N = 'GLOBAL' am using setting

问题 I am playing around with the Kafka Connect JDBC connector and specifically looking at what the actual format of the data that is put onto the topic is. I have been able to see new inserts and updates to the database, but I have not been able to detect deletes from the database. First: Does the JDBC source support detecting these changes? I can't find documentation one way or another. If it does, what format does it take on the actual topic? 回答1: The Confluent JDBC source connector is able to

问题 I installed kafka confluent oss 4.0 on a fresh linux centos 7 but kafka connect failed to start. Steps to reproduce : - Install Oracle JDK 8 - Copy confluent-4.0.0 folder on opt/confluent-4.0.0 - Run /opt/confluent-4.0.0/confluent start Result : Starting zookeeper zookeeper is [UP] Starting kafka kafka is [UP] Starting schema-registry schema-registry is [UP] Starting kafka-rest kafka-rest is [UP] Starting connect \Kafka Connect failed to start connect is [DOWN] Error Log (connect.stderr) :

问题 I'm getting the below issue while trying to use the confluent for kafka-connect. I'm trying to follow the demo given at https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/ Error: ./bin/confluent load jdbc_source_mysql_foobar_01 -d /tmp/kafka-connect-jdbc-source.json { This CLI is intended for development only, not for production https://docs.confluent.io/current/cli/index.html { "error_code": 400, "message": "Connector configuration is invalid

问题 I am using confluent hdfs sink connector 5.0.0 with kafka 2.0.0 and I need to use ExtractTopic transformation (https://docs.confluent.io/current/connect/transforms/extracttopic.html). My connector works fine but when I add this transformation I get NullPointerException, even on simple data sample with only 2 attributes. ERROR Task hive-table-test-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerSinkTask:482) java.lang.NullPointerException at io.confluent

问题 How do I add Confluent Control Center Interceptor to an existing S3(Sink) Connector? To monitor the Sink. I am looking for documentation. Any help is appreciated. 回答1: To be absolutely clear, you need interceptors on your sink and source . If you don't, you can't monitor your pipelines with Confluent Control Center as it stands today. To enable interceptors in Kafka Connect, add to the worker properties file: consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor

问题 I want to make real-time data pipeline in Apache Kafka. I have database which is located at remote location and that database continuously updating. Can anybody which Kafka connect API i should use to pull the data from database and ingest into Kafka broker in real time? later on i would use kafka stream and KSQL to run ad-hoc queries to perform the metrics. Any help would be highly appreciated! 回答1: If you want to create a real-time data pipeline you need to use a Change Data Capture (CDC)

问题 I am trying to configure Sink Kafka Connect for Google Cloud PubSub Service. Using following command to configure Kafka Connect: curl -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{ "name": "pubsub_test", "config": { "connector.class": "com.google.pubsub.kafka.sink.CloudPubSubSinkConnector", "tasks.max": "1", "topics": "kafka_test_topic", "cps.topic": "cps_test_topic", "cps.project": "cps_test_project" } }' http://localhost:8083/connectors In status, I have a