apache-kafka-connect

问题 I have installed and tested kafka connect in distributed mode, it works now and it connects to the configured sink and reads from the configured source. That being the case, I moved to enhance my installation. The one area I think needs immediate attention is the fact that to create a connector, the only available mean is through REST calls, this means I need to send my information through the wire, unprotected. In order to secure this, kafka introduced the new ConfigProvider seen here. This

问题 I try to start JDBC sink connector with following configuration: { "name": "crm_data-sink_hh", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "tasks.max": 6, "topics": "crm_account,crm_competitor,crm_event,crm_event_participation", "connection.url": "jdbc:postgresql://db_host/hh?prepareThreshold=0", "connection.user": "db_user", "connection.password": "${file:db_hh_kafka_connect_pass}", "dialect.name": "PostgreSqlDatabaseDialect", "insert.mode": "upsert", "pk

问题 I am using a 3rd party CDC tool that replicates data from a source database into Kafka topics. An example row is shown below: { "data":{ "USER_ID":{ "string":"1" }, "USER_CATEGORY":{ "string":"A" } }, "beforeData":{ "Data":{ "USER_ID":{ "string":"1" }, "USER_CATEGORY":{ "string":"B" } } }, "headers":{ "operation":"UPDATE", "timestamp":"2018-05-03T13:53:43.000" } } What configuration is needed in the sink file in order to extract all the (sub)fields under data and headers and ignore those

问题 I have downloaded confluent version 5.4 . And i have connector running good in stand alone mode . wget /usr/local http://packages.confluent.io/archive/5.4/confluent-5.4.0-2.11.tar.gz -P ~/Downloads/ tar -zxvf ~/Downloads/confluent-5.4.0-2.11.tar.gz -C ~/Downloads/ sudo mv ~/Downloads/confluent-5.4.0 /usr/local/ and then i am trying to run Kafak connect on distributed mode so i modified my connect-distributed.properties like below bootstrap.servers=b-***.eu-west-1.amazonaws.com:9092,b-***.eu

问题 My task is transfer data from Postgres table to KSQL table (for future joins with streams). Let's imagine table has three records: id | name | description ------------------------- 1 | name1 | description1 2 | name2 | description2 3 | name3 | description3 It is easy to do by means of Kafka JdbcSourceConnector. But there is one little problem - data in table may be changed. Changes must be in KTable too. According to documentation there is no way to track changes except bulk mode. But bulk

问题 I am having two issues. 1) I enabled the delete.enabled as true . But since it needs the pk.mode to be set as record_key I am not able to use the composite key of my MySQL table as I could use when I set pk.mode as record_value . I tried to add a composite key in the Stream while converting the records into AVRO format using the partition by option. But it added a new column in the MySQL table. The row is not set if I am using a single primary key instead of a composite key. 2) The delete

问题 Question: How do I correctly register the SqlServer connector with Kafka Connect to connect to a standalone SQL Server Instance? Note: I am NOT running SQL Server in Docker. Error: Error: Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host 127.0.0.1, port 1433 has failed. Error: "Connection refused (Connection refused). Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at

问题 This question follows this one. The main task is to make joins on KSQL side. Example below will illustrate it. Incidents messages arrive In Kafka topic. The structure of that messages: [ { "name": "from_ts", "type": "bigint" }, { "name": "to_ts", "type": "bigint" }, { "name": "rulenode_id", "type": "int" } ] And there is a Postgres table rulenode : id | name | description Data from both sources need to be joined by fields rulenode_id = rulenode.id so as to get single record with fields from

问题 We can update/upsert the record in mongodb BUT is there is any method or function from which we can update or upsert the document directly in mongodb and the source system is kafka and destination is mongodb. 回答1: Yes we can update/upsert the data. For update you have to define a parameter in Kafka connector. and whitelist the column on which bases you want to update the record. The property is as followed: document.id.strategy=com.mongodb.kafka.connect.sink.processor.id.strategy

问题 We can update/upsert the record in mongodb BUT is there is any method or function from which we can update or upsert the document directly in mongodb and the source system is kafka and destination is mongodb. 回答1: Yes we can update/upsert the data. For update you have to define a parameter in Kafka connector. and whitelist the column on which bases you want to update the record. The property is as followed: document.id.strategy=com.mongodb.kafka.connect.sink.processor.id.strategy