apache-kafka-connect

问题 [2017-08-31 10:15:20,715] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:231) [2017-08-31 10:15:20,715] WARN The configuration 'status.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:231) [2017-08-31 10:15:20,715] WARN The configuration 'internal.value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka

问题 I receive a lot of the messages (by http-protocol) per second (50000 - 100000) and want to save them to PostgreSql. I decided to use Kafka JDBC Sink for this purpose. The messages are saved to database by one record, not in batches. I want to insert records in PostgreSQL in batches with size 500-1000 records. I found some answers on this problem in issue: How to use batch.size? I tried to use related options in configuration, but it seems that they no have any effect. My Kafka JDBC Sink

问题 I receive a lot of the messages (by http-protocol) per second (50000 - 100000) and want to save them to PostgreSql. I decided to use Kafka JDBC Sink for this purpose. The messages are saved to database by one record, not in batches. I want to insert records in PostgreSQL in batches with size 500-1000 records. I found some answers on this problem in issue: How to use batch.size? I tried to use related options in configuration, but it seems that they no have any effect. My Kafka JDBC Sink

问题 I'd like to read from a multiple topics from cdc debezium from source postgres database, using a key from kafka message holding a primary keys. Then, the connector performs ETL operations in source database. When I set delete.enabled to true I cannot use kafka primary keys, it says I have to specify record_key and pk_fields . My idea is, set regex to read multiple desired topics, get table name from topic name and use primary keys holding by kafka topic, which is being currently read. name

问题 I use Spring KafKa anf I set ackMode to MANUAL_IMMEDIATE props.setAckMode(AbstractMessageListenerContainer.AckMode.MANUAL_IMMEDIATE); the scenario is that for some reason my app could not acknowledge ( acknowledgment.acknowledge() ) and just miss it without exception. 1- How can I set consumer retry for missed message 2- How configure to call a function after max retry count that I configured reached 回答1: See the documentation about SeekToCurrentErrorHandlers. When the listener throws an

问题 From https://docs.confluent.io/3.1.1/connect/connect-jdbc/docs/sink_config_options.html#jdbc-sink-configuration-options Specifies how many records to attempt to batch together for insertion into the destination table, when possible. Type: int Default: 3000 Valid Values: [0,…] Importance: medium So, this is from Confluent site. Importance is medium, default is 3000. What if I want the KAFKA changes every 30 secs even if there are say, only 27 KAFKA messages for the topic? What is default

问题 I'm trying to secure the REST endpoint in kafka connect with https. Below is my config in connect.distributed.properties file, I created self-signed certificate in my Linux VM, listeners=https://myhostname.xxxxx.xx.com:8085 listeners.https.ssl.keystore.location=/home/kafka/server.keystore.jks listeners.https.ssl.keystore.password=******* listeners.https.ssl.key.password=****** listeners.https.ssl.truststore.location=/home/kafka/server.truststore.jks listeners.https.ssl.truststore.password=***

问题 I setup ACL for 3 node Kafka cluster and able to send and receive for a topic through producer console and consumer console. Now I want to configure Kafka connect with ACL. I tried with SASL_PLAINTEXT combinations and in connect.log file it shows the following error. it is not syncing to from source table to topic, please help where I am missing any configuration. error log [2020-10-14 07:24:35,874] ERROR WorkerSourceTask{id=oracle-jdbc-source-mtx_domains_acl5-0} Failed to flush, timed out

问题 We are currently trying to use Debezium for capturing changes of 4 tables in a Postgres database. We are currently aware that for this use case we could use a kafka-streams app to join/aggregate tables for a KTable, however we want to keep kafka-stream topology simple, so the idea would be to use a Materialized View from Postgres and capture it changes. Is it possible to do this, if so than how should we configure it? Kafka Connect Source Configuration: { "name": "campaign-db-source-connector

问题 I would love to setup a cluster of JDBC Kafka Connectors and configure them to pull from multiple databases running on the same host. I've been looking through the Kafka Connect documentation, and it appears that after you configure the JDBC connector that it can only pull data from a single database. Can anyone confirm this? 回答1: Depending on the mode you start your workers (standalone or distributed) : In the standalone mode , you can start multiple jdbc connectors by using : bin/connect