apache-kafka-connect

问题 I'm trying to configure SSL for Kafka Connect REST API (2.11-2.1.0). The problem I tried two configurations (worker config): with listeners.https. prefix listeners=https://localhost:9000 listeners.https.ssl.keystore.location=/mypath/keystore.jks listeners.https.ssl.keystore.password=mypassword listeners.https.ssl.key.password=mypassword and without listeners.https. prefix listeners=https://localhost:9000 ssl.keystore.location=/mypath/keystore.jks ssl.keystore.password=mypassword ssl.key

问题 I have a running instance of Kafka 0.10 and I'm currently using Gobblin to store data into HDFS. I want to switch to Kafka Connect, and as I researched I found that Confluent provide a connector. However, is there a way to use this connector without using the entire Confluent platform? Meaning can I for example copy the relevant scripts from Confluent source and somehow make my Kafka instance use it? I'm basically still learning my way through this stuff so I'm not yet very well versed in

问题 Using Debezium 0.7 to read from MySQL but getting flush timeout and OutOfMemoryError errors in the initial snapshot phase. Looking at the logs below it seems like the connector is trying to write too many messages in one go: WorkerSourceTask{id=accounts-connector-0} flushing 143706 outstanding messages for offset commit [org.apache.kafka.connect.runtime.WorkerSourceTask] WorkerSourceTask{id=accounts-connector-0} Committing offsets [org.apache.kafka.connect.runtime.WorkerSourceTask] Exception

问题 I am going through kafka connect, and i am trying to get the concepts. Let us say I have kafka cluster (nodes k1, k2 and k3) setup and it is running, now i want to run kafka connect workers in different nodes say c1 and c2 in distributed mode. Few questions. 1) To run or launch kafka connect in distributed mode I need to use command ../bin/connect-distributed.sh , which is available in kakfa cluster nodes, so I need to launch kafka connect from any one of the kafka cluster nodes? or any node

问题 I am trying to read kafka data from flink and as I am new to kafka and flink, I don't know how to connect them. 回答1: Flink provides Kafka connector. In order read data from Kafka topics, first you need add Flink -Kafka connector dependency. <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka-0.8_2.10</artifactId> <version>1.1.3</version> </dependency> Next you simply invoke Streaming execution environment and add Kafka source. Here is a sample Properties

问题 I have a Kafka connector with the following code for the poll() method in the SourceTask implementation. @Override public List<SourceRecord> poll() throws InterruptedException { SomeType item = mQueue.take(); List<SourceRecord> records = new ArrayList<>(); SourceRecord[] sourceRecords = new SourceRecord[]{ new SourceRecord(null, null, "data", null, Schema.STRING_SCHEMA, "foo", Schema.STRING_SCHEMA, "bar") }; Collections.addAll(records, sourceRecords); return records; } If I attach a consumer

问题 I am using kafka_2.11-0.10.2.1 and the pubsub connector provided by google here. All I care to do is push data from a Kafka Topic to a PubSub one using a standalone connector. I followed all steps as I should have: Produced the cps-kafka-connector.jar Added the cps-sink-connector.properties file in kafka's config directory. The file looks like this: name=CPSConnector connector.class=com.google.pubsub.kafka.sink.CloudPubSubSinkConnector tasks.max=10 topics=kafka_topic cps.topic=pubsub_topic

问题 Produce to/Consume from Kafka in JSON. Save to HDFS in JSON using below properties : key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=false value.converter.schemas.enable=false Producer : curl -X POST -H "Content-Type: application/vnd.kafka.json.v1+json" \ --data '{"schema": {"type": "boolean", "optional": false, "name": "bool", "version": 2, "doc": "the documentation", "parameters": {"foo": "bar

问题 I was following step #7 (Use Kafka Connect to import/export data) at this link: http://kafka.apache.org/documentation.html#quickstart It was working well until I deleted the 'test.txt' file. Mainly because that's how log4j files would work. After certain time, the file will get rotated - I mean - it will be renamed & a new file with the same name will start getting written to. But after, I deleted 'test.txt', the connector stopped working. I restarted connector, broker, zookeeper etc, but the

问题 I get task error for a distributed configuration for a connector sink cassandra. I was running the command : curl -s localhost:8083/connectors/cassandraSinkConnector2/status | jq to get the status { "name": "cassandraSinkConnector2", "connector": { "state": "RUNNING", "worker_id": localhost:8083" }, "tasks": [ { "id": 0, "state": "FAILED", "worker_id": "localhost:8083", "trace": "org.apache.kafka.common.KafkaException: Failed to construct kafka consumer\n\tat org.apache.kafka.clients.consumer