mongodb-kafka-connector

来源： https://stackoverflow.com/questions/64123986/the-official-mongodb-kafka-source-connector-does-not-publish-clean-extended-json

来源： https://stackoverflow.com/questions/64123986/the-official-mongodb-kafka-source-connector-does-not-publish-clean-extended-json

来源： https://stackoverflow.com/questions/64123986/the-official-mongodb-kafka-source-connector-does-not-publish-clean-extended-json

问题 I am running Apache Kafka on my Windows machine with two Kafka-Connect-Workers(Port 8083, 8084) and three partitions(replication of one). My issue is that I am able to see the fail-over to other Kafka-Connect worker whenever I shutdown one of them, but load balancing is not happening because the number of tasks is always ONE. I am using Official MongoDB-Kafka-Connector as Source(ChangeStream) with tasks.max=6. I tried updating MongoDB with multiple threads so that it could push more data into

问题 I have a requirement where I have to capture MongoDB ChangeStream (inserts/updates etc) events and take some actions (save to OracleDB). I have thought of this design which seems to be good: "MongoDB-ChangeStream" --> "MongoDB Source Connector for Apache Kafka" --> "Kafka Broker-Topic" --> "Java Service" --> OracleDB My question here is, I am using MongoDB-4.0 and "MongoDB Source Connector for Apache Kafka" was introduced in MongoDB-4.2. Can I still use "MongoDB Source Connector for Apache

问题 We can update/upsert the record in mongodb BUT is there is any method or function from which we can update or upsert the document directly in mongodb and the source system is kafka and destination is mongodb. 回答1: Yes we can update/upsert the data. For update you have to define a parameter in Kafka connector. and whitelist the column on which bases you want to update the record. The property is as followed: document.id.strategy=com.mongodb.kafka.connect.sink.processor.id.strategy

问题 We can update/upsert the record in mongodb BUT is there is any method or function from which we can update or upsert the document directly in mongodb and the source system is kafka and destination is mongodb. 回答1: Yes we can update/upsert the data. For update you have to define a parameter in Kafka connector. and whitelist the column on which bases you want to update the record. The property is as followed: document.id.strategy=com.mongodb.kafka.connect.sink.processor.id.strategy

问题 We can update/upsert the record in mongodb BUT is there is any method or function from which we can update or upsert the document directly in mongodb and the source system is kafka and destination is mongodb. 回答1: Yes we can update/upsert the data. For update you have to define a parameter in Kafka connector. and whitelist the column on which bases you want to update the record. The property is as followed: document.id.strategy=com.mongodb.kafka.connect.sink.processor.id.strategy

问题 I am working on debezium mongodb source connector. Can I run connector in local machine in distributed mode by giving kafka bootstrap server address as remote machine (deployed in Kubernetes) and remote MongoDB url? I tried this and I see connector starts successfully, no errors, just few warnings but no data is flowing from mongodb. Using below command to run connector ./bin/connect-distributed ./etc/schema-registry/connect-avro-distributed.properties ./etc/kafka/connect-mongodb-source