问题
I've been successfully using a simple custom Partitioner class written in Java for a Kafka Connect sink on Confluent 3.2.x (Kafka 0.10.x). I want to upgrade to Confluent 4.1 (Kafka 1.1) and am experiencing errors.
Kafka Connect's plugin loading mechanism seems to have been changed in CP 3.3.0. Previously, there was just the CLASSPATH option, but with CP 3.3.0+ there is a newer and recommended plugin.path mechanism.
If I try to keep using the legacy CLASSPATH plugin mechanism, when I try to use my plugin, I get:
java.lang.ClassNotFoundException: io.confluent.connect.storage.partitioner.DefaultPartitioner
That is a CP internal class. With the older CP 3.2.x, that was available on the classpath, however with the new classpath isolation efforts in CP >= 3.3.0, I presume that must be provided along with the plugin.
I figure it's wise to switch to the newer recommended plugin.path mechanism. I remove the CLASSPATH entry. In the default /etc/kafka/connect-distributed.properties, I see plugin.path=/usr/share/java, so I install my plugin .jar to /usr/share/java/my-custom-partitioner/my-custom-partitioner.jar. I have tried adding and not adding dependency .jar files there as well.
My plugin seems to get loaded when the Kafka Connect service starts up:
INFO Loading plugin from: /usr/share/java/my-custom-partitioner (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:194)
INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/my-custom-partitioner/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:217)
When I do:
curl -X PUT -H "Content-Type: application/json" --data-binary "@sink_test_1.json" my-dev-test-vm:8083/connectors/sink-test-1/config
I get:
{"error_code":500,"message":null}%
I can see in the kafka connect systemd log:
java.lang.NullPointerException
at io.confluent.connect.storage.partitioner.PartitionerConfig.classNameEquals(PartitionerConfig.java:270)
at io.confluent.connect.storage.partitioner.PartitionerConfig.access$000(PartitionerConfig.java:33)
at io.confluent.connect.storage.partitioner.PartitionerConfig$PartitionerClassDependentsRecommender.visible(PartitionerConfig.java:238)
at org.apache.kafka.common.config.ConfigDef.validate(ConfigDef.java:617)
at org.apache.kafka.common.config.ConfigDef.validate(ConfigDef.java:625)
at org.apache.kafka.common.config.ConfigDef.validate(ConfigDef.java:525)
at org.apache.kafka.common.config.ConfigDef.validateAll(ConfigDef.java:508)
at org.apache.kafka.common.config.ConfigDef.validate(ConfigDef.java:490)
at org.apache.kafka.connect.connector.Connector.validate(Connector.java:133)
It's not apparent what is going wrong or why my partitioner class isn't loading correctly.
FYI, I have rebuilt my Java plugin with CP 4.1 + Kafka 1.1 dependencies and made small updates to match API changes such as adding an implementation for getSchemaGeneratorClass to my partitioner class.
回答1:
Custom Kafka Connect Partitioner classes will not work via the old CLASSPATH mechanism and they will not work as plugins with the newer Kafka 0.11.0+ isolated plugin mechanism.
The only working solution is to copy your custom .jar file with your custom Kafka Connect Partitioner class into the kafka-connect-storage-common plugin directorry at /usr/share/java/kafka-connect-storage-common/. Custom Kafka Connect Partitioner plugin classes must exist in that same directory so they are in the same isolated class loader.
FYI, you can see that the Kafka 0.11.0+ isolated plugin mechanism will only load subclasses of four specific Java classes that do not cover the Kafka Connect partitioners here:
https://github.com/apache/kafka/blob/fdcf75ea326b8e07d8d7c4f5cc14f1e70451bcd4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/isolation/DelegatingClassLoader.java#L279
Thanks to cricket_007 for recommending this exact solution: putting custom Kafka Connect partitioner .jar files in the /share/java/kafka-storage-common directory. I learned the hard way exactly why this has to be done and why alternatives don't work.
来源：https://stackoverflow.com/questions/50121329/packaging-a-custom-java-partitioner-class-plugin-for-kafka-connect-in-confluen