问题
So I am learning Kafka currently and have attempted to duplicate the examples provided from Apache here. This is example code for the consumer and I have written it in java just as shown. When I attempt to execute the file however I run into some issues. I am able to get the file to compile but it will not run properly.
I am executing the program with the following line without the quotations, "java TestConsumer localhost:2181 group1 test 4" This passes the 4 arguments necessary in the example code. I am provided with the following error though when I execute this command.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Category
at kafka.utils.VerifiableProperties.<init>(Unknown Source)
at kafka.consumer.ConsumerConfig.<init>(Unknown Source)
at TestConsumer.ConsumerProps(TestConsumer.java:69)
at TestConsumer.<init>(TestConsumer.java:31)
at TestConsumer.main(TestConsumer.java:97)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Category
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 5 more
I have tried going in an manually replacing the arguments with the necessary values and attempting to execute that way but I am given a different issue. Below is the error message along with the code I'm using just in case I screwed something up from the example provided. If anyone can help me out I would be incredibly appreciative since I am attempting to write my own consumer to test with parsing given information, etc. Thanks
log4j:WARN No appenders could be found for logger (kafka.utils.VerifiableProperties).
log4j:WARN Please initialize the log4j system properly.
Exception in thread "main" java.lang.NoClassDefFoundError: org/I0Itec/zkclient/IZkStateListener
at kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(Unknown Source)
at kafka.javaapi.consumer.ZookeeperConsumerConnector.<init>(Unknown Source)
at kafka.consumer.Consumer$.createJavaConsumerConnector(Unknown Source)
at kafka.consumer.Consumer.createJavaConsumerConnector(Unknown Source)
at TestConsumer.<init>(TestConsumer.java:31)
at TestConsumer.main(TestConsumer.java:97)
Caused by: java.lang.ClassNotFoundException: org.I0Itec.zkclient.IZkStateListener
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 6 more
/*
* Test Consumer to gather input from
* a Producer. Attempt to perform functions
* from the produced data
*/
// Kafka API
import kafka.consumer.ConsumerConfig;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;
import java.util.Map;
import java.util.HashMap;
import java.util.Properties;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
public class TestConsumer{
private final ConsumerConnector consumer;
private final String topic;
private ExecutorService executor;
// CONSTRUCTOR
public TestConsumer(String zookeeper, String groupid, String aTopic){
consumer = kafka.consumer.Consumer.createJavaConsumerConnector(ConsumerProps(zookeeper, groupid));
this.topic = aTopic;
}
// END CONSTRUCTOR
// RUN FUNCTION
public void run(int threads){
Map<String, Integer> topicMap = new HashMap<String, Integer>();
topicMap.put(topic, new Integer(threads));
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicMap);
List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
executor = Executors.newFixedThreadPool(threads); // process threads
int numThread = 0; // thread counter for consumption
// consumer all messages
for(final KafkaStream stream : streams){
executor.submit(new TestConsumerRun(stream, numThread));
numThread ++;
}
}
// END RUN FUNCTION
// CREATE PROPERTIES FUNCTION
private static ConsumerConfig ConsumerProps(String zookeeper, String groupid){
Properties properties = new Properties(); // config properties file
properties.put("zookeeper.connect", zookeeper);
properties.put("group.id", groupid);
properties.put("zookeeper.session.timeout.ms", "400");
properties.put("zookeeper.sync.time.ms", "200");
properties.put("auto.commit.interval.ms", "1000");
properties.put("auto.offset.reset", "smallest");
return new ConsumerConfig(properties);
}
// END CREATE PROPERTIES FUNCTION
// SHUTDOWN FUNCTION
public void shutdown(){
if (consumer != null) consumer.shutdown();
if (executor != null) executor.shutdown();
try{
if (!executor.awaitTermination(5000, TimeUnit.MILLISECONDS)){
System.out.println("Timed out waiting for consumer threads to shut down, exiting uncleanly");
}
} catch (InterruptedException e){
System.out.println("Interrupted during shutdown, exiting uncleanly");
}
}
// END SHUTDOWN FUNCTION
// MAIN FUNCTION
public static void main(String[] args){
String zookeeper = args[0];
String groupid = args[1];
String topic = args[2];
int threads = Integer.parseInt(args[3]);
TestConsumer test = new TestConsumer(zookeeper, groupid, topic); // send information to constructor
test.run(threads); // pass threads for iteration
try{
Thread.sleep(10000);
} catch (InterruptedException ie){
}
test.shutdown(); // close program
}
// END MAIN FUNCTION
}/*
* Test Consumer to gather input from
* a Producer. Attempt to perform functions
* from the produced data
*/
// Kafka API
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
public class TestConsumerRun implements Runnable{
private KafkaStream aStream;
private int aThread;
// CONSTRUCTOR
public TestConsumerRun(KafkaStream stream, int thread){
aStream = stream; // set stream from main read
aThread = thread; // set thread from main read
}
// END CONSTRUCTOR
// RUN FUNCTION
public void run(){
ConsumerIterator<byte[], byte[]> iterator = aStream.iterator(); // used to check throughout the list continiously
while(iterator.hasNext())
System.out.println("Thread " + aThread + ": " + new String(iterator.next().message()));
System.out.println("Shutting down Thread: " + aThread);
}
// END RUN FUNCTION
}回答1:
I had the same problem. You need to add log4j jar to your classpath. Also you might need to add slf4j and commons-logging.
java.lang.NoClassDefFoundError occurs when JVM can't find the class at runtime. (But it was there during compile.) Happens when a jar is missing during runtime and also for many other reasons. Your classpath during the compile and runtime needs to be the same. Sometimes you might have the same jar with different versions, so at runtime JVM might find the different version rather than the one used in compile.
来源：https://stackoverflow.com/questions/30468024/kafka-consumer-in-java