apache-zookeeper

Consuming from Kafka failed Iterator is in failed state

懵懂的女人 提交于 2019-12-04 01:57:47
问题 I am getting exception while consuming the messages from kafka. org.springframework.messaging.MessagingException: Consuming from Kafka failed; nested exception is java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Iterator is in failed state I have one consumer in the application context with one outbound adapter. Consumer configuration in application context <int-kafka:consumer-context id="consumerContext" consumer-timeout="4000" zookeeper-connect="zookeeperConnect">

embedded zookeeper for unit/integration test

左心房为你撑大大i 提交于 2019-12-04 01:30:33
Is there an embedded zookeeper so that we could use it in unit testing? It can be shipped with the test and run out of the box. Maybe we could mock some service and register to the embedded zookeeper The Curator framework has TestingServer and TestingCluster classes (see https://github.com/Netflix/curator/wiki/Utilities ) that are in a separate maven artifact (curator-test - see the Maven/Artifacts section of https://github.com/Netflix/curator/wiki ). They're pretty self explanatory, or you can download the curator code base and see how they're used internally in their own test cases. We've

How do I delete a Kafka Consumer Group to reset offsets?

隐身守侯 提交于 2019-12-04 00:22:51
问题 I want to delete a Kakfa consumer group so that when the application creates a consumer and subscribes to a topic it can start at the beginning of the topic data. This is with a single node development vm using the current latest Confluent Platform 3.1.2 which uses Kafka 0.10.1.1. I try the normal syntax: sudo /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server localhost:9092 --delete --group my_consumer_group I get the error: Option [delete] is only valid with [zookeeper]. Note

Why can't my Zookeeper server rejoin the Quorum?

半腔热情 提交于 2019-12-03 23:52:36
I have three servers in my quorum. They are running ZooKeeper 3.4.5. Two of them appear to be running fine based on the output from mntr . One of them was restarted a couple days ago due to a deploy, and since then has not been able to join the quorum. Some lines in the logs that stick out are: 2014-03-03 18:44:40,995 [myid:1] - INFO [main:QuorumPeer@429] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation and: 2014-03-03 18:44:41,233 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumCnxManager@190] - Have

Killing node with __consumer_offsets leads to no message consumption at consumers

二次信任 提交于 2019-12-03 20:43:18
I have 3 node(nodes0,node1,node2) Kafka cluster(broker0, broker1, broker2) with replication factor 2 and Zookeeper(using zookeeper packaged with Kafka tar) running on a different node (node 4). I had started broker 0 after starting zookeper and then remaining nodes. It is seen in broker 0 logs that it is reading __consumer_offsets and seems they are stored on broker 0. Below are sample logs: Kafka Version: kafka_2.10-0.10.2.0 2017-06-30 10:50:47,381] INFO [GroupCoordinator 0]: Loading group metadata for console-consumer-85124 with generation 2 (kafka.coordinator.GroupCoordinator) [2017-06-30

Connecting an external Accumulo instance and java

僤鯓⒐⒋嵵緔 提交于 2019-12-03 20:28:50
I am trying to connect to a VM with Accumulo. The problem is, I can't get it hooked up in Java. I can see the webpage Apache throws up, but I can't get it to work with code. I think this is a lack of knowledge issue rather than a real problem, but I can't find documentation on it. All the examples use localhost as the zooServer name, this obviously doesn't work for me. So, here is my code: String instanceName = "accumulo-02" String zooServers = "192.168.56.5, accumulo-02.localdomain:9997" ZooKeeperInstance inst = new ZooKeeperInstance(instanceName, zooServers) .... The rest isn't important

Kafka in Kubernetes - Marking the coordinator dead for group

情到浓时终转凉″ 提交于 2019-12-03 17:36:51
问题 I am pretty new to Kubernetes and wanted to setup Kafka and zookeeper with it. I was able to setup Apache Kafka and Zookeeper in Kubernetes using StatefulSets. I followed this and this to build my manifest file. I made 1 replica of kafka and zookeeper each and also used persistent volumes. All pods are running and ready. I tried to expose kafka and used Service for this by specifying a nodePort(30010). Seemingly this would expose kafka to the outside world where they can send messages to the

Got user-level KeeperException when processing

倾然丶 夕夏残阳落幕 提交于 2019-12-03 15:12:01
问题 I know this has been asked before, so please forgive me for asking again, but none of the solutions pointed to by that thread worked for me. I'm trying to setup Kafka 0.10.20 following the docs. When I start ZooKeeper, I get the following logs: $ bin/zookeeper-server-start.sh config/zookeeper.properties [2017-04-22 12:25:15,810] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2017-04-22 12:25:15,814] INFO autopurge

How to point autoconf/automake to non-standard packages

匆匆过客 提交于 2019-12-03 13:13:18
问题 I'm trying to build ZooKeeper on a RedHat Linux box. (Exactly what ZooKeeper is is probably not important :-) When I follow the package instructions, I get: $ autoreconf -if aclocal:configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not found in library aclocal:configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not found in library configure.ac:33: error: possibly undefined macro: AM_PATH_CPPUNIT If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf

How to use kazoo client for leader election?

丶灬走出姿态 提交于 2019-12-03 12:06:44
This is the code mentioned on kazoo readthedocs election=zk.Election("/electionpath", "my-identifier") what are the input arguments to be passed to make particular node as leader? (i.e what does /electionpath and my-identifier refers here?) In Short: "/electionpath" is your path of interest where you will be creating nodes, adding data and watching nodes using dataWatchers. "my-identifier" is identifier to the non re-entrant lock which will be used to verify who is the leader out of the contenders and allow writes only to leader. In Detail: To simplify it explaining first why there should be