apache-zookeeper

I'm a bit confused about where offsets are stored when using Kafka and Zookeeper. It seems like offsets in some cases are stored in Zookeeper, in other cases they are stored in Kafka. What determines whether the offset is stored in Kafka or in Zookeeper? And what the pros and cons? NB: Of course I could also store the offset on my own in some different data store but that is not part of the picture for this post. Some more details about my setup: I run these versions: KAFKA_VERSION="0.10.1.0", SCALA_VERSION="2.11" I connect to Kafka/Zookeeper using kafka-node from my NodeJS application. Older

So my objective here is to set up a cluster of several kafka-brokers in a distributed fashion. But I can't see the way to make the brokers aware of each other. As far as i understand, every broker needs a separate ID in their config, which I cannot guarantee or configure if I launch the containers from kubernetes? They also need to have the same advertised_host? Are there any parameters I'm missing that would need to be changed for the nodes to discover each other? Would it be viable to do such a configuration at the end of the Dockerfile with a script? And/or a shared volume? I'm currently

In my zookeeper/conf/log4j.properties file I set the zookeeper.log.dir to $HOME/zklogs When I use zkServer.sh it does not use that directory. Instead it uses the ${ZOO_LOG_DIR} which when I echo it, comes out to "." I don't understand how fix this issue, I don't see the ${ZOO_LOG_DIR} set anywhere. I am not sure how it gets set to "." at all. I also don't know how to launch zookeeper without zkServer.sh. I am noobish at linux too and a little lost on this issue... Does anybody know how I can fix this issue so that it uses the directory set in my log4j.properties file in the conf directory? **

Can Apache Mesos 'master' nodes be co-located on the same machine as Mesos 'slave' nodes? Similarly (for high-availability (HA) deploys), can the Apache Zookeeper nodes used in Mesos 'master' election be deployed on the same machines as Mesos 'slave' nodes? Mesos recommends 3 'masters' be used for HA deploys, and Zookeeper recommends 5 nodes be used for its quorum election system. It would be nice to have these services running along side Mesos 'slave' processes instead of committing 8 machines to effectively 'non-productive' tasks. If such a setup is feasible, what are the pros/cons of such a

I exploring a few options to setup kafka and I knew that the Zookeeper has to be up and running to initiate a kafka. I would like to know how can I find the below. 1) hostname and port for my zookeeper instance---I checked the zoo.cfg and I could only find the ClientPort not the hostname, will hostname be the hostname of my box?? 2) To check if ZooKeeper is up and running---I tried to do a ps -ef | grep "zoo" I could not find anything. May be I am using a wrong key word to search?? Any help would be really appreciated? To check if Zookeeper is accessible. One method is to simply telnet to the

I am new to Hadoop/ZooKeeper. I cannot understand the purpose of using ZooKeeper with Hadoop, is ZooKeeper writing data in Hadoop? If not, then why we do we use ZooKeeper with Hadoop? Arnon Rotem-Gal-Oz Hadoop 1.x does not use Zookeeper. HBase does use zookeeper even in Hadoop 1.x installations. Hadoop adopted Zookeeper as well starting with version 2.0. The purpose of Zookeeper is cluster management. This fits with the general philosophy of *nix of using smaller specialized components - so components of Hadoop that want clustering capabilities rely on Zookeeper for that rather than develop

I have an HBase-0.98.3-hadoop2 instance running on a ubuntu VM, I could not connect to it (the main thread stuck when creating HTable instance) with the following code: Configuration config = HBaseConfiguration.create(); config.set("hbase.zookeeper.quorum", "192.168.56.101"); HTableInterface usersTable = new HTable(config, "users"); Here is what I got in the console SLF4J: The requested version 1.5.8 by your slf4j binding is not compatible with [1.6, 1.7] SLF4J: See http://www.slf4j.org/codes.html#version_mismatch for further details. 14/06/14 12:40:05 INFO zookeeper.ZooKeeper: Client