kafa 集群部署与配置

kafka 简介

关键词

部署

初始化系统环境

yum -y install java-1.8.0-openjdk supervisor

安装zookeeper 集群

wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz

mkdir -p /data/apps/zookeeper-data/zk1/data
mkdir -p /data/apps/zookeeper-data/zk1/logs 

cat > /data/apps/zookeeper-3.4.14/conf/zoo.conf <<EOF
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/apps/zookeeper-data/zk1/data
dataLogDir=/data/apps/zookeeper-data/zk1/logs
clientPort=2181

server.1=10.93.216.94:2287:3387
server.2=10.93.216.67:2288:3388
EOF

echo "1" >/data/apps/zookeeper-data/zk1/data/myid

./bin/zkServer.sh start conf/zoo.conf

mkdir -p /data/apps/zookeeper-data/zk2/data
mkdir -p /data/apps/zookeeper-data/zk2/logs 

cat > /data/apps/zookeeper-3.4.14/conf/zoo.conf <<EOF
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/apps/zookeeper-data/zk2/data
dataLogDir=/data/apps/zookeeper-data/zk2/logs
clientPort=2181

server.1=10.93.216.94:2287:3387
server.2=10.93.216.67:2288:3388
EOF

echo "2" >/data/apps/zookeeper-data/zk2/data/myid

## 启动zookeeper
cd /data/apps/zookeeper  
./bin/zkServer.sh start conf/zoo1.conf

安装kafka集群

wget https://archive.apache.org/dist/kafka/2.1.0/kafka_2.12-2.1.0.tgz

mkdir kafka_logs

server.properties

       broker.id=0

       listeners=PLAINTEXT://10.93.216.94:9092
       log.dirs=/data/apps/kafka-logs

cat >/usr/lib/systemd/system/kafka.service <<EOF
[Unit]
Description=The Kafka server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
ExecStart=/data/apps/kafka_2.12-2.1.0/bin/kafka-server-start.sh -daemon /data/apps/kafka_2.12-2.1.0/config/server.properties
ExecStop=/data/apps/kafka_2.12-2.1.0/bin/kafka-server-stop.sh  /data/apps/kafka_2.12-2.1.0/config/server.properties &

[Install]
WantedBy=multi-user.target

EOF

###创建主题
./bin/kafka-topics.sh --create --zookeeper 10.93.216.67:2181 --replication-factor 2 --partitions 8 --topic test

###查看主题
./bin/kafka-topics.sh --list --zookeeper 10.93.216.67:2181

###删除主题
./bin/kafka-topics.sh --delete --topic test --zookeeper 10.93.216.67:2181

安全认证

WEB管理

kafka-manager
腾讯源 https://mirrors.cloud.tencent.com/

安装依赖的nodejs


wget https://nodejs.org/dist/v8.9.3/node-v8.9.3-linux-x64.tar.xz && xz -d node-v8.9.3-linux-x64.tar.xz && tzr xmf node-v8.9.3-linux-x64.tar && ln -s node-v8.9.3-linux-x64 nodejs

echo "export NODE_HOME=/data/apps/nodejs" > /etc/profile.d/nodejs.sh
echo "export PATH=$NODE_HOME/bin:$PATH" >> /etc/profile.d/nodejs.sh

安装kafka-manager

https://github.com/yahoo/kafka-manager/archive/2.0.0.2.tar.gz
tar zxmf 2.0.0.2.tar.gz
 curl https://bintray.com/sbt/rpm/rpm > /etc/yum.repos.d/bintray-sbt-rpm.repo
 yum install sbt maven

vi ~/.sbt/repositories
[repositories]
local
aliyun: http://maven.aliyun.com/nexus/content/groups/public/
typesafe: http://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
sonatype-oss-releases
maven-central
sonatype-oss-snapshots

sbt-version

sbt -v

cd /data/apps

# 编译很慢，需要等。。。。。。
./sbt clean dist

cp target/universal/kafka-manager-2.0.0.2.zip /data/apps

cd /data/apps
upzip kafka-manager-2.0.0.2

vi conf/application.conf

kafka-manager.zkhosts="10.93.216.67:2181,10.93.216.94:2181"

#启动很慢，一直等到端口启动算是完毕
nohup ./bin/kafka-manager -Dconfig.file=conf/application.conf -Dhttp.port=8088 &

nohup ./bin/kafka-manager -Dconfig.file=conf/application.conf -Dhttp.port=8088 >kafka-manager.log 2>&1 &

来源：51CTO

作者：东凌阁

链接：https://blog.51cto.com/13673090/2432722

标签

zookeeper集群

data

Kafka