kafka自带sh脚本使用示例:
(1)启动/关闭kafka服务:
```shell
nohup env JMX_PORT=9999 /path/to/kafka_2.10-0.8.2.2/bin/kafka-server-start.sh config/server.properties >/dev/null 2>&1 &
/path/to/kafka_2.10-0.8.2.2/bin/zookeeper-server-stop.sh config/zookeeper.properties >/dev/null 2>&1 &
```
(2)创建topic
/path/to/kafka_2.10-0.8.2.2/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
查看topic列表
/path/to/kafka_2.10-0.8.2.2/bin/kafka-topics.sh --list --zookeeper localhost:2181
(3)发送msg
/path/to/kafka_2.10-0.8.2.2/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
(4)消费msg
/path/to/kafka_2.10-0.8.2.2/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
(5)查看topic状态
/path/to/kafka_2.10-0.8.2.2/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
返回值:
Topic:my-replicated-topicPartitionCount:1ReplicationFactor:3Configs:
Topic: my-replicated-topicPartition: 0Leader: 1Replicas: 1,2,0Isr: 1,0,2
(6)删除topic,删除的时候需要在server.properties中设置"delete.topic.enable=true"才能删除,否则topic只是被标记删除:
/path/to/kafka_2.10-0.8.2.2/bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test
(7)修改topic
/path/to/kafka_2.10-0.8.2.2/bin/kafka-topics.sh --zookeeper localhost:2181 --alert --topic test --partitions 40
(8)添加/删除配置
/path/to/kafka_2.10-0.8.2.2/bin/kafka-topics.sh --zookeeper localhost:2181 --alert --topic test --config x=y
/path/to/kafka_2.10-0.8.2.2/bin/kafka-topics.sh --zookeeper localhost:2181 --alert --topic test --deleteConfig x
(9)balance leadership
第一种(CLI):
/path/to/kafka_2.10-0.8.2.2/bin/kafka-preferred-replica-election.sh --zookeeper localhost:2181
第二种(配置文件):
auto.leader.rebalance.enable=true
(10)集群之间mirror数据
/path/to/kafka_2.10-0.8.2.2/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config consumer-1.properties --consumer.config consumer-2.properties --producer.config producer.properties --whitelist my-topic
(11)check consumer position
/path/to/kafka_2.10-0.8.2.2/bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect localhost:2181 --group test
(12)自动前一数据到新机器(Automatically migrating data to new machines)
将topic:foo1、foo2上的所有partitions迁移到新的broker:5、6上,迁移结束之后,topic:foo1、foo2上的所有partitions只会在broker:5、6
创建 topic-to-move.json
>cat topic-to-move.json
{"topics":[{"topic":"foo1"},{"topic":"foo2"}],
"version":1
}
创建好json文件之后,使用如下命令:
--generate
/path/to/kafka_2.10-0.8.2.2/bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "5,6" --generate
--execute(执行操作)
/path/to/kafka_2.10-0.8.2.2/bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --execute
--verify(查看迁移进度)
/path/to/kafka_2.10-0.8.2.2/bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --verify
(13)自定义分区分配和迁移
下面的示例会讲topic foo1的partition 0分区迁移到broker5,6;同时把topic foo2的partition 1迁移到broker2、3。
>cat custom-reassignment.json
{
"version":1,
"partitions":[
{"topic":"foo1","partition":0,"replicas":[5,6]},
{"topic":"foo2","partition":1,"replicas":[2,3]}
]
}
--execute
/path/to/kafka/bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --execute
--verify
/path/to/kafka/bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --verify
(14)增加副本(increase replication factor)
下面的示例会将topic:foo的partition:0的副本数量从1增加到3个;在增加副本数量之前,这个partition的唯一副本是broker:5上,在这里我们将会给broker:6、7增加副本。
第一步是手工写自定义的重新分配计划的JSON文件:
>cat increase-replication-factor.json
{
"version":1,
"partitions":[
{"topic":"foo","partition":0,"replicas":[5,6,7]}
]
}
使用--execute参数和json文件增加副本数量:
/path/to/kafka/bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute
或者--verify选项可与工具用来检查分区重新分配的状态:
/path/to/kafka/bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --verify
来源:oschina
链接:https://my.oschina.net/u/867531/blog/539301