kafka集群部署

安装java环境(略)

安装zookeeper集群(略)

安装kafka集群

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
tar -zxvf kafka_2.12-2.3.0.tgz -C /export/servers/
mv /export/servers/kafka_2.12-2.3.0/ /export/servers/kafka
mkdir /export/data/kafka
cp /export/servers/kafka/config/server.properties /export/servers/kafka/config/server.properties.bak

vim /export/servers/kafka/config/server.properties
broker.id=0
port=9092
host.name=172.26.10.167
advertised.host.name=172.26.10.167
listeners=PLAINTEXT://172.26.10.167:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.message.max.bytes=10000000
socket.replica.fetch.max.bytes=10
log.dirs=/export/data/kafka/
num.partitions=5
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=72
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=172.26.10.167:2181,172.26.10.168:2181,172.26.10.169:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
num.replica.fetchers=3
replica.fetch.min.bytes=1jieidan
replica.fetch.max.bytes=5242880

vim kafka/bin/kafka-server-start.sh
调整KAFKA_HEAP_OPTS="-Xmx2G -Xms2G”的值

上述操作在三台主机上都执行,或者直接拷贝。

第二个与第三个节点需要修改以下参数:修改成不同的id与对应的主机IP
broker.id=0
host.name=172.26.10.167
advertised.host.name=172.26.10.167
listeners=PLAINTEXT://172.26.10.167:9092

启动方式:

编写kafka的启动文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
vim  /usr/lib/systemd/system/kafka.service

[Unit]
Description=Apache Kafka server (broker)
After=network.target zookeeper.service

[Service]
Type=simple
Environment=JAVA_HOME=/export/servers/jdk1.8.0_192/
ExecStart=/export/servers/kafka/bin/kafka-server-start.sh /export/servers/kafka/config/server.properties
ExecStop=/export/servers/kafka/bin/kafka-server-stop.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target

上述操作在三台主机上都执行,或者直接拷贝。
1
2
3
4
5
6
7
8
9
10
11
重新加载:systemctl daemon-reload

启动:systemctl start kafka

开机自启:systemctl enable kafka

停止:systemctl stop kafka

重启:systemctl restart kafka

查看状态:systemctl status kafka