kafka安装运行(docker)
zookeeper参照https://www.cnblogs.com/wintersoft/p/11128484.html
mkdir /opt/kafka -p
vim /opt/kafka/Dockerfile
FROM wurstmeister/kafka:2.12-2.3.0
EXPOSE 9092
sudo mkdir -p /var/log/kafka;sudo chmod -R 777 /var/log/kafka
vim /opt/kafka/docker-compose.yml
version: '2'
services:
kafka:
image: v-kafka
container_name: kafka
build:
context: .
dockerfile: Dockerfile
restart: always
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: kafkaserver
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: "zookeeperserver:2181"
volumes:
- /var/log/kafka/:/kafka
- /var/run/docker.sock:/var/run/docker.sock
extra_hosts:
- "kafkaserver:192.168.0.101"
- "zookeeperserver:192.168.0.101"
生成启动
cd /opt/kafka/
docker-compose build
docker-compose up -d --force-recreate
docker-compose down
docker-compose restart
查看进程
netstat -anltp|grep 9092
查看日志
docker logs --tail="500" kafka
docker logs -f kafka
进入容器
docker exec -it kafka /bin/bash
伪集群
sudo mkdir -p /var/log/kafka/node1;sudo chmod -R 777 /var/log/kafka/node1
sudo mkdir -p /var/log/kafka/node2;sudo chmod -R 777 /var/log/kafka/node2
sudo mkdir -p /var/log/kafka/node3;sudo chmod -R 777 /var/log/kafka/node3
vim /opt/kafka/docker-compose.yml
version: '2' services:
kafka1:
image: v-kafka1
container_name: kafka1
build:
context: .
dockerfile: Dockerfile
restart: always
ports:
- 9011:9092
environment:
KAFKA_PORT: 9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafkaserver1:9011
KAFKA_ADVERTISED_HOST_NAME: kafkaserver1
KAFKA_ADVERTISED_PORT: 9011
KAFKA_BROKER_ID: 1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_ZOOKEEPER_CONNECT: zookeeperserver1:2181,zookeeperserver2:2182,zookeeperserver3:2183
KAFKA_DELETE_TOPIC_ENABLE: "true"
volumes:
- /var/log/kafka/node1:/kafka
- /var/run/docker.sock:/var/run/docker.sock
extra_hosts:
- "kafkaserver1:192.168.0.101"
- "kafkaserver2:192.168.0.101"
- "kafkaserver3:192.168.0.101"
- "zookeeperserver1:192.168.0.101"
- "zookeeperserver2:192.168.0.101"
- "zookeeperserver3:192.168.0.101"
kafka2:
image: v-kafka2
container_name: kafka2
build:
context: .
dockerfile: Dockerfile
restart: always
ports:
- 9012:9092
environment:
KAFKA_PORT: 9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafkaserver2:9012
KAFKA_ADVERTISED_HOST_NAME: kafkaserver2
KAFKA_ADVERTISED_PORT: 9012
KAFKA_BROKER_ID: 2
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_ZOOKEEPER_CONNECT: zookeeperserver1:2181,zookeeperserver2:2182,zookeeperserver3:2183
KAFKA_DELETE_TOPIC_ENABLE: "true"
volumes:
- /var/log/kafka/node2:/kafka
- /var/run/docker.sock:/var/run/docker.sock
extra_hosts:
- "kafkaserver1:192.168.0.101"
- "kafkaserver2:192.168.0.101"
- "kafkaserver3:192.168.0.101"
- "zookeeperserver1:192.168.0.101"
- "zookeeperserver2:192.168.0.101"
- "zookeeperserver3:192.168.0.101"
kafka3:
image: v-kafka3
container_name: kafka3
build:
context: .
dockerfile: Dockerfile
restart: always
ports:
- 9013:9092
environment:
KAFKA_PORT: 9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafkaserver3:9013
KAFKA_ADVERTISED_HOST_NAME: kafkaserver3
KAFKA_ADVERTISED_PORT: 9013
KAFKA_BROKER_ID: 3
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_ZOOKEEPER_CONNECT: zookeeperserver1:2181,zookeeperserver2:2182,zookeeperserver3:2183
KAFKA_DELETE_TOPIC_ENABLE: "true"
volumes:
- /var/log/kafka/node3:/kafka
- /var/run/docker.sock:/var/run/docker.sock
extra_hosts:
- "kafkaserver1:192.168.0.101"
- "kafkaserver2:192.168.0.101"
- "kafkaserver3:192.168.0.101"
- "zookeeperserver1:192.168.0.101"
- "zookeeperserver2:192.168.0.101"
- "zookeeperserver3:192.168.0.101"
配置key规则:在前面加KAFKA_前缀 全部大写 “.”用“_”代替
如:
增加 Kafka 堆的内存大小 KAFKA_HEAP_OPTS=-Xmx4G -Xms4G
KAFKA_LOG_DIRS=/kafka/logs 时 volumes:- "./kafka3/logs:/kafka/logs"
kafka-manager的environment可设置APPLICATION_SECRET: "xxx"
KAFKA_LISTENERS的值 为内网地址
没配置delete.topic.enable=true,只是软删除
如果将topic软删除,java客户端会报:
WARN Error while fetching metadata with correlation id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
报 org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-3] 1 partitions have leader brokers without a matching listener, including [log-0]
一般是zookeeper的ip:port配置导致kafka报错问题,配置好后需清理zookeeper数据才会正常。
复制配置
docker cp kafka1:/opt/kafka/config/ /opt/kafka/kafka1_config_bak/
kafka-manager需在界面手动添加集群配置才能显示。
测试kafka
进入容器
docker exec -it kafka1 /bin/bash
创建topic
/opt/kafka/bin/kafka-topics.sh --create --bootstrap-server 192.168.0.101:9011,192.168.0.101:9012,192.168.0.101:9013 --topic myTestTopic --partitions 3 --replication-factor 3
注:replication-factor个数不能超过broker的个数
查看当前topic列表
/opt/kafka/bin/kafka-topics.sh --list --bootstrap-server 192.168.0.101:9011,192.168.0.101:9012,192.168.0.101:9013
运行一个消息生产者,指定topic为刚刚创建的myTestTopic
/opt/kafka/bin/kafka-console-producer.sh --broker-list 192.168.0.101:9011,192.168.0.101:9012,192.168.0.101:9013 --topic myTestTopic
输入任意字符 然后ctrl+c退出
查看指定topic明细
/opt/kafka/bin/kafka-topics.sh --describe --bootstrap-server 192.168.0.101:9011,192.168.0.101:9012,192.168.0.101:9013 --topic myTestTopic
消费消息
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.101:9011,192.168.0.101:9012,192.168.0.101:9013 --topic myTestTopic --from-beginning
ctrl+c退出
删除topic
/opt/kafka/bin/kafka-topics.sh --delete --bootstrap-server 192.168.0.101:9011,192.168.0.101:9012,192.168.0.101:9013 --topic myTestTopic
如果不能删除,docker启动时配置KAFKA_DELETE_TOPIC_ENABLE: "true"
kafka安装运行(docker)的更多相关文章
- Docker——MacOS上安装运行docker
近几年来,Docker越来越流行,使用场景也越来越广泛.为了能尽快跟上时代步伐.学习并应用到实际工作中,我也开始了Docker之旅. Docker版本 Docker共有两种版本: 社区版(Commun ...
- elastic stack安装运行(docker)
https://www.docker.elastic.co 注:目前阿里云为7.4 elasticsearch 参考https://www.elastic.co/guide/en/elasticsea ...
- TICK/TIGK运维栈安装运行 docker【中】
InfluxDB docker search influxdb docker pull influxdb docker run -d -p 8086:8086 -v /var/lib/influxdb ...
- Centos7基于容器安装运行Docker私有仓库及添加认证
一.前言 官方的Docker hub是一个用于管理公共镜像的好地方,我们可以在上面找到我们想要的镜像,也可以把我们自己的镜像推送上去.但是,有时候,我们的使用场景需要我们拥有一个私有的镜像仓库用于管理 ...
- skywalking安装运行(docker)
https://github.com/apache/skywalking-docker/tree/master/6/6.5 https://hub.docker.com/r/apache/skywal ...
- zookeeper安装运行(docker)
拉取镜像docker pull zookeeper:latest 获取镜像基本信息docker inspect zookeeper mkdir /opt/zookeeper -p vim /opt/z ...
- Windows OS上安装运行Apache Kafka教程
Windows OS上安装运行Apache Kafka教程 下面是分步指南,教你如何在Windows OS上安装运行Apache Zookeeper和Apache Kafka. 简介 本文讲述了如何在 ...
- Windows系统下安装运行Kafka
一.安装JAVA JDK 1.下载安装包 http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151. ...
- 在Windows安装运行Kafka
一.安装JAVA JDK 1.下载安装包 http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151. ...
随机推荐
- map字典,储存cookie,切换账户,展示购物车不同商品
1:首页 1,静态html5+css做好基本样式 html5,css,jQery, sass 2,jsonp的方式src引入模拟的json数据//这里用的jsonp方式拿数据,详情有使用ajax 遍历 ...
- CRM BP函数
REPORT ZCRM_BP_TEST. """""""""""""& ...
- Excel 批量导入Mysql(创建表-追加数据)
之前弄数据库的时候, 测试excel导mysql, 中间用pandas 处理后再入库. 直接上代码, 此种有真意, 尽在不言中. #!/usr/bin/env python # coding: ut ...
- java 8大happen-before原则
1.单线程happen-before原则:在同一个线程中,书写在前面的操作happen-before后面的操作. 2.锁的happen-before原则:同一个锁的unlock操作happen-bef ...
- java读取Properties文件的方法
resource.properties的内容: com.tsinkai.ettp.name=imooc com.tsinkai.ettp.website=www.imooc.com com.tsink ...
- win10 LTSC系统 安装应用商店和纸牌合集,解决从应用商店安装Solitaire Collection纸牌打开空白的问题
家里台式机换了win10系统,想给老妈玩那个纸牌游戏(我也超喜欢的!. 发现这个系统没有自带纸牌游戏Microsoft Solitaire Collection, 过分的是,连应用商店都没有...呵呵 ...
- 大数据:Hadoop(HDFS 读写数据流程及优缺点)
一.HDFS 写数据流程 写的过程: CLIENT(客户端):用来发起读写请求,并拆分文件成多个 Block: NAMENODE:全局的协调和把控所有的请求,提供 Block 存放在 DataNode ...
- 动态创建form 完成form 提交
document.body.appendChild(jForm) won't work because jForm is not a dom element, it is a jQuery objec ...
- shell 编程生成日期文件;Server虚拟机上进行Web服务器配置
shell 编程生成日期文件 1. 请编写一个脚本,命名为sh01.sh,其功能是: 键盘输入文件名(要求使用名字全拼作为文件名). 自动创建3个文件. 1个为系统当天日期(CCYYMMDD). 1个 ...
- spark-shell操作hive
本文是在集群已经搭建好的基础上来说的,还没有搭建好集群的小伙伴还请自行百度! 启动spark-shell之前要先启动hive metastore 和 hiveservice2 hive --servi ...