zookeeper参照https://www.cnblogs.com/wintersoft/p/11128484.html

mkdir /opt/kafka -p
vim /opt/kafka/Dockerfile

FROM wurstmeister/kafka:2.12-2.3.0
EXPOSE 9092

sudo mkdir -p /var/log/kafka;sudo chmod -R 777 /var/log/kafka

vim /opt/kafka/docker-compose.yml

version: '2'
services:
kafka:
image: v-kafka
container_name: kafka
build:
context: .
dockerfile: Dockerfile
restart: always
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: kafkaserver
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: "zookeeperserver:2181"
volumes:
- /var/log/kafka/:/kafka
- /var/run/docker.sock:/var/run/docker.sock
extra_hosts:
- "kafkaserver:192.168.0.101"
- "zookeeperserver:192.168.0.101"

生成启动
cd /opt/kafka/
docker-compose build
docker-compose up -d --force-recreate
docker-compose down
docker-compose restart

查看进程
netstat -anltp|grep 9092

查看日志
docker logs --tail="500" kafka
docker logs -f kafka

进入容器
docker exec -it kafka /bin/bash

伪集群

sudo mkdir -p /var/log/kafka/node1;sudo chmod -R 777 /var/log/kafka/node1
sudo mkdir -p /var/log/kafka/node2;sudo chmod -R 777 /var/log/kafka/node2
sudo mkdir -p /var/log/kafka/node3;sudo chmod -R 777 /var/log/kafka/node3

vim /opt/kafka/docker-compose.yml

version: '2'

services:
kafka1:
image: v-kafka1
container_name: kafka1
build:
context: .
dockerfile: Dockerfile
restart: always
ports:
- 9011:9092
environment:
KAFKA_PORT: 9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafkaserver1:9011
KAFKA_ADVERTISED_HOST_NAME: kafkaserver1
KAFKA_ADVERTISED_PORT: 9011
KAFKA_BROKER_ID: 1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_ZOOKEEPER_CONNECT: zookeeperserver1:2181,zookeeperserver2:2182,zookeeperserver3:2183
KAFKA_DELETE_TOPIC_ENABLE: "true"
volumes:
- /var/log/kafka/node1:/kafka
- /var/run/docker.sock:/var/run/docker.sock
extra_hosts:
- "kafkaserver1:192.168.0.101"
- "kafkaserver2:192.168.0.101"
- "kafkaserver3:192.168.0.101"
- "zookeeperserver1:192.168.0.101"
- "zookeeperserver2:192.168.0.101"
- "zookeeperserver3:192.168.0.101"
kafka2:
image: v-kafka2
container_name: kafka2
build:
context: .
dockerfile: Dockerfile
restart: always
ports:
- 9012:9092
environment:
KAFKA_PORT: 9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafkaserver2:9012
KAFKA_ADVERTISED_HOST_NAME: kafkaserver2
KAFKA_ADVERTISED_PORT: 9012
KAFKA_BROKER_ID: 2
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_ZOOKEEPER_CONNECT: zookeeperserver1:2181,zookeeperserver2:2182,zookeeperserver3:2183
KAFKA_DELETE_TOPIC_ENABLE: "true"
volumes:
- /var/log/kafka/node2:/kafka
- /var/run/docker.sock:/var/run/docker.sock
extra_hosts:
- "kafkaserver1:192.168.0.101"
- "kafkaserver2:192.168.0.101"
- "kafkaserver3:192.168.0.101"
- "zookeeperserver1:192.168.0.101"
- "zookeeperserver2:192.168.0.101"
- "zookeeperserver3:192.168.0.101"
kafka3:
image: v-kafka3
container_name: kafka3
build:
context: .
dockerfile: Dockerfile
restart: always
ports:
- 9013:9092
environment:
KAFKA_PORT: 9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafkaserver3:9013
KAFKA_ADVERTISED_HOST_NAME: kafkaserver3
KAFKA_ADVERTISED_PORT: 9013
KAFKA_BROKER_ID: 3
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_ZOOKEEPER_CONNECT: zookeeperserver1:2181,zookeeperserver2:2182,zookeeperserver3:2183
KAFKA_DELETE_TOPIC_ENABLE: "true"
volumes:
- /var/log/kafka/node3:/kafka
- /var/run/docker.sock:/var/run/docker.sock
extra_hosts:
- "kafkaserver1:192.168.0.101"
- "kafkaserver2:192.168.0.101"
- "kafkaserver3:192.168.0.101"
- "zookeeperserver1:192.168.0.101"
- "zookeeperserver2:192.168.0.101"
- "zookeeperserver3:192.168.0.101"

配置key规则:在前面加KAFKA_前缀 全部大写 “.”用“_”代替

如:
增加 Kafka 堆的内存大小 KAFKA_HEAP_OPTS=-Xmx4G -Xms4G
KAFKA_LOG_DIRS=/kafka/logs 时 volumes:- "./kafka3/logs:/kafka/logs"

kafka-manager的environment可设置APPLICATION_SECRET: "xxx"
KAFKA_LISTENERS的值 为内网地址

没配置delete.topic.enable=true,只是软删除

如果将topic软删除,java客户端会报:

WARN Error while fetching metadata with correlation id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

报 org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-3] 1 partitions have leader brokers without a matching listener, including [log-0]

一般是zookeeper的ip:port配置导致kafka报错问题,配置好后需清理zookeeper数据才会正常。

复制配置
docker cp kafka1:/opt/kafka/config/ /opt/kafka/kafka1_config_bak/

kafka-manager需在界面手动添加集群配置才能显示。

测试kafka

进入容器
docker exec -it kafka1 /bin/bash

创建topic
/opt/kafka/bin/kafka-topics.sh --create --bootstrap-server 192.168.0.101:9011,192.168.0.101:9012,192.168.0.101:9013 --topic myTestTopic --partitions 3 --replication-factor 3
注:replication-factor个数不能超过broker的个数

查看当前topic列表
/opt/kafka/bin/kafka-topics.sh --list --bootstrap-server 192.168.0.101:9011,192.168.0.101:9012,192.168.0.101:9013

运行一个消息生产者,指定topic为刚刚创建的myTestTopic
/opt/kafka/bin/kafka-console-producer.sh --broker-list 192.168.0.101:9011,192.168.0.101:9012,192.168.0.101:9013 --topic myTestTopic
输入任意字符 然后ctrl+c退出

查看指定topic明细
/opt/kafka/bin/kafka-topics.sh --describe --bootstrap-server 192.168.0.101:9011,192.168.0.101:9012,192.168.0.101:9013 --topic myTestTopic

消费消息
/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.101:9011,192.168.0.101:9012,192.168.0.101:9013 --topic myTestTopic --from-beginning
ctrl+c退出

删除topic
/opt/kafka/bin/kafka-topics.sh --delete --bootstrap-server 192.168.0.101:9011,192.168.0.101:9012,192.168.0.101:9013 --topic myTestTopic
如果不能删除,docker启动时配置KAFKA_DELETE_TOPIC_ENABLE: "true"

 

kafka安装运行(docker)的更多相关文章

  1. Docker——MacOS上安装运行docker

    近几年来,Docker越来越流行,使用场景也越来越广泛.为了能尽快跟上时代步伐.学习并应用到实际工作中,我也开始了Docker之旅. Docker版本 Docker共有两种版本: 社区版(Commun ...

  2. elastic stack安装运行(docker)

    https://www.docker.elastic.co 注:目前阿里云为7.4 elasticsearch 参考https://www.elastic.co/guide/en/elasticsea ...

  3. TICK/TIGK运维栈安装运行 docker【中】

    InfluxDB docker search influxdb docker pull influxdb docker run -d -p 8086:8086 -v /var/lib/influxdb ...

  4. Centos7基于容器安装运行Docker私有仓库及添加认证

    一.前言 官方的Docker hub是一个用于管理公共镜像的好地方,我们可以在上面找到我们想要的镜像,也可以把我们自己的镜像推送上去.但是,有时候,我们的使用场景需要我们拥有一个私有的镜像仓库用于管理 ...

  5. skywalking安装运行(docker)

    https://github.com/apache/skywalking-docker/tree/master/6/6.5 https://hub.docker.com/r/apache/skywal ...

  6. zookeeper安装运行(docker)

    拉取镜像docker pull zookeeper:latest 获取镜像基本信息docker inspect zookeeper mkdir /opt/zookeeper -p vim /opt/z ...

  7. Windows OS上安装运行Apache Kafka教程

    Windows OS上安装运行Apache Kafka教程 下面是分步指南,教你如何在Windows OS上安装运行Apache Zookeeper和Apache Kafka. 简介 本文讲述了如何在 ...

  8. Windows系统下安装运行Kafka

    一.安装JAVA JDK 1.下载安装包 http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151. ...

  9. 在Windows安装运行Kafka

    一.安装JAVA JDK 1.下载安装包 http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151. ...

随机推荐

  1. map字典,储存cookie,切换账户,展示购物车不同商品

    1:首页 1,静态html5+css做好基本样式 html5,css,jQery, sass 2,jsonp的方式src引入模拟的json数据//这里用的jsonp方式拿数据,详情有使用ajax 遍历 ...

  2. CRM BP函数

    REPORT ZCRM_BP_TEST. """""""""""""& ...

  3. Excel 批量导入Mysql(创建表-追加数据)

    之前弄数据库的时候, 测试excel导mysql, 中间用pandas 处理后再入库.  直接上代码, 此种有真意, 尽在不言中. #!/usr/bin/env python # coding: ut ...

  4. java 8大happen-before原则

    1.单线程happen-before原则:在同一个线程中,书写在前面的操作happen-before后面的操作. 2.锁的happen-before原则:同一个锁的unlock操作happen-bef ...

  5. java读取Properties文件的方法

    resource.properties的内容: com.tsinkai.ettp.name=imooc com.tsinkai.ettp.website=www.imooc.com com.tsink ...

  6. win10 LTSC系统 安装应用商店和纸牌合集,解决从应用商店安装Solitaire Collection纸牌打开空白的问题

    家里台式机换了win10系统,想给老妈玩那个纸牌游戏(我也超喜欢的!. 发现这个系统没有自带纸牌游戏Microsoft Solitaire Collection, 过分的是,连应用商店都没有...呵呵 ...

  7. 大数据:Hadoop(HDFS 读写数据流程及优缺点)

    一.HDFS 写数据流程 写的过程: CLIENT(客户端):用来发起读写请求,并拆分文件成多个 Block: NAMENODE:全局的协调和把控所有的请求,提供 Block 存放在 DataNode ...

  8. 动态创建form 完成form 提交

    document.body.appendChild(jForm) won't work because jForm is not a dom element, it is a jQuery objec ...

  9. shell 编程生成日期文件;Server虚拟机上进行Web服务器配置

    shell 编程生成日期文件 1. 请编写一个脚本,命名为sh01.sh,其功能是: 键盘输入文件名(要求使用名字全拼作为文件名). 自动创建3个文件. 1个为系统当天日期(CCYYMMDD). 1个 ...

  10. spark-shell操作hive

    本文是在集群已经搭建好的基础上来说的,还没有搭建好集群的小伙伴还请自行百度! 启动spark-shell之前要先启动hive metastore 和 hiveservice2 hive --servi ...