linux虚拟机中的kafka docker 容器外网显示:

原因:

kafka的外网IP端口配置参数设置错误。

原--》设置了容器的IP端口。

改--》设置宿主机的ip以及宿主机上的端口。

摘取:

Kafka跨网络访问设置

 
实际场景:
  • kafka应用通过docker进行部署并暴露出不同kafka节点到不同的指定端口;
  • 业务系统通过虚拟机进行部署;
  • docker宿主机与业务系统在同一个局域网;
报错:
  • 业务系统连接kafka时返回docker kafka服务名及原端口(9092)
  • can not found the leader of kafka

原因:

  1.  kafka配置文件server.properties中,host.name只绑定在了内部的IP上面,对外的网卡无法访问,及在物理业务系统服务器上面并不能识别到kafka服务名;
  2.  把值设置为空的话会kafka监听端口在所有的网卡上绑定。但是在外网访问时,客户端又遇到了“java.nio.channels.ClosedChannelException”异常信息,server端用tcpdump分析的时候发现客户端有传递kafka所在容器的服务名过来。在client里断点跟踪一下发现是findLeader的时候返回的元信息是容器服务名而不是IP。业务系统客户端无法解析这个机器名所以出现了前面的异常。
  在server.properties 里还有另一个参数是解决这个问题的, advertised.host.name参数用来配置返回的host.name值,把这个参数配置为局域网IP地址即可。
这个参数默认没有启用,默认是返回的java.net.InetAddress.getCanonicalHostName的值。
  除了IP之外,还有PORT,外网对应的PORT也需要修改。
  以下是server.properties文件对应位置。
  针对docker容器可以用环境变量参数将具体的参数传递进去:
 
  KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://xxx.xxx.xxx.xxx:port (局域网宿主机的IP地址而非容器的IP,及暴露出来的端口)
 
  针对局域网向公网IP端口暴露的话,这修改配置文件,加入以下配置:
  
  advertised.listeners=PLAINTEXT://xxx.xxx.xxx.xxx:port
  
  当Kafka broker启动时,它会在ZK上注册自己的IP和端口号,客户端就通过这个IP和端口号来连接。此时就需要显示指定 advertised.host.name, advertised.listeners参数,让注册到ZK上的IP是外外部IP。需要在 server.properties 配置文件里增加如下配置(版本0.10.x broker及之后的新版本):
  新版本0.10.x broker配置弃用了advertised.host.name和 advertised.port这两个个配置项,就配置advertised.listeners就可以了。

参考文章:

Kafka跨网络访问设置

配置文件的内容:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# see kafka.server.KafkaConfig for additional details and defaults

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

# add my config kongfanjun
#port=9092
#端口号
#host.name=127.0.0.1
#服务器IP地址,修改为自己的服务器IP
log.dirs=/usr/local/kafka_2.11-1.1.0/logs
#日志存放路径,上面创建的目录
zookeeper.connect=localhost:2181
#zookeeper地址和端口,单机配置部署,localhost:2181

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092

# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
advertised.listeners=PLAINTEXT://192.168.153.133:3092

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600

############################# Log Basics #############################

# A comma separated list of directories under which to store log files
#log.dirs=/tmp/kafka-logs

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
# 1. Durability: Unflushed data may be lost if you are not using replication.
# 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
# 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

############################# Group Coordinator Settings #############################

# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0

docker kafka 外网访问不到的更多相关文章

  1. 外网访问内网Docker容器

    外网访问内网Docker容器 本地安装了Docker容器,只能在局域网内访问,怎样从外网也能访问本地Docker容器? 本文将介绍具体的实现步骤. 1. 准备工作 1.1 安装并启动Docker容器 ...

  2. kafka的advertised.host.name参数 外网访问配置

    kafka的server.properties文件 ```host.name```开始只绑定在了内部IP上,对外网卡无法访问. 把值设置为空的话会kafka监听端口在所有的网卡上绑定.但是在外网访问时 ...

  3. Kafka集群无法外网访问问题解决攻略

    Kafka无法集群外网访问问题解决方法  讲解本地消费者和生产者无法使用远程Kafka服务器的处理办法 服务搭建好Kafka服务后,机本.测试 OK,外面机器却无法访问,很是怪异. 环境说明:  Ka ...

  4. docker安装redis并允许外网访问

    拉取redis镜像 docker pull redis:3.2 本地新建redis配置文件 redis.conf ,写入以下内容 #允许外网访问bind 0.0.0.0 daemonize NO pr ...

  5. ubuntu下安装mysql及外网访问设置

    这么多年一直是mssql或者Oracle,mysql基本没用过,借着.net即将跨平台之际,也mysql一把.windows安装基本没啥难度,然后就是试了把linux下...结果坑不少,由于linux ...

  6. Kafka内外网访问

    本文介绍了Kafka内外网访问的设置. kafka的两个配置listeners和advertised.listeners listeners kafka监听的网卡的ip,假设你机器上有两张网卡,内网1 ...

  7. TortoiseSVN和VisualSVN-Server的配置使用,外网访问SVN版本库

    TortoiseSVN和VisualSVN-Server的配置使用,外网访问SVN版本库 SVN客户端程序:TortoiseSVN SVN服务器程序:VisualSVN-Server ######## ...

  8. vs visual studio 让外网访问设置

    vs2015 提供外网访问我是这么解决的 有时我们经常会用到连接外网的方式来调试自己写的莫名bug.而我们通常有两种解决方式 一.捕捉错误日志进行代码分析. 二.则是将我们的源码项目提供外网访问进行直 ...

  9. svn设置外网访问

    1.设置路由器 默认协议为:https 端口号:443 服务器地址:https://主机名/svn/版本库 例:https://mleo-pc/svn/Share/ 也可就主机名用IP地址代替 如:h ...

随机推荐

  1. 带你从零学ReactNative开发跨平台App开发(七)

    ReactNative跨平台开发系列教程: 带你从零学ReactNative开发跨平台App开发(一) 带你从零学ReactNative开发跨平台App开发(二) 带你从零学ReactNative开发 ...

  2. WiFi 干扰器,有时间可以去试试呦!

    转自社区: 0X01 引言 想不想搞个WIFI干扰器?网上搜集了一下资料,发现用esp8266可以实现简单的干扰功能,包括断网.复制.欺骗等等.刚好手上有块Tpyboard V202(30元),也是e ...

  3. SQLSERVER的 筛选索引(Fiter Index)

    fiter index(筛选索引)是SQL Server的一项功能,可使此数据库与众不同. 筛选索引的概念 SQL Server中常用的索引是一种物理结构,它包含来自所有行的一组选定列的值 在一张桌子 ...

  4. 名词解释:Linux内存管理之RSS和VSZ

    Linux内存管理中不管是top命令还是pmap命令,都会有RSS和VSZ这两个名词,这里解释一下: RSS( Resident Set Size )常驻内存集合大小,表示相应进程在RAM中占用了多少 ...

  5. [翻译] JTSReachability

    JTSReachabilit An adaptation of Apple's Reachability with some block-based conveniences. 这是一个苹果的网络检测 ...

  6. 超简易复制Model对象(为后续备忘录设计模式博文做铺垫)

    超简易复制Model对象(为后续备忘录设计模式博文做铺垫) 复制整个Model需要实现NSCopy协议,可以想象是非常麻烦的一件事情,今天我跟大家分享一个不需要你做任何操作的复制Model对象的方法, ...

  7. SCCM2012安装、配置

    1.sql server2012,排序规则选择:SQL_Latin1_General_CP1_CI_AS1.扩展AD架构2.打开ad用户和计算机,高级--system 容器授予 sccm服务器 完全控 ...

  8. Celery学习--- Celery 最佳实践之与django结合实现异步任务

    django 可以轻松跟celery结合实现异步任务,只需简单配置即可 同步执行和异步执行 注意:即使Celery的任务没有执行完成,但是已经创建了任务ID.可以利用前台的定时任务发送Ajax异步请求 ...

  9. python2.7与3.5共存windows平台安装

    文:铁乐与猫 2018-3-18 周日 01.首先是安装python2.7: 官网下载 https://www.python.org 点击安装包进行安装 可以选择自定义的路径 将默认打x的[add p ...

  10. ssh连接CentOS7服务器

    ssh原理: ssh是一种专为远程登陆会话和其他网络服务提供安全性的协议,主要用于远程登陆. ssh采用公钥加密,在远程连接时,远程主机接收到用户的登录请求,将自己的公钥发送给用户,用户使用这个公钥将 ...