背景

因磁盘满了,导致kafka所有的服务器全部宕机了,然后重启kafka集群,服务是启动成功了,但有一些报错:

broker1:

broker2:

broker3:一直在刷以下错误信息

虽然报了这些错,但kafka正常启动了,通过命令测试了集群能正常生产和消费消息,但是看kafka-manager界面,出现副本未分配的异常情况:

检查消费这些主题的程序,果然是消费失败了,一直在刷如下异常信息:

注:图中IP的是broker3节点

截止到这里可以看出,broker3节点出问题了,导致消费者程序连接不上,但奇怪的话,通过命令创建主题测试,在broker3节点又能消费。

继续分析broker3的日志,报错原因:集群要求的副本数是2,但只找到1个。

于是查看相关主题的详细信息,发现确实ISR列表中是少了副本

猜测由于宕机后,有些节点落后leader太多,还没有追上来,所以脱离了ISR列表,于是等它自动追上来。

等到第2天一看,还是一样,没有追上来,于是决定重启kafka集群,发现有些分区的会自动扩展成2,出问题的那些分区还是没有。。。。

然后想通过重新分配分区指定副本,看能否让它自动恢复一下副本,通过以下命令进行处理:

bin/kafka-reassign-partitions.sh --zookeeper 10.0.xx.x:,10.0.xx.x:,10.0.xx.x: --reassignment-json-file reassign.json --execute
reassign.json文件内容:
{"version":, "partitions":[
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]},
{"topic":"__consumer_offsets","partition":,"replicas":[,]}
]}`

重新分区指定副本的方法也不行,于是修改kafka配置,把集群要求的副本数改为1:

vi server.properties

重启kafka集群后,broker3不在就报错了,在重启消费都程序,也能正常连上kafka进行消费了。

总结:

kafka出现宕机后,副本脱离ISR列表(落后leader太多),按正常来说它会慢慢追上来后在自动重新加入ISR列表中,但我的等了20个小时后还没有,重启kafka集群后也没有恢复。导致服务启动有问题。

现在临时解决方案是调整成1,让它先跑一段时间后,看能否恢复回来,到时在设置成2。

问题:

1、原因尚未找到;

2、这样调整后,kafka会出现数据丢失的情况(出问题期间的数据都丢失了)。

Kafka管理与监控——broker宕机后无法消费问题的更多相关文章

  1. 假如Kafka集群中一个broker宕机无法恢复,应该如何处理?

    假如Kafka集群中一个broker宕机无法恢复, 应该如何处理? 今天面试时遇到这个问题, 网上资料说添加新的broker, 是不会自动同步旧数据的. 笨办法 环境介绍 三个broker的集群, z ...

  2. 万答#4,延迟从库加上MASTER_DELAY,主库宕机后如何快速恢复服务

    欢迎来到 GreatSQL社区分享的MySQL技术文章,如有疑问或想学习的内容,可以在下方评论区留言,看到后会进行解答 当主库宕机后,延迟从库如何才能"取消"主动延迟,以便恢复服务 ...

  3. 解Bug之路-记一次对端机器宕机后的tcp行为

    解Bug之路-记一次对端机器宕机后的tcp行为 前言 机器一般过质保之后,就会因为各种各样的问题而宕机.而这一次的宕机,让笔者观察到了平常观察不到的tcp在对端宕机情况下的行为.经过详细跟踪分析原因之 ...

  4. 记一次 oracle 数据库在宕机后的恢复

    系统:redhat 6.6 oracle版本: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production 问题描述: ...

  5. 实验:zk master宕机后,临时节点在新的master上是否存在,结果出人意料

    一.实验 实验说明:3台zk集群,主要验证:master上的客户端,在master上建立临时节点,当master宕机时,其他follower选为主后,临时节点是否存在. 主要是通过此来验证,基于zk的 ...

  6. 『叶问』#41,三节点的MGR集群,有两个节点宕机后还能正常工作吗

    『叶问』#41,三节点的MGR集群,有两个节点宕机后还能正常工作吗 每周学点MGR知识. 1. 三节点的MGR集群,有两个节点宕机后还能正常工作吗 要看具体是哪种情况. 如果两个节点是正常关闭的话,则 ...

  7. 关于mysql主从架构master宕机后,请求转移问题解决办法

    mysql架构:一主一从 问题一:有两台mysql数据库,已做好主从.如果运行某一天master服务器mysql故障导致前端请求无法处理怎么办? 答:将前端需要数据库处理的请求转移到slave机上. ...

  8. Kafka管理与监控——彻底删除topic

    一.配置 server.properties 设置 delete.topic.enable=true 如果没有设置 delete.topic.enable=true,则调用kafka 的delete命 ...

  9. Kafka管理与监控——调优

    1.JVM参数配置优化 如果使用的CMS GC算法,建议JVM Heap不要太大,在4GB以内就可以.JVM太大,导致Major GC或者Full GC产生的“stop the world”时间过长, ...

随机推荐

  1. LoadRunner在Controller场景中配置获取Windows Resources

    一.首先需要在被监控Windows服务器端(只支持Windows)进行如下设置: 启动服务: Remote Procedure Call (RPC) RemoteRegistry 操作方法: 按Win ...

  2. vue 2.0 watch 监听对象的变化

  3. 人们为什么在Python脚本的第一行上编写#!/ usr / bin / env python shebang?

    在我看来,如果没有该行,文件运行相同. #1楼 您可以使用virtualenv尝试此问题 这是test.py #! /usr/bin/env python import sys print(sys.v ...

  4. DOM(innerHTML和className)

    <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8&quo ...

  5. SIGAI机器学习第一集 机器学习简介

    讲授机器学习的基本概念.发展历史与典型应用 大纲: 人工智能简介机器学习简介为什么需要机器学习机器学习的发展历史机器学习的典型应用人工智能主要的公司本课程讲授的算法 机器学习并不是人工智能一上来就采用 ...

  6. sql server 交集,差集的用法 (集合运算)

    概述 为什么使用集合运算: 在集合运算中比联接查询和EXISTS/NOT EXISTS更方便. 并集运算(UNION) 并集:两个集合的并集是一个包含集合A和B中所有元素的集合. 在T-SQL中.UN ...

  7. sql server 常见约束

    1.not null 非空约束 ①强制列不接受空值 ②例:创建表时,name varchar(6) not null, 2.unique 唯一性约束 ①约束唯一标识数据库表中的每条记录 ②unique ...

  8. junit报错

    java.lang.RuntimeException: iwap 环境还没有初始化,请先调用IWapContext.init(). at com.nantian.ofpiwap.IWapContext ...

  9. 「CF375E」Red and Black Tree「树形DP」

    题意 给定一个结点颜色红或黑的树,问最少进行多少次交换黑.红结点使得每个红结点离最近的黑结点距离\(\leq x\). \(1\leq n \leq 500, 1 \leq x \leq 10^9\) ...

  10. \ddd和\xddd 转义序列

    转自 http://blog.csdn.net/todd911/article/details/8851475 书中有如下描述: \ddd  ddd表示1~3个八进制数字,这个转义符表示的字符就是给定 ...