kafka 的心跳是 kafka consumer 和 broker 之间的健康检查,只有当 broker coordinator 正常时,consumer 才会发送心跳。

consumer 和 rebalance 相关的 2 个配置参数:

参数名                --> MemberMetadata 字段
session.timeout.ms --> MemberMetadata.sessionTimeoutMs
max.poll.interval.ms --> MemberMetadata.rebalanceTimeoutMs

broker 端,sessionTimeoutMs 参数

broker 处理心跳的逻辑在 GroupCoordinator 类中:如果心跳超期, broker coordinator 会把消费者从 group 中移除,并触发 rebalance。

   private def completeAndScheduleNextHeartbeatExpiration(group: GroupMetadata, member: MemberMetadata) {
// complete current heartbeat expectation
member.latestHeartbeat = time.milliseconds()
val memberKey = MemberKey(member.groupId, member.memberId)
heartbeatPurgatory.checkAndComplete(memberKey) // reschedule the next heartbeat expiration deadline
// 计算心跳截止时刻
val newHeartbeatDeadline = member.latestHeartbeat + member.sessionTimeoutMs
val delayedHeartbeat = new DelayedHeartbeat(this, group, member, newHeartbeatDeadline, member.sessionTimeoutMs)
heartbeatPurgatory.tryCompleteElseWatch(delayedHeartbeat, Seq(memberKey))
} // 心跳过期
def onExpireHeartbeat(group: GroupMetadata, member: MemberMetadata, heartbeatDeadline: Long) {
group.inLock {
if (!shouldKeepMemberAlive(member, heartbeatDeadline)) {
info(s"Member ${member.memberId} in group ${group.groupId} has failed, removing it from the group")
removeMemberAndUpdateGroup(group, member)
}
}
} private def shouldKeepMemberAlive(member: MemberMetadata, heartbeatDeadline: Long) =
member.awaitingJoinCallback != null ||
member.awaitingSyncCallback != null ||
member.latestHeartbeat + member.sessionTimeoutMs > heartbeatDeadline

consumer 端:sessionTimeoutMs,rebalanceTimeoutMs 参数

如果客户端发现心跳超期,客户端会标记 coordinator 为不可用,并阻塞心跳线程;如果超过了 poll 消息的间隔超过了 rebalanceTimeoutMs,则 consumer 告知 broker 主动离开消费组,也会触发 rebalance

org.apache.kafka.clients.consumer.internals.AbstractCoordinator.HeartbeatThread 代码片段:

if (coordinatorUnknown()) {
if (findCoordinatorFuture != null || lookupCoordinator().failed())
// the immediate future check ensures that we backoff properly in the case that no
// brokers are available to connect to.
AbstractCoordinator.this.wait(retryBackoffMs);
} else if (heartbeat.sessionTimeoutExpired(now)) {
// the session timeout has expired without seeing a successful heartbeat, so we should
// probably make sure the coordinator is still healthy.
markCoordinatorUnknown();
} else if (heartbeat.pollTimeoutExpired(now)) {
// the poll timeout has expired, which means that the foreground thread has stalled
// in between calls to poll(), so we explicitly leave the group.
maybeLeaveGroup();
} else if (!heartbeat.shouldHeartbeat(now)) {
// poll again after waiting for the retry backoff in case the heartbeat failed or the
// coordinator disconnected
AbstractCoordinator.this.wait(retryBackoffMs);
} else {
heartbeat.sentHeartbeat(now); sendHeartbeatRequest().addListener(new RequestFutureListener<Void>() {
@Override
public void onSuccess(Void value) {
synchronized (AbstractCoordinator.this) {
heartbeat.receiveHeartbeat(time.milliseconds());
}
} @Override
public void onFailure(RuntimeException e) {
synchronized (AbstractCoordinator.this) {
if (e instanceof RebalanceInProgressException) {
// it is valid to continue heartbeating while the group is rebalancing. This
// ensures that the coordinator keeps the member in the group for as long
// as the duration of the rebalance timeout. If we stop sending heartbeats,
// however, then the session timeout may expire before we can rejoin.
heartbeat.receiveHeartbeat(time.milliseconds());
} else {
heartbeat.failHeartbeat(); // wake up the thread if it's sleeping to reschedule the heartbeat
AbstractCoordinator.this.notify();
}
}
}
});
}
 /**
* A helper class for managing the heartbeat to the coordinator
*/
public final class Heartbeat {
private final long sessionTimeout;
private final long heartbeatInterval;
private final long maxPollInterval;
private final long retryBackoffMs; private volatile long lastHeartbeatSend; // volatile since it is read by metrics
private long lastHeartbeatReceive;
private long lastSessionReset;
private long lastPoll;
private boolean heartbeatFailed; public Heartbeat(long sessionTimeout,
long heartbeatInterval,
long maxPollInterval,
long retryBackoffMs) {
if (heartbeatInterval >= sessionTimeout)
throw new IllegalArgumentException("Heartbeat must be set lower than the session timeout"); this.sessionTimeout = sessionTimeout;
this.heartbeatInterval = heartbeatInterval;
this.maxPollInterval = maxPollInterval;
this.retryBackoffMs = retryBackoffMs;
} public void poll(long now) {
this.lastPoll = now;
} public void sentHeartbeat(long now) {
this.lastHeartbeatSend = now;
this.heartbeatFailed = false;
} public void failHeartbeat() {
this.heartbeatFailed = true;
} public void receiveHeartbeat(long now) {
this.lastHeartbeatReceive = now;
} public boolean shouldHeartbeat(long now) {
return timeToNextHeartbeat(now) == 0;
} public long lastHeartbeatSend() {
return this.lastHeartbeatSend;
} public long timeToNextHeartbeat(long now) {
long timeSinceLastHeartbeat = now - Math.max(lastHeartbeatSend, lastSessionReset);
final long delayToNextHeartbeat;
if (heartbeatFailed)
delayToNextHeartbeat = retryBackoffMs;
else
delayToNextHeartbeat = heartbeatInterval; if (timeSinceLastHeartbeat > delayToNextHeartbeat)
return 0;
else
return delayToNextHeartbeat - timeSinceLastHeartbeat;
} public boolean sessionTimeoutExpired(long now) {
return now - Math.max(lastSessionReset, lastHeartbeatReceive) > sessionTimeout;
} public long interval() {
return heartbeatInterval;
} public void resetTimeouts(long now) {
this.lastSessionReset = now;
this.lastPoll = now;
this.heartbeatFailed = false;
} public boolean pollTimeoutExpired(long now) {
return now - lastPoll > maxPollInterval;
} }

join group 的处理逻辑:kafka.coordinator.group.GroupCoordinator#onCompleteJoin

kafka 心跳和 reblance的更多相关文章

  1. kafka consumer 分区reblance算法

    转载请注明原创地址 http://www.cnblogs.com/dongxiao-yang/p/6238029.html 最近需要详细研究下kafka reblance过程中分区计算的算法细节,网上 ...

  2. kafka consumer频繁reblance

    转载请注明地址http://www.cnblogs.com/dongxiao-yang/p/5417956.html 结论与下文相同,kafka不同topic的consumer如果用的groupid名 ...

  3. Kafka知识总结及面试题

    目录 概念 Kafka基础概念 命令行 Kafka 数据存储设计 kafka在zookeeper中存储结构 生产者 生产者设计 消费者 消费者设计 面试题 kafka设计 请说明什么是Apache K ...

  4. Kafka集成SparkStreaming

    Spark Streaming + Kafka集成指南 Kafka项目在版本0.8和0.10之间引入了一个新的消费者API,因此有两个独立的相应Spark Streaming包可用.请选择正确的包,  ...

  5. kafka Auto offset commit faild reblance

    今天在使用python消费kafka时遇到了一些问题, 特记录一下. 场景一. 特殊情况: 单独写程序只用来生产消费数据 开始时间: 10:42 Topic: t_facedec Partition: ...

  6. Kafka消费与心跳机制

    1.概述 最近有同学咨询Kafka的消费和心跳机制,今天笔者将通过这篇博客来逐一介绍这些内容. 2.内容 2.1 Kafka消费 首先,我们来看看消费.Kafka提供了非常简单的消费API,使用者只需 ...

  7. Kafka技术内幕 读书笔记之(四) 新消费者——心跳任务

    消费者拉取数据是在拉取器中完成的,发送心跳是在消费者的协调者上完成的,但并不是说拉取器和消费者的协调者就没有关联关系 . “消费者的协调者”的作用是确保客户端的消费者和服务端的协调者之间的正常通信,如 ...

  8. Kafka:主要参数详解(转)

    原文地址:http://kafka.apache.org/documentation.html ############################# System ############### ...

  9. Kafka主要参数详解(转)

    原文档地址:http://kafka.apache.org/documentation.html ############################# System ############## ...

随机推荐

  1. 2018-2019-3 网络对抗技术 20165305 Exp3 免杀原理与实践

    1.实验内容及步骤 1.1 正确使用msf编码器,msfvenom生成如jar之类的其他文件,veil-evasion,加壳工具,使用shellcode编程 将做实验二时生成的后门文件用virusto ...

  2. java之过滤器

    form.jsp <%@ page language="java" contentType="text/html; charset=UTF-8" page ...

  3. 2017-12-26--mysql(5.6.15),linux下安装使用

    本文档的目的是,指导使用者安装.配置.还原所需要用到的mysql数据库.仅提供linux版本服务器的安装说明,且linux版本为64位的Centos6.4.同时,会提供的mysql安装包(MySQL- ...

  4. 【ubuntu】-桌面假死的解决办法

    第一,通过ctrl+art+F1(1-6),启动本地终端 切换到了字符界面tty1 第二,查询进程,ps -e |grep tty7 得到tty7的pid号 第三,杀死tty7的进程 , kill 9 ...

  5. kafka 控制台命令

    后台启动:bin/kafka-server-start.sh config/server.properties > /dev/null 2>&1 & 启动生产者:bin/k ...

  6. objectarx 读取外部DWG图到当前图形

    void CTrimeDraw::MyReadDwgFile(CString str){ AcDbDatabase pExternalDb(Adesk::kFalse); // 外部图形数据库 if ...

  7. VS2010下安装boost库

    在我们的C++项目中安装boost库,下面以VS2010版本作为例子,其它版本的设置也差不多. 一.编译生成boost库 1.下载最新的boost(本人下载的是boost_1_56_0).boost官 ...

  8. unity 根据平板 或者 手机 确立横竖屏

    /* ######### ############ ############# ## ########### ### ###### ##### ### ####### #### ### ####### ...

  9. jieba库和好玩的词云

    首先,通过pip3 install jieba安装jieba库,随后在网上下载<斗破>. 代码如下: import jieba.analyse path = '小说路径' fp = ope ...

  10. 通过nginx反向代理接收应用宝回调

    年前的时候,正好公司把用来接收应用宝回调的服务从Windows服务器上迁移带Linux服务器上,没想到最后折腾了两个工作日,腾讯的文档真的无力吐槽,不知道以后会不会有人碰到我相同的问题,我先记录一下. ...