序

本文主要研究一下rocketmq的sendBatchMessage

SendMessageRequestHeader

rocketmq-all-4.6.0-source-release/common/src/main/java/org/apache/rocketmq/common/protocol/header/SendMessageRequestHeader.java

public class SendMessageRequestHeader implements CommandCustomHeader {

    @CFNotNull

    private String producerGroup;

    @CFNotNull

    private String topic;

    @CFNotNull

    private String defaultTopic;

    @CFNotNull

    private Integer defaultTopicQueueNums;

    @CFNotNull

    private Integer queueId;

    @CFNotNull

    private Integer sysFlag;

    @CFNotNull

    private Long bornTimestamp;

    @CFNotNull

    private Integer flag;

    @CFNullable

    private String properties;

    @CFNullable

    private Integer reconsumeTimes;

    @CFNullable

    private boolean unitMode = false;

    @CFNullable

    private boolean batch = false;

    private Integer maxReconsumeTimes;

    //......

}
  • SendMessageRequestHeader定义了batch属性，用于标识是否是MessageBatch

processRequest

rocketmq-all-4.6.0-source-release/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java

public class SendMessageProcessor extends AbstractSendMessageProcessor implements NettyRequestProcessor {

    private List<ConsumeMessageHook> consumeMessageHookList;

    public SendMessageProcessor(final BrokerController brokerController) {

        super(brokerController);

    }

    @Override

    public RemotingCommand processRequest(ChannelHandlerContext ctx,

                                          RemotingCommand request) throws RemotingCommandException {

        SendMessageContext mqtraceContext;

        switch (request.getCode()) {

            case RequestCode.CONSUMER_SEND_MSG_BACK:

                return this.consumerSendMsgBack(ctx, request);

            default:

                SendMessageRequestHeader requestHeader = parseRequestHeader(request);

                if (requestHeader == null) {

                    return null;

                }

                mqtraceContext = buildMsgContext(ctx, requestHeader);

                this.executeSendMessageHookBefore(ctx, request, mqtraceContext);

                RemotingCommand response;

                if (requestHeader.isBatch()) {

                    response = this.sendBatchMessage(ctx, request, mqtraceContext, requestHeader);

                } else {

                    response = this.sendMessage(ctx, request, mqtraceContext, requestHeader);

                }

                this.executeSendMessageHookAfter(response, mqtraceContext);

                return response;

        }

    }

    //......

}
  • processRequest方法在判断requestHeader.isBatch()时会执行sendBatchMessage

sendBatchMessage

rocketmq-all-4.6.0-source-release/broker/src/main/java/org/apache/rocketmq/broker/processor/SendMessageProcessor.java

public class SendMessageProcessor extends AbstractSendMessageProcessor implements NettyRequestProcessor {

    private List<ConsumeMessageHook> consumeMessageHookList;

    public SendMessageProcessor(final BrokerController brokerController) {

        super(brokerController);

    }

    //......

    private RemotingCommand sendBatchMessage(final ChannelHandlerContext ctx,

                                             final RemotingCommand request,

                                             final SendMessageContext sendMessageContext,

                                             final SendMessageRequestHeader requestHeader) throws RemotingCommandException {

        final RemotingCommand response = RemotingCommand.createResponseCommand(SendMessageResponseHeader.class);

        final SendMessageResponseHeader responseHeader = (SendMessageResponseHeader)response.readCustomHeader();

        response.setOpaque(request.getOpaque());

        response.addExtField(MessageConst.PROPERTY_MSG_REGION, this.brokerController.getBrokerConfig().getRegionId());

        response.addExtField(MessageConst.PROPERTY_TRACE_SWITCH, String.valueOf(this.brokerController.getBrokerConfig().isTraceOn()));

        log.debug("Receive SendMessage request command {}", request);

        final long startTimstamp = this.brokerController.getBrokerConfig().getStartAcceptSendRequestTimeStamp();

        if (this.brokerController.getMessageStore().now() < startTimstamp) {

            response.setCode(ResponseCode.SYSTEM_ERROR);

            response.setRemark(String.format("broker unable to service, until %s", UtilAll.timeMillisToHumanString2(startTimstamp)));

            return response;

        }

        response.setCode(-1);

        super.msgCheck(ctx, requestHeader, response);

        if (response.getCode() != -1) {

            return response;

        }

        int queueIdInt = requestHeader.getQueueId();

        TopicConfig topicConfig = this.brokerController.getTopicConfigManager().selectTopicConfig(requestHeader.getTopic());

        if (queueIdInt < 0) {

            queueIdInt = Math.abs(this.random.nextInt() % 99999999) % topicConfig.getWriteQueueNums();

        }

        if (requestHeader.getTopic().length() > Byte.MAX_VALUE) {

            response.setCode(ResponseCode.MESSAGE_ILLEGAL);

            response.setRemark("message topic length too long " + requestHeader.getTopic().length());

            return response;

        }

        if (requestHeader.getTopic() != null && requestHeader.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {

            response.setCode(ResponseCode.MESSAGE_ILLEGAL);

            response.setRemark("batch request does not support retry group " + requestHeader.getTopic());

            return response;

        }

        MessageExtBatch messageExtBatch = new MessageExtBatch();

        messageExtBatch.setTopic(requestHeader.getTopic());

        messageExtBatch.setQueueId(queueIdInt);

        int sysFlag = requestHeader.getSysFlag();

        if (TopicFilterType.MULTI_TAG == topicConfig.getTopicFilterType()) {

            sysFlag |= MessageSysFlag.MULTI_TAGS_FLAG;

        }

        messageExtBatch.setSysFlag(sysFlag);

        messageExtBatch.setFlag(requestHeader.getFlag());

        MessageAccessor.setProperties(messageExtBatch, MessageDecoder.string2messageProperties(requestHeader.getProperties()));

        messageExtBatch.setBody(request.getBody());

        messageExtBatch.setBornTimestamp(requestHeader.getBornTimestamp());

        messageExtBatch.setBornHost(ctx.channel().remoteAddress());

        messageExtBatch.setStoreHost(this.getStoreHost());

        messageExtBatch.setReconsumeTimes(requestHeader.getReconsumeTimes() == null ? 0 : requestHeader.getReconsumeTimes());

        String clusterName = this.brokerController.getBrokerConfig().getBrokerClusterName();

        MessageAccessor.putProperty(messageExtBatch, MessageConst.PROPERTY_CLUSTER, clusterName);

        PutMessageResult putMessageResult = this.brokerController.getMessageStore().putMessages(messageExtBatch);

        return handlePutMessageResult(putMessageResult, response, request, messageExtBatch, responseHeader, sendMessageContext, ctx, queueIdInt);

    }

    //......

}
  • sendBatchMessage方法会执行msgCheck，之后构造messageExtBatch，然后执行brokerController.getMessageStore().putMessages(messageExtBatch)，之后通过handlePutMessageResult方法处理PutMessageResult

MessageExtBatch

rocketmq-all-4.6.0-source-release/common/src/main/java/org/apache/rocketmq/common/message/MessageExtBatch.java

public class MessageExtBatch extends MessageExt {

    private static final long serialVersionUID = -2353110995348498537L;

    public ByteBuffer wrap() {

        assert getBody() != null;

        return ByteBuffer.wrap(getBody(), 0, getBody().length);

    }

    private ByteBuffer encodedBuff;

    public ByteBuffer getEncodedBuff() {

        return encodedBuff;

    }

    public void setEncodedBuff(ByteBuffer encodedBuff) {

        this.encodedBuff = encodedBuff;

    }

}
  • MessageExtBatch继承了MessageExt，它提供了wrap方法，用于将body包装为ByteBuffer；它同时定义了encodedBuff，并提供了get、set方法

MessageExtBatchEncoder

rocketmq-all-4.6.0-source-release/store/src/main/java/org/apache/rocketmq/store/CommitLog.java

    public static class MessageExtBatchEncoder {

        // Store the message content

        private final ByteBuffer msgBatchMemory;

        // The maximum length of the message

        private final int maxMessageSize;

        MessageExtBatchEncoder(final int size) {

            this.msgBatchMemory = ByteBuffer.allocateDirect(size);

            this.maxMessageSize = size;

        }

        public ByteBuffer encode(final MessageExtBatch messageExtBatch) {

            msgBatchMemory.clear(); //not thread-safe

            int totalMsgLen = 0;

            ByteBuffer messagesByteBuff = messageExtBatch.wrap();

            int sysFlag = messageExtBatch.getSysFlag();

            int bornHostLength = (sysFlag & MessageSysFlag.BORNHOST_V6_FLAG) == 0 ? 4 + 4 : 16 + 4;

            int storeHostLength = (sysFlag & MessageSysFlag.STOREHOSTADDRESS_V6_FLAG) == 0 ? 4 + 4 : 16 + 4;

            ByteBuffer bornHostHolder = ByteBuffer.allocate(bornHostLength);

            ByteBuffer storeHostHolder = ByteBuffer.allocate(storeHostLength);

            while (messagesByteBuff.hasRemaining()) {

                // 1 TOTALSIZE

                messagesByteBuff.getInt();

                // 2 MAGICCODE

                messagesByteBuff.getInt();

                // 3 BODYCRC

                messagesByteBuff.getInt();

                // 4 FLAG

                int flag = messagesByteBuff.getInt();

                // 5 BODY

                int bodyLen = messagesByteBuff.getInt();

                int bodyPos = messagesByteBuff.position();

                int bodyCrc = UtilAll.crc32(messagesByteBuff.array(), bodyPos, bodyLen);

                messagesByteBuff.position(bodyPos + bodyLen);

                // 6 properties

                short propertiesLen = messagesByteBuff.getShort();

                int propertiesPos = messagesByteBuff.position();

                messagesByteBuff.position(propertiesPos + propertiesLen);

                final byte[] topicData = messageExtBatch.getTopic().getBytes(MessageDecoder.CHARSET_UTF8);

                final int topicLength = topicData.length;

                final int msgLen = calMsgLength(messageExtBatch.getSysFlag(), bodyLen, topicLength, propertiesLen);

                // Exceeds the maximum message

                if (msgLen > this.maxMessageSize) {

                    CommitLog.log.warn("message size exceeded, msg total size: " + msgLen + ", msg body size: " + bodyLen

                        + ", maxMessageSize: " + this.maxMessageSize);

                    throw new RuntimeException("message size exceeded");

                }

                totalMsgLen += msgLen;

                // Determines whether there is sufficient free space

                if (totalMsgLen > maxMessageSize) {

                    throw new RuntimeException("message size exceeded");

                }

                // 1 TOTALSIZE

                this.msgBatchMemory.putInt(msgLen);

                // 2 MAGICCODE

                this.msgBatchMemory.putInt(CommitLog.MESSAGE_MAGIC_CODE);

                // 3 BODYCRC

                this.msgBatchMemory.putInt(bodyCrc);

                // 4 QUEUEID

                this.msgBatchMemory.putInt(messageExtBatch.getQueueId());

                // 5 FLAG

                this.msgBatchMemory.putInt(flag);

                // 6 QUEUEOFFSET

                this.msgBatchMemory.putLong(0);

                // 7 PHYSICALOFFSET

                this.msgBatchMemory.putLong(0);

                // 8 SYSFLAG

                this.msgBatchMemory.putInt(messageExtBatch.getSysFlag());

                // 9 BORNTIMESTAMP

                this.msgBatchMemory.putLong(messageExtBatch.getBornTimestamp());

                // 10 BORNHOST

                this.resetByteBuffer(bornHostHolder, bornHostLength);

                this.msgBatchMemory.put(messageExtBatch.getBornHostBytes(bornHostHolder));

                // 11 STORETIMESTAMP

                this.msgBatchMemory.putLong(messageExtBatch.getStoreTimestamp());

                // 12 STOREHOSTADDRESS

                this.resetByteBuffer(storeHostHolder, storeHostLength);

                this.msgBatchMemory.put(messageExtBatch.getStoreHostBytes(storeHostHolder));

                // 13 RECONSUMETIMES

                this.msgBatchMemory.putInt(messageExtBatch.getReconsumeTimes());

                // 14 Prepared Transaction Offset, batch does not support transaction

                this.msgBatchMemory.putLong(0);

                // 15 BODY

                this.msgBatchMemory.putInt(bodyLen);

                if (bodyLen > 0)

                    this.msgBatchMemory.put(messagesByteBuff.array(), bodyPos, bodyLen);

                // 16 TOPIC

                this.msgBatchMemory.put((byte) topicLength);

                this.msgBatchMemory.put(topicData);

                // 17 PROPERTIES

                this.msgBatchMemory.putShort(propertiesLen);

                if (propertiesLen > 0)

                    this.msgBatchMemory.put(messagesByteBuff.array(), propertiesPos, propertiesLen);

            }

            msgBatchMemory.flip();

            return msgBatchMemory;

        }

        private void resetByteBuffer(final ByteBuffer byteBuffer, final int limit) {

            byteBuffer.flip();

            byteBuffer.limit(limit);

        }

    }
  • MessageExtBatchEncoder提供了encode方法，它首先通过messageExtBatch.wrap()得到messagesByteBuff，之后重新组装数据到msgBatchMemory

putMessages

rocketmq-all-4.6.0-source-release/store/src/main/java/org/apache/rocketmq/store/CommitLog.java

public class CommitLog {

	//......

    public PutMessageResult putMessages(final MessageExtBatch messageExtBatch) {

        messageExtBatch.setStoreTimestamp(System.currentTimeMillis());

        AppendMessageResult result;

        StoreStatsService storeStatsService = this.defaultMessageStore.getStoreStatsService();

        final int tranType = MessageSysFlag.getTransactionValue(messageExtBatch.getSysFlag());

        if (tranType != MessageSysFlag.TRANSACTION_NOT_TYPE) {

            return new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null);

        }

        if (messageExtBatch.getDelayTimeLevel() > 0) {

            return new PutMessageResult(PutMessageStatus.MESSAGE_ILLEGAL, null);

        }

        InetSocketAddress bornSocketAddress = (InetSocketAddress) messageExtBatch.getBornHost();

        if (bornSocketAddress.getAddress() instanceof Inet6Address) {

            messageExtBatch.setBornHostV6Flag();

        }

        InetSocketAddress storeSocketAddress = (InetSocketAddress) messageExtBatch.getStoreHost();

        if (storeSocketAddress.getAddress() instanceof Inet6Address) {

            messageExtBatch.setStoreHostAddressV6Flag();

        }

        long eclipsedTimeInLock = 0;

        MappedFile unlockMappedFile = null;

        MappedFile mappedFile = this.mappedFileQueue.getLastMappedFile();

        //fine-grained lock instead of the coarse-grained

        MessageExtBatchEncoder batchEncoder = batchEncoderThreadLocal.get();

        messageExtBatch.setEncodedBuff(batchEncoder.encode(messageExtBatch));

        //......

    }

    //......

}
  • putMessages方法会使用batchEncoder.encode(messageExtBatch)来设置messageExtBatch的encodedBuff

doAppend

rocketmq-all-4.6.0-source-release/store/src/main/java/org/apache/rocketmq/store/CommitLog.java

public class CommitLog {

	//......

    class DefaultAppendMessageCallback implements AppendMessageCallback {

        public AppendMessageResult doAppend(final long fileFromOffset, final ByteBuffer byteBuffer, final int maxBlank,

            final MessageExtBatch messageExtBatch) {

            byteBuffer.mark();

            //physical offset

            long wroteOffset = fileFromOffset + byteBuffer.position();

            // Record ConsumeQueue information

            keyBuilder.setLength(0);

            keyBuilder.append(messageExtBatch.getTopic());

            keyBuilder.append('-');

            keyBuilder.append(messageExtBatch.getQueueId());

            String key = keyBuilder.toString();

            Long queueOffset = CommitLog.this.topicQueueTable.get(key);

            if (null == queueOffset) {

                queueOffset = 0L;

                CommitLog.this.topicQueueTable.put(key, queueOffset);

            }

            long beginQueueOffset = queueOffset;

            int totalMsgLen = 0;

            int msgNum = 0;

            msgIdBuilder.setLength(0);

            final long beginTimeMills = CommitLog.this.defaultMessageStore.now();

            ByteBuffer messagesByteBuff = messageExtBatch.getEncodedBuff();

            int sysFlag = messageExtBatch.getSysFlag();

            int storeHostLength = (sysFlag & MessageSysFlag.STOREHOSTADDRESS_V6_FLAG) == 0 ? 4 + 4 : 16 + 4;

            ByteBuffer storeHostHolder = ByteBuffer.allocate(storeHostLength);

            //......

        }

    }

    //......

}
  • doAppend方法则读取messageExtBatch.getEncodedBuff()获取messagesByteBuff

小结

SendMessageRequestHeader定义了batch属性，用于标识是否是MessageBatch；processRequest方法在判断requestHeader.isBatch()时会执行sendBatchMessage；sendBatchMessage方法会执行msgCheck，之后构造messageExtBatch，然后执行brokerController.getMessageStore().putMessages(messageExtBatch)，之后通过handlePutMessageResult方法处理PutMessageResult

doc

  • SendMessageProcessor

来源：http://www.1994july.club/seo/?p=1697

聊聊rocketmq的sendBatchMessage的更多相关文章

  1. 聊聊rocketmq的ConsumeMode.CONCURRENTLY

    序 本文主要研究一下rocketmq的ConsumeMode.CONCURRENTLY ConsumeMode.CONCURRENTLY rocketmq-spring-boot-2.0.4-sour ...

  2. 消息队列面试题、RabbitMQ面试题、Kafka面试题、RocketMQ面试题 （史上最全、持续更新、吐血推荐）

    文章很长,建议收藏起来,慢慢读! 疯狂创客圈为小伙伴奉上以下珍贵的学习资源: 疯狂创客圈 经典图书 : <Netty Zookeeper Redis 高并发实战> 面试必备 + 大厂必备 ...

  3. 读完 RocketMQ 源码，我学会了如何优雅的创建线程

    RocketMQ 是一款开源的分布式消息系统,基于高可用分布式集群技术,提供低延时.高可靠的消息发布与订阅服务. 这篇文章,笔者整理了 RocketMQ 源码中创建线程的几点技巧,希望大家读完之后,能 ...

  4. 【Java进阶面试系列之一】哥们，你们的系统架构中为什么要引入消息中间件？

    转: [Java进阶面试系列之一]哥们,你们的系统架构中为什么要引入消息中间件? **这篇文章开始,我们把消息中间件这块高频的面试题给大家说一下,也会涵盖一些MQ中间件常见的技术问题. 这里大家可以关 ...

  5. 【Apache RocketMQ】RocketMQ捐赠给Apache那些鲜为人知的故事-转自阿里中间件

    序言 今年的双十一对阿里巴巴中间件消息团队来说,注定是个不平凡的日子.在这一天,稳定性小组重点攻克的低延迟存储解决方案成功地经受住了大考.整个大促期间,99.996%的延迟落在了10ms以内,极个别由 ...

  6. 关于ActiveMQ、RocketMQ、RabbitMQ、Kafka一些总结和区别

    这是一篇分享文 转自:http://www.cnblogs.com/williamjie/p/9481780.html  尊重原作,谢谢 消息队列 为什么写这篇文章? 博主有两位朋友分别是小A和小B: ...

  7. RocketMQ入门（简介、特点）

    简介: RocketMQ作为一款纯java.分布式.队列模型的开源消息中间件,支持事务消息.顺序消息.批量消息.定时消息.消息回溯等. 发展历程: 1. Metaq(Metamorphosis) 1. ...

  8. RocketMQ中Broker的消息存储源码分析

    Broker和前面分析过的NameServer类似,需要在Pipeline责任链上通过NettyServerHandler来处理消息 [RocketMQ中NameServer的启动源码分析] 实际上就 ...

  9. RocketMQ之九：RocketMQ消息发送流程解读

    在讨论这个问题之前,我们先看一下Client的整体架构. Producer与Consumer类体系 从下图可以看出以下几点:(1)Producer与Consumer的共同逻辑,封装在MQClientI ...

随机推荐

  1. KEGG富集分析散点图.md

    输入数据格式 pathway = read.table("kegg.result",header=T,sep="\t") pp = ggplot(pathway ...

  2. Linux下指定so共享库的加载路径

    转载地址:https://blog.csdn.net/zorelemn/article/details/52596293,做了部分修改 一.库文件的搜索路径: 1.在配置文件/etc/ld.so.co ...

  3. 【转帖】K8S Deployment 命令

    K8S Deployment 命令 https://www.cnblogs.com/Tempted/p/7831604.html 今天学习了一下 kubectl scale deployment xx ...

  4. [ERROR] /D:/IDEA文件/travel/src/main/java/cn/itcast/travel/util/MailUtils.java:[1,1] 非法字符: '\ufeff'

    报错: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (defau ...

  5. Tomcat报错：No result type specified for result named 'success'

    今天学Struts, tomcat报出了异常信息 Exception starting filter [struts2] Unable to load configuration.还有 No resu ...

  6. IAR_EW_MSP430下载

    附带完整安装过程,来自本人下载截图. 附带四种花色的花样灯源码和仿真图(ps:不用担心是错的,有疑问欢迎博客留言) 链接:https://pan.baidu.com/s/1ShDRlEQLwkYNOu ...

  7. C语言变量和常量

    常量 在程序执行过程中,其值不能被改变 常量一般出现在表达式或者赋值语句 利用const 修饰的变量为常量,不可修改 利用define定义的一般为常量,定义时候不需要分号 利用extern修饰的量知识 ...

  8. errgroup 分析

    errgroup 在 WaitGroup 的基础上实现子协程错误传递, 同时使用 context 控制协程的生命周期. 使用 errgroup 的使用非常简单 package main import ...

  9. CH09 开机自动烧录QSPI

    版本信息: 版本 REV2018 时间 05/22/2018       XILINX ZYNQ LINUX篇 基于米联MZ7X系列                       电子版自学资料   常 ...

  10. zabbix添加自定义监控(自动发现)遇到的问题

    问题:zabbix添加自动发现端口,提示Value should be a JSON object [root@localhost zabbix_agentd.d]# zabbix_get -s 19 ...