ActiveMQ 常见集群模式

Master Slave 架构模式

这种模式是基于文件共享锁的高可用集群，个人理解其实是一种 Failover 模式，可以是一主一从，也可以是一主两从。

本文使用 Docker 搭建一个主从模式的 ActiveMQ 5.8.0 集群，一个主节点，两个从节点。

ActiveMQ 镜像的创建可以参考构建ActiveMQ镜像并运行

version: "2.1"

services:

  activemq-A:

    image: cocowool/activemq:5.8.0

    hostname: activemq-A

    expose:

      - "8161"

      - "61613"

    ports:

      - "8161:8161"

      - "61613:61613"

    volumes:

      - ~/Projects/sh-valley/docker-conf/activemq/activemq-5.8.0-A.xml:/usr/local/apache-activemq-5.8.0/conf/activemq.xml

      - ~/Projects/sh-valley/docker-conf/activemq/kahadb:/usr/local/apache-activemq-5.8.0/data/kahadb

    networks:

      - default

  activemq-B:

    image: cocowool/activemq:5.8.0

    hostname: activemq-B

    expose:

      - "8162"

      - "61614"

    ports:

      - "8162:8161"

      - "61614:61613"

    volumes:

      - ~/Projects/sh-valley/docker-conf/activemq/activemq-5.8.0-B.xml:/usr/local/apache-activemq-5.8.0/conf/activemq.xml

      - ~/Projects/sh-valley/docker-conf/activemq/kahadb:/usr/local/apache-activemq-5.8.0/data/kahadb

    networks:

      - default

其实两个配置文件是一样的，关键是将 kahadb 的目录利用本地文件实现在两个容器见共享。验证过程记录。

$ docker-compose up

Attaching to activemq_activemq-B_1, activemq_activemq-A_1

activemq-B_1  | INFO: Using default configuration

activemq-B_1  | (you can configure options in one of these file: /etc/default/activemq /root/.activemqrc)

activemq-B_1  |

activemq-B_1  | INFO: Invoke the following command to create a configuration file

activemq-B_1  | /usr/local/apache-activemq-5.8.0/bin/activemq setup [ /etc/default/activemq | /root/.activemqrc ]

activemq-B_1  |

activemq-A_1  | INFO: Using default configuration

activemq-A_1  | (you can configure options in one of these file: /etc/default/activemq /root/.activemqrc)

activemq-A_1  |

activemq-A_1  | INFO: Invoke the following command to create a configuration file

activemq-B_1  | INFO: Using java '/usr/local/jdk1.8.0_144/bin/java'

activemq-A_1  | /usr/local/apache-activemq-5.8.0/bin/activemq setup [ /etc/default/activemq | /root/.activemqrc ]

activemq-A_1  |

activemq-B_1  | INFO: Starting in foreground, this is just for debugging purposes (stop process by pressing CTRL+C)

activemq-A_1  | INFO: Using java '/usr/local/jdk1.8.0_144/bin/java'

activemq-A_1  | INFO: Starting in foreground, this is just for debugging purposes (stop process by pressing CTRL+C)

activemq-B_1  | Java Runtime: Oracle Corporation 1.8.0_144 /usr/local/jdk1.8.0_144/jre

activemq-B_1  |   Heap sizes: current=1005056k  free=989327k  max=1005056k

activemq-B_1  |     JVM args: -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Djava.io.tmpdir=/usr/local/apache-activemq-5.8.0/tmp -Dactivemq.classpath=/usr/local/apache-activemq-5.8.0/conf; -Dactivemq.home=/usr/local/apache-activemq-5.8.0 -Dactivemq.base=/usr/local/apache-activemq-5.8.0 -Dactivemq.conf=/usr/local/apache-activemq-5.8.0/conf -Dactivemq.data=/usr/local/apache-activemq-5.8.0/data

activemq-B_1  | Extensions classpath:

activemq-B_1  |   [/usr/local/apache-activemq-5.8.0/lib,/usr/local/apache-activemq-5.8.0/lib/camel,/usr/local/apache-activemq-5.8.0/lib/optional,/usr/local/apache-activemq-5.8.0/lib/web,/usr/local/apache-activemq-5.8.0/lib/extra]

activemq-B_1  | ACTIVEMQ_HOME: /usr/local/apache-activemq-5.8.0

activemq-B_1  | ACTIVEMQ_BASE: /usr/local/apache-activemq-5.8.0

activemq-B_1  | ACTIVEMQ_CONF: /usr/local/apache-activemq-5.8.0/conf

activemq-B_1  | ACTIVEMQ_DATA: /usr/local/apache-activemq-5.8.0/data

activemq-A_1  | Java Runtime: Oracle Corporation 1.8.0_144 /usr/local/jdk1.8.0_144/jre

activemq-A_1  |   Heap sizes: current=1005056k  free=989327k  max=1005056k

activemq-A_1  |     JVM args: -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Djava.io.tmpdir=/usr/local/apache-activemq-5.8.0/tmp -Dactivemq.classpath=/usr/local/apache-activemq-5.8.0/conf; -Dactivemq.home=/usr/local/apache-activemq-5.8.0 -Dactivemq.base=/usr/local/apache-activemq-5.8.0 -Dactivemq.conf=/usr/local/apache-activemq-5.8.0/conf -Dactivemq.data=/usr/local/apache-activemq-5.8.0/data

activemq-A_1  | Extensions classpath:

activemq-A_1  |   [/usr/local/apache-activemq-5.8.0/lib,/usr/local/apache-activemq-5.8.0/lib/camel,/usr/local/apache-activemq-5.8.0/lib/optional,/usr/local/apache-activemq-5.8.0/lib/web,/usr/local/apache-activemq-5.8.0/lib/extra]

activemq-A_1  | ACTIVEMQ_HOME: /usr/local/apache-activemq-5.8.0

activemq-A_1  | ACTIVEMQ_BASE: /usr/local/apache-activemq-5.8.0

activemq-A_1  | ACTIVEMQ_CONF: /usr/local/apache-activemq-5.8.0/conf

activemq-A_1  | ACTIVEMQ_DATA: /usr/local/apache-activemq-5.8.0/data

activemq-B_1  | Loading message broker from: xbean:activemq.xml

activemq-A_1  | Loading message broker from: xbean:activemq.xml

activemq-B_1  |  INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@68c4039c: startup date [Wed Sep 07 13:16:20 UTC 2022]; root of context hierarchy

activemq-A_1  |  INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@68c4039c: startup date [Wed Sep 07 13:16:20 UTC 2022]; root of context hierarchy

activemq-B_1  |  INFO | PListStore:[/usr/local/apache-activemq-5.8.0/data/localhost/tmp_storage] started

activemq-B_1  |  INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/usr/local/apache-activemq-5.8.0/data/kahadb]

activemq-A_1  |  INFO | PListStore:[/usr/local/apache-activemq-5.8.0/data/localhost/tmp_storage] started

activemq-A_1  |  INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/usr/local/apache-activemq-5.8.0/data/kahadb]

activemq-A_1  |  INFO | Database /usr/local/apache-activemq-5.8.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.8.0/data/kahadb/lock' could not be locked.

activemq-B_1  |  INFO | KahaDB is version 4

activemq-B_1  |  INFO | Recovering from the journal ...

activemq-B_1  |  INFO | Recovery replayed 1 operations from the journal in 0.022 seconds.

activemq-B_1  |  INFO | Apache ActiveMQ 5.8.0 (localhost, ID:activemq-B-41795-1662556582026-0:1) is starting

activemq-B_1  |  INFO | Listening for connections at: tcp://activemq-B:61616?maximumConnections=1000&wireformat.maxFrameSize=104857600

activemq-B_1  |  INFO | Connector openwire Started

activemq-B_1  |  INFO | Listening for connections at: amqp://activemq-B:5672?maximumConnections=1000&wireformat.maxFrameSize=104857600

activemq-B_1  |  INFO | Connector amqp Started

activemq-B_1  |  INFO | Listening for connections at: stomp://activemq-B:61612?transport.closeAsync=false

activemq-B_1  |  INFO | Connector stomp Started

activemq-B_1  |  INFO | Listening for connections at: stomp+nio://activemq-B:61613?transport.closeAsync=false

activemq-B_1  |  INFO | Connector stomp+nio Started

activemq-B_1  |  INFO | Apache ActiveMQ 5.8.0 (localhost, ID:activemq-B-41795-1662556582026-0:1) started

activemq-B_1  |  INFO | For help or more information please see: http://activemq.apache.org

activemq-B_1  | ERROR | Temporary Store limit is 51200 mb, whilst the temporary data directory: /usr/local/apache-activemq-5.8.0/data/localhost/tmp_storage only has 42398 mb of usable space

activemq-B_1  |  INFO | Web console type: embedded

activemq-B_1  |  INFO | ActiveMQ WebConsole initialized.

activemq-B_1  |  INFO | Initializing Spring FrameworkServlet 'dispatcher'

activemq-B_1  |  INFO | jolokia-agent: No access restrictor found at classpath:/jolokia-access.xml, access to all MBeans is allowed

activemq-A_1  |  INFO | Database /usr/local/apache-activemq-5.8.0/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/usr/local/apache-activemq-5.8.0/data/kahadb/lock' could not be locked.

根据启动的顺序不同，A、B 都有可能抢到锁对外提供服务，未抢到锁的实例日志一直会报无法解锁的错误，同时不会对外提供 Web Console 服务。使用 Python 的连接例子，发现 stomp.py 有一个问题不能很好的支持 failover。

Networks of Brokers

这是一种负载均衡的架构，各个 Broker 通过互相连接并共享队列信息，实现集群的负载均衡，即只要消费者连接到 Broker 网络中任意一台，就可以消费所有的消息。

这种方案主要关注 ActiveMQ 配置中的两个参数

TransportConnector，传输连接器参数主要控制服务端和客户端之间的通信方式
NetworkConnector，网络连接器参数主要控制服务端和服务端之间的通信

本文构建了这样一种架构，Broker1 作为与生产者沟通的节点，负责接收消息以及传递消费者反馈给生产者的消息，Broker2、Broker3作为与消费者沟通的节点，支持与大量的消费者建立链接。这种架构适用于消费者数量非常多，通常是超过1万个节点的情况。根据实际生产运行的情况，一台4C16G的虚拟机，一个ActiveMQ实例通常能够承受2000个左右的链接能保持长期稳定运行。

具体的 docker 配置如下，相关的文件可以从我的 Github 上下载。

version: "2.1"

services:

  activemq-broker1:

    image: cocowool/activemq:5.8.0

    hostname: activemq-broker1

    expose:

      - "8161"

      - "61611"

    ports:

      - "8161:8161"

      - "61611:61612"

    volumes:

      - ~/Projects/sh-valley/docker-conf/activemq-network/activemq-5.8.0-broker1.xml:/usr/local/apache-activemq-5.8.0/conf/activemq.xml

    networks:

      activemq-network:

        ipv4_address: 172.28.1.11

  activemq-broker2:

    image: cocowool/activemq:5.8.0

    hostname: activemq-broker2

    expose:

      - "8162"

      - "61612"

    ports:

      - "8162:8161"

      - "61612:61612"

    volumes:

      - ~/Projects/sh-valley/docker-conf/activemq-network/activemq-5.8.0-broker2.xml:/usr/local/apache-activemq-5.8.0/conf/activemq.xml

    networks:

      activemq-network:

        ipv4_address: 172.28.1.12

  activemq-broker3:

    image: cocowool/activemq:5.8.0

    hostname: activemq-broker3

    expose:

      - "8163"

      - "61613"

    ports:

      - "8163:8161"

      - "61613:61612"

    volumes:

      - ~/Projects/sh-valley/docker-conf/activemq-network/activemq-5.8.0-broker3.xml:/usr/local/apache-activemq-5.8.0/conf/activemq.xml

    networks:

      activemq-network:

        ipv4_address: 172.28.1.13

networks:

  activemq-network:

    ipam:

      config:

      - subnet: 172.28.1.0/24

执行 docker-compose up 命令可以启动三个 ActiveMQ 实例，然后分别通过 http://localhost:8161\8162\8163 可以查看三个实例的控制台，在 Broker2\Broker3 的 Network 页签下可以看到到 Broker1 的链接。我们在生产实践过程中发现过这个连接出问题的情况，此时 ActiveMQ 实例进程并无异常，这时就会影响连接到这个实例的消费者。

要测试这种架构的可用性，可以参考 Python 与 ActiveMQ 交互的一些例子中的代码示例，修改对应的端口地址就能看到效果。

这个架构可以配合 Master Slave 进一步优化，如下图所示：

生产者连接的实例采用 Master Slave Failover 方式，消费者连接的实例在配置 NetworkConnector 时也采用 Failover 模式，这样 Broker1 就提供了主从的高可用模式，进一步提高了整个集群应对风险的能力。

参考资料