分布式-信息方式-ActiveMQ的消息存储持久化
ActiveMQ的消息存储持久化
■概述
ActiveMQ不仅支持 persistent和 non-persistent两种方式,还支持消息的
恢复( recovery)方式
PTP
Queue的存储是很简单的,就是一个FIFO的 Queue
图1-1
PUB/SUB
对于持久化订阅主题,每一个消费者将获得一个消息的复制
图1-2
有效的消息存储
ActiveMQ提供了一个插件式的消息存储,类似于消息的多点传播,主要实现了如下几种:
1:AMQ消息存储-基于文件的存储方式,是以前的默认消息存储
2: KahaDB消息存储-提供了容量的提升和恢复能力,是现在的默认存储方式
3:JDBC消息存储-消息基于JDBC存储的
4:Memory信息存储-基于内存的信息存储
KahaDB Message Store
概述
KahaDB是目前默认的存储方式,可用于任何场景,提高了性能和恢复能力。消息存储使用一个事
务日志和仅仅用一个索引文件来存储它所有的地址。
KahaDB是一个专门针对消息持久化的解决方案,它对典型的消息使用模式进行了优化。在Kaha
中,数据被追加到data logs中。当不再需要log文件中的数据的时候,log文件会被丢弃
KahaDB基本配置例子
<persistenceAdapter>
<kahaDB directory="{activemq.data}/kahadb"/>
</persistenceAdapter>
可用的属性有:
1: director: KahaDB存放的路径,默认值 activemq-data
2: indexWriteBatchsize:批量写入磁盘的索引page数量,默认值1000
3: indexCachesize:内存中缓存索引page的数量,默认值10000
4: enableIndexWriteAsync:是否异步写出索引,默认 false
5: journalMaxFileLength:设置每个消息data1og的大小,默认是32MB
6: enableJournalDiskSyncs:设置是否保证每个没有事务的内容,被同步写入磁盘,JMS持久化的时候需
要,默认为true
7: cleanupInterval: 在检查到不再使用的消息后,在具体删除消息前的时间,默认30000
8: checkpointInterval: checkpoint的间隔时间,默认5000
9: ignoreMissingJournalfiles:是否忽略丢失的消息日志文件,默认fa1se
10: checkForCorruptJournalFiles:在启动的时候,将会验证消息文件是否损坏,默认fa1se
11: checksumJournalFiles:是否为每个消息日志文件提供 checksum,默认 false
12: archiveDatalogs:是否移动文件到特定的路径,而不是删除它们,默认fa1se
13: directoryArchive:定义消息已经被消费过后,移动 data log到的路径,默认nu11
14: databaseLockedWailayDelay:获得数据库锁的等待时间( used by shared master/ slave),默认
10000
15: maxAsyncJobs:设置最大的可以存储的异步消息队列,默认值10000,可以和 concurrent
MessageProducers设置成一样的值
16: concurrentStoreAndDispatchTransactions:是否分发消息到客户端,同时事务存储消息,默认
true
17: concurrentstoreAndDispatchTopics:是否分发 Topic消息到客户端,同时进行存储,默认true
18: concurrentStoreAndDispatchQueues:是否分发 queue消息到客户端,同时进行存储,默认true
在Java中内嵌使用 Broker,使用 KahaDB的例子
package com.mq.test.activeMQ; import java.io.File; import org.apache.activemq.broker.BrokerService;
import org.apache.activemq.store.kahadb.KahaDBStore; public class KahaDB {
BrokerService createEmbeddedBroker() throws Exception {
BrokerService broker =new BrokerService();
File dataFileDir =new File( "target/amg-in-action/kahadb");
KahaDBStore kaha= new KahaDBStore();
kaha.setDirectory (dataFileDir);
// Using a bi gger journal file
kaha.setJournalMaxFileLength(*);
// small batch means more frequent and smaller writes
kaha.setIndexWriteBatchSize();
// do the Andex write in a separate thread
kaha.setEnableIndexWriteAsync(true);
broker.setPersistenceAdapter(kaha);
//create a transport connector
broker.addConnector("tcp://localhost:61616");
//start the broker
broker.start();
return broker;
}
}
AMQ Message Store概述
AMQ Message Store是 ActiveMQ5.0缺省的持久化存储,它是一个基于文件、事
务存储设计为快速消息存储的一个结构,该结构是以流的形式来进行消息交互的。
这种方式中, Messages被保存到 data logs中,同时被 reference store进行索
引以提高存取速度。 Date logs由一些单独的 data log文件组成,缺省的文件大小是
32M,如果某个消息的大小超过了 data log文件的大小,那么可以修改配置以增加
data log文件的大小。如果某个 data log文件中所有的消息都被成功消费了,那么这
个 data log文件将会被标记,以便在下一轮的清理中被删除或者归档。
AMQ Message Store配置示例
<broker brokerName=broker persistent=true useShutdownHook="false">
<persistenceAdapter>
<amqPersistenceAdapter directory="${activemq.basel/data" maxFileLength="32mb"/>
</persistenceAdapter>
</broker>
使用JDBC来持久化消息
ActiveMQ支持使用JDBC来持久化消息,预定义的表如下:
1:消息表,缺省表名为 ACTIVEMQ_MSGS, queue和 topic都存在里面i,结构如下:
图2-1
2: ACTIVEMQ_ACKS表存储持久订阅的信息和最后一个持久订阅接收的消息ID,结构如下
图2-2
3:锁定表,缺省表名为 ACTIVEMQ_LOCK,用来确保在某一时刻,只能有一个
ActiveMQ broker实例来访问数据库,结构如下:
图2-3
■使用JDBC来持久化消息的配置示例
图2-4
代码如下:
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- START SNIPPET: example -->
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd"> <!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean> <!-- Allows accessing the server log -->
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean> <!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}"> <destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see: http://activemq.apache.org/slow-consumer-handling.html -->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit=""/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy> <!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see: http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext> <!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see: http://activemq.apache.org/persistence.html
-->
<persistenceAdapter>
<!--
<kahaDB directory="${activemq.data}/kahadb"/>
-->
<jdbcPersistenceAdapter dataSource="#mysql-ds" createTablesOnStartup="true"/>
</persistenceAdapter> <!--
The systemUsage controls the maximum amount of space the broker will
use before disabling caching and/or slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage> <!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see: http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5673?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors> <!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks> </broker>
<bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost:3306/activemq?relaxAutoCommit=true"/>
<property name="username" value="root"/>
<property name="password" value=""/>
<property name="poolPreparedStatements" value="true"/>
</bean>
<!--
Enable web consoles, REST and Ajax APIs and demos
The web consoles requires by default login, you can disable this in the jetty.xml file Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
-->
<import resource="jetty.xml"/> </beans>
<!-- END SNIPPET: example -->
问题:报org.apache.commons.dbcp.BasicDataSource 找不到
解决:导入jar报lib中mysql-connector-java-5.1.35.jar
JDBC Message Store with ActiveMQ Journal
这种方式克服了 JDBC Store的不足,用快速的缓存写入技术,大大提高了性能。配置示例如下:
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- START SNIPPET: example -->
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd"> <!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean> <!-- Allows accessing the server log -->
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean> <!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}"> <destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see: http://activemq.apache.org/slow-consumer-handling.html -->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit=""/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy> <!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see: http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext> <!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see: http://activemq.apache.org/persistence.html
-->
<!--
<persistenceAdapter> <kahaDB directory="${activemq.data}/kahadb"/> <jdbcPersistenceAdapter dataSource="#mysql-ds" createTablesOnStartup="true"/> </persistenceAdapter>
-->
<persistenceFactory>
<journalPersistenceAdapterFactory
journalLogFiles=""
journalLogFileSize=""
useJournal="true"
useQuickJournal="true"
dataSource="#mysql-ds"
dataDirectory="activemq-data" />
</persistenceFactory>
<!--
The systemUsage controls the maximum amount of space the broker will
use before disabling caching and/or slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage> <!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see: http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5673?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors> <!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks> </broker>
<bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost:3306/activemq?relaxAutoCommit=true"/>
<property name="username" value="root"/>
<property name="password" value=""/>
<property name="poolPreparedStatements" value="true"/>
</bean>
<!--
Enable web consoles, REST and Ajax APIs and demos
The web consoles requires by default login, you can disable this in the jetty.xml file Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
-->
<import resource="jetty.xml"/> </beans>
<!-- END SNIPPET: example -->
JDBC Store和 JDBC Message Store with ActiveMQ Journal的区别
1: Jdbc with journal的性能优表jdbc
2:Jdbc用于 master/ slave模式的数据库分享
3: Jdbc with journal不能用于 master/ slave模式
4:一般情况下,推荐使用 jdbc with journal
Memory Message Store
内存游息存储主要是存储所有的持久化的消息在内存中,这里没有动态的缓存存在,所以
你必须注意设置你的 broker所在的JM和内存限制
Memory Message Store配置示例
<beans>
<broker brokerName="test-broker" persistent= "false" xmlns="http://activemq.apacheorg/schema/core">
<transportConnectors>
<ransport Connector uri="tcp: //localhost: 61635"/>
</transportConnectors>
<broker>
</beans>
在Java中内嵌使用 Broker,使用 Memory的例子
package test.mq.topic1; import java.net.URI; import org.apache.activemq.broker.BrokerFactory;
import org.apache.activemq.broker.BrokerService; public class InnerBroker {
public static void main(String[] args) throws Exception {
BrokerService broker=new BrokerService();
broker.setPersistent(false);
broker.setUseJmx(true);
broker.addConnector("tcp://localhost:61616");
broker.start(); }
}
分布式-信息方式-ActiveMQ的消息存储持久化的更多相关文章
- 分布式-信息方式-ActiveMQ基础
ActiveMQ简介 ActiveMQ是什么ActiveMQ是Apache推出的,一款开源全支持JMS.1和J2EE1.4范的JMS Provider实现的信息中间件.(message oriente ...
- 分布式-信息方式-ActiveMQ的Destination高级特性3
虚拟destination用来创建逻辑destination,客户端可以通过它来生产和消费消息,它会把消息映射到物理destination. ActiveMQ支持2种方式: 1:虚拟主题(Virtua ...
- 分布式-信息方式-ActiveMQ的Message dispatch高级特性之(指针) Message cursors
Message dispatch高级特性之 Message cursors概述 ActiveMQ发送持久消息的典型处现方式是:当消息的消费者准备就绪时,消息发送系统把存储的 消息 ...
- 分布式-信息方式-ActiveMQ的Destination高级特性2
使用filtered destinations,在xml配置如下: <destinationInterceptors> <virtualDestinationInterceptor& ...
- 分布式-信息方式-ActiveMQ的Destination高级特性1
ActiveMQ的Destination高级特性 Destination高级特性----->Composite Destinations 组合队列Composite Destinations : ...
- 分布式-信息方式-ActiveMQ的集群
ActiveMQ的集群Queue consumer clusters ActiveMQ支持 Consumer对消息高可靠性的负载平衡消费,如果一个 Consumer死掉,该消 ...
- 分布式-信息方式-ActiveMQ静态网络连接的容错
容错的链接Failover Protocol 前面讲述的都是client配置链接到指定的 broker上.但是,如果 Broker的链接失败怎么办呢?此时, Client有两个选项:要么立刻死掉,要么 ...
- 分布式-信息方式-ActiveMQ的动态网络链接
ActiveMQ的动态网络链接多播协议 multicast ActiveMQ使用 Multicast协议将一个 Service和其他的 Broker的 Service连接起来,IPmulticast是 ...
- 分布式-信息方式-ActiveMQ静态网络连接信息回流功能
“丢失”的消息 有这样的场景, broker1和 broker2通过 netwoskconnector连接,一些 consumers连接到 broker1,消费 broker2上的消息.消息先被 br ...
随机推荐
- 108、如何使用 Secret? (Swarm15)
参考https://www.cnblogs.com/CloudMan6/p/8068057.html 我们经常要想容器传递敏感信息,最常见的就是密码.比如: docker run -e MYS ...
- 简单了解journalctl
journalctl 命令 journalctl是什么以及作用? journalctl 用来查询 systemd-journald 服务收集到的日志.systemd-journald 服务是 syst ...
- k8s自签TLS证书
自签TLS证书 TLS证书用于进行通信使用,k8s组件需要的证书有: 第一步:安装证书生成工具cfssl 在这之前需要先建立一个目录来存放安装的工具mkdir ssl,后面将安装的工具移动到各自的目录 ...
- css折叠表格
1.html <div class="custom-fold-table"> <table cellpadding="0" cellspaci ...
- dedecms 调用内容页分页
arc.archives.class.php $TRUEfilename = $this->GetTruePath().$fileFirst."_".$i.".&q ...
- PAT Basic 1007 素数对猜想 (20 分)
让我们定义dn为:dn=pn+1−pn,其中pi是第i个素数.显然有d1=1,且对于n>1有dn是偶数.“素数对猜想”认为“存在无穷多对相邻且差为2的素 ...
- trigger添加及表达式
创建触发器 点击Configuration(配置) → Hosts(主机) 点击hosts(主机)相关行的trigger 点击右上角的创建触发器(create trigger) name : 触发器名 ...
- HihoCoder1087Hamiltonian Cycle(DP状态压缩)
时间限制:10000ms 单点时限:1000ms 内存限制:256MB 描述 Given a directed graph containing n vertice (numbered from 1 ...
- JAVA笔记7-Object类之toString方法和equals方法
位于rt.jar\java\lang\Object.class Object类是所有Java类的根基类(Java中只有一个根基类,C++中有多个).如果在类声明中未使用extends关键字指明其基类, ...
- Vue中如何使用axios请求跨域数据
1.axios不支持jsonp,因为axios的作者觉得jsonp不太友好,推荐用CORS方式更为干净: 2.在使用axios发送请求时,服务器端设置 res.header("Access- ...