分布式-信息方式-ActiveMQ的消息存储持久化
ActiveMQ的消息存储持久化
■概述
ActiveMQ不仅支持 persistent和 non-persistent两种方式,还支持消息的
恢复( recovery)方式
PTP
Queue的存储是很简单的,就是一个FIFO的 Queue
图1-1
PUB/SUB
对于持久化订阅主题,每一个消费者将获得一个消息的复制
图1-2
有效的消息存储
ActiveMQ提供了一个插件式的消息存储,类似于消息的多点传播,主要实现了如下几种:
1:AMQ消息存储-基于文件的存储方式,是以前的默认消息存储
2: KahaDB消息存储-提供了容量的提升和恢复能力,是现在的默认存储方式
3:JDBC消息存储-消息基于JDBC存储的
4:Memory信息存储-基于内存的信息存储
KahaDB Message Store
概述
KahaDB是目前默认的存储方式,可用于任何场景,提高了性能和恢复能力。消息存储使用一个事
务日志和仅仅用一个索引文件来存储它所有的地址。
KahaDB是一个专门针对消息持久化的解决方案,它对典型的消息使用模式进行了优化。在Kaha
中,数据被追加到data logs中。当不再需要log文件中的数据的时候,log文件会被丢弃
KahaDB基本配置例子
<persistenceAdapter>
<kahaDB directory="{activemq.data}/kahadb"/>
</persistenceAdapter>
可用的属性有:
1: director: KahaDB存放的路径,默认值 activemq-data
2: indexWriteBatchsize:批量写入磁盘的索引page数量,默认值1000
3: indexCachesize:内存中缓存索引page的数量,默认值10000
4: enableIndexWriteAsync:是否异步写出索引,默认 false
5: journalMaxFileLength:设置每个消息data1og的大小,默认是32MB
6: enableJournalDiskSyncs:设置是否保证每个没有事务的内容,被同步写入磁盘,JMS持久化的时候需
要,默认为true
7: cleanupInterval: 在检查到不再使用的消息后,在具体删除消息前的时间,默认30000
8: checkpointInterval: checkpoint的间隔时间,默认5000
9: ignoreMissingJournalfiles:是否忽略丢失的消息日志文件,默认fa1se
10: checkForCorruptJournalFiles:在启动的时候,将会验证消息文件是否损坏,默认fa1se
11: checksumJournalFiles:是否为每个消息日志文件提供 checksum,默认 false
12: archiveDatalogs:是否移动文件到特定的路径,而不是删除它们,默认fa1se
13: directoryArchive:定义消息已经被消费过后,移动 data log到的路径,默认nu11
14: databaseLockedWailayDelay:获得数据库锁的等待时间( used by shared master/ slave),默认
10000
15: maxAsyncJobs:设置最大的可以存储的异步消息队列,默认值10000,可以和 concurrent
MessageProducers设置成一样的值
16: concurrentStoreAndDispatchTransactions:是否分发消息到客户端,同时事务存储消息,默认
true
17: concurrentstoreAndDispatchTopics:是否分发 Topic消息到客户端,同时进行存储,默认true
18: concurrentStoreAndDispatchQueues:是否分发 queue消息到客户端,同时进行存储,默认true
在Java中内嵌使用 Broker,使用 KahaDB的例子
package com.mq.test.activeMQ; import java.io.File; import org.apache.activemq.broker.BrokerService;
import org.apache.activemq.store.kahadb.KahaDBStore; public class KahaDB {
BrokerService createEmbeddedBroker() throws Exception {
BrokerService broker =new BrokerService();
File dataFileDir =new File( "target/amg-in-action/kahadb");
KahaDBStore kaha= new KahaDBStore();
kaha.setDirectory (dataFileDir);
// Using a bi gger journal file
kaha.setJournalMaxFileLength(*);
// small batch means more frequent and smaller writes
kaha.setIndexWriteBatchSize();
// do the Andex write in a separate thread
kaha.setEnableIndexWriteAsync(true);
broker.setPersistenceAdapter(kaha);
//create a transport connector
broker.addConnector("tcp://localhost:61616");
//start the broker
broker.start();
return broker;
}
}
AMQ Message Store概述
AMQ Message Store是 ActiveMQ5.0缺省的持久化存储,它是一个基于文件、事
务存储设计为快速消息存储的一个结构,该结构是以流的形式来进行消息交互的。
这种方式中, Messages被保存到 data logs中,同时被 reference store进行索
引以提高存取速度。 Date logs由一些单独的 data log文件组成,缺省的文件大小是
32M,如果某个消息的大小超过了 data log文件的大小,那么可以修改配置以增加
data log文件的大小。如果某个 data log文件中所有的消息都被成功消费了,那么这
个 data log文件将会被标记,以便在下一轮的清理中被删除或者归档。
AMQ Message Store配置示例
<broker brokerName=broker persistent=true useShutdownHook="false">
<persistenceAdapter>
<amqPersistenceAdapter directory="${activemq.basel/data" maxFileLength="32mb"/>
</persistenceAdapter>
</broker>
使用JDBC来持久化消息
ActiveMQ支持使用JDBC来持久化消息,预定义的表如下:
1:消息表,缺省表名为 ACTIVEMQ_MSGS, queue和 topic都存在里面i,结构如下:
图2-1
2: ACTIVEMQ_ACKS表存储持久订阅的信息和最后一个持久订阅接收的消息ID,结构如下
图2-2
3:锁定表,缺省表名为 ACTIVEMQ_LOCK,用来确保在某一时刻,只能有一个
ActiveMQ broker实例来访问数据库,结构如下:
图2-3
■使用JDBC来持久化消息的配置示例
图2-4
代码如下：
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- START SNIPPET: example -->
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd"> <!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean> <!-- Allows accessing the server log -->
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean> <!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}"> <destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see: http://activemq.apache.org/slow-consumer-handling.html -->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit=""/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy> <!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see: http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext> <!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see: http://activemq.apache.org/persistence.html
-->
<persistenceAdapter>
<!--
<kahaDB directory="${activemq.data}/kahadb"/>
-->
<jdbcPersistenceAdapter dataSource="#mysql-ds" createTablesOnStartup="true"/>
</persistenceAdapter> <!--
The systemUsage controls the maximum amount of space the broker will
use before disabling caching and/or slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage> <!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see: http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5673?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors> <!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks> </broker>
<bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost:3306/activemq?relaxAutoCommit=true"/>
<property name="username" value="root"/>
<property name="password" value=""/>
<property name="poolPreparedStatements" value="true"/>
</bean>
<!--
Enable web consoles, REST and Ajax APIs and demos
The web consoles requires by default login, you can disable this in the jetty.xml file Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
-->
<import resource="jetty.xml"/> </beans>
<!-- END SNIPPET: example -->
问题：报org.apache.commons.dbcp.BasicDataSource 找不到
解决：导入jar报lib中mysql-connector-java-5.1.35.jar
JDBC Message Store with ActiveMQ Journal
这种方式克服了 JDBC Store的不足,用快速的缓存写入技术,大大提高了性能。配置示例如下:
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- START SNIPPET: example -->
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd"> <!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean> <!-- Allows accessing the server log -->
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean> <!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}"> <destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see: http://activemq.apache.org/slow-consumer-handling.html -->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit=""/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy> <!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see: http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext> <!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see: http://activemq.apache.org/persistence.html
-->
<!--
<persistenceAdapter> <kahaDB directory="${activemq.data}/kahadb"/> <jdbcPersistenceAdapter dataSource="#mysql-ds" createTablesOnStartup="true"/> </persistenceAdapter>
-->
<persistenceFactory>
<journalPersistenceAdapterFactory
journalLogFiles=""
journalLogFileSize=""
useJournal="true"
useQuickJournal="true"
dataSource="#mysql-ds"
dataDirectory="activemq-data" />
</persistenceFactory>
<!--
The systemUsage controls the maximum amount of space the broker will
use before disabling caching and/or slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage> <!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see: http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5673?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors> <!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks> </broker>
<bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost:3306/activemq?relaxAutoCommit=true"/>
<property name="username" value="root"/>
<property name="password" value=""/>
<property name="poolPreparedStatements" value="true"/>
</bean>
<!--
Enable web consoles, REST and Ajax APIs and demos
The web consoles requires by default login, you can disable this in the jetty.xml file Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
-->
<import resource="jetty.xml"/> </beans>
<!-- END SNIPPET: example -->
JDBC Store和 JDBC Message Store with ActiveMQ Journal的区别
1: Jdbc with journal的性能优表jdbc
2:Jdbc用于 master/ slave模式的数据库分享
3: Jdbc with journal不能用于 master/ slave模式
4:一般情况下,推荐使用 jdbc with journal
Memory Message Store
内存游息存储主要是存储所有的持久化的消息在内存中,这里没有动态的缓存存在,所以
你必须注意设置你的 broker所在的JM和内存限制
Memory Message Store配置示例
<beans>
<broker brokerName="test-broker" persistent= "false" xmlns="http://activemq.apacheorg/schema/core">
<transportConnectors>
<ransport Connector uri="tcp: //localhost: 61635"/>
</transportConnectors>
<broker>
</beans>
在Java中内嵌使用 Broker,使用 Memory的例子
package test.mq.topic1; import java.net.URI; import org.apache.activemq.broker.BrokerFactory;
import org.apache.activemq.broker.BrokerService; public class InnerBroker {
public static void main(String[] args) throws Exception {
BrokerService broker=new BrokerService();
broker.setPersistent(false);
broker.setUseJmx(true);
broker.addConnector("tcp://localhost:61616");
broker.start(); }
}
分布式-信息方式-ActiveMQ的消息存储持久化的更多相关文章
- 分布式-信息方式-ActiveMQ基础
ActiveMQ简介 ActiveMQ是什么ActiveMQ是Apache推出的,一款开源全支持JMS.1和J2EE1.4范的JMS Provider实现的信息中间件.(message oriente ...
- 分布式-信息方式-ActiveMQ的Destination高级特性3
虚拟destination用来创建逻辑destination,客户端可以通过它来生产和消费消息,它会把消息映射到物理destination. ActiveMQ支持2种方式: 1:虚拟主题(Virtua ...
- 分布式-信息方式-ActiveMQ的Message dispatch高级特性之(指针) Message cursors
Message dispatch高级特性之 Message cursors概述 ActiveMQ发送持久消息的典型处现方式是:当消息的消费者准备就绪时,消息发送系统把存储的 消息 ...
- 分布式-信息方式-ActiveMQ的Destination高级特性2
使用filtered destinations,在xml配置如下: <destinationInterceptors> <virtualDestinationInterceptor& ...
- 分布式-信息方式-ActiveMQ的Destination高级特性1
ActiveMQ的Destination高级特性 Destination高级特性----->Composite Destinations 组合队列Composite Destinations : ...
- 分布式-信息方式-ActiveMQ的集群
ActiveMQ的集群Queue consumer clusters ActiveMQ支持 Consumer对消息高可靠性的负载平衡消费,如果一个 Consumer死掉,该消 ...
- 分布式-信息方式-ActiveMQ静态网络连接的容错
容错的链接Failover Protocol 前面讲述的都是client配置链接到指定的 broker上.但是,如果 Broker的链接失败怎么办呢?此时, Client有两个选项:要么立刻死掉,要么 ...
- 分布式-信息方式-ActiveMQ的动态网络链接
ActiveMQ的动态网络链接多播协议 multicast ActiveMQ使用 Multicast协议将一个 Service和其他的 Broker的 Service连接起来,IPmulticast是 ...
- 分布式-信息方式-ActiveMQ静态网络连接信息回流功能
“丢失”的消息 有这样的场景, broker1和 broker2通过 netwoskconnector连接,一些 consumers连接到 broker1,消费 broker2上的消息.消息先被 br ...
随机推荐
- 为什么说Python采用的是基于值的内存管理模式？
Python中的变量并不直接存储值,而是存储了值的内存地址或者引用,假如为不同变量赋值为相同值,这个值在内存中只有一份,多个变量指向同一块内存地址.
- 15、R语言聚类树的绘图原理
聚类广泛用于数据分析.去年研究了一下R语言聚类树的绘图原理.以芯片分析为例,我们来给一些样品做聚类分析.聚类的方法有很多种,我们选择Pearson距离.ward方法. 选择的样品有: "GS ...
- (转)Cvte提前批
1. 加密解密了解么?几种算法,讲一下你了解的(链接) 算法选择:对称加密AES,非对称加密: ECC,消息摘要: MD5,数字签名:DSA 常见加密算法 1.DES(Data Encryption ...
- iview发布到IIS 路由问题
<?xml version="1.0" encoding="UTF-8"?> <configuration> <system.we ...
- wex5 file文件存储
在js中需要引入file的cordova包 require("cordova!cordova-plugin-file"); 如果要存到手机的根目录下,在Native文件夹的对应项目 ...
- linux复习5
权限----------------- r //100 = 4 //文件 :读取内容, //文件夹:是查看文件夹的内容 w //文件 :写数据到文件 //文件夹:增删文件. //10 = 2 x // ...
- Webpack loaderUtils.parseQuery()
https://blog.256pages.com/webpack-loaderutils-parsequery/
- openlayers之天地图为底图 叠加其他底图 加载遇到的各种报错
今天以前使用天地图为底图时,map里source的定义:http://t3.tianditu.com~~~ 今天突然报跨域的错误,原有地址访问受限,可是key值是有的 最后发现就是前半段的原因 将前半 ...
- ubuntu自己定义环境变量,替代常用的操作命令
问题背景是这样的,因为自己会经常用自己的用户链接服务器,比如自己的用户是yongjie,然后服务器的ip是162.105.97.31 所以经常执行的命令是ssh yongjie@162.105.97. ...
- appium 自动化测试环境搭建
最近再学习appium,把学习的过程记录下来,以防止到时候 换个电脑就不知道这么安装搭建appium环境了. 环境搭建: 0.JDK环境是必备的,这里大家自行百度, 1.安装 node 环境,前辈 ...