分布式-信息方式-ActiveMQ的消息存储持久化
ActiveMQ的消息存储持久化
■概述
ActiveMQ不仅支持 persistent和 non-persistent两种方式,还支持消息的
恢复( recovery)方式
PTP
Queue的存储是很简单的,就是一个FIFO的 Queue
图1-1
PUB/SUB
对于持久化订阅主题,每一个消费者将获得一个消息的复制
图1-2
有效的消息存储
ActiveMQ提供了一个插件式的消息存储,类似于消息的多点传播,主要实现了如下几种:
1:AMQ消息存储-基于文件的存储方式,是以前的默认消息存储
2: KahaDB消息存储-提供了容量的提升和恢复能力,是现在的默认存储方式
3:JDBC消息存储-消息基于JDBC存储的
4:Memory信息存储-基于内存的信息存储
KahaDB Message Store
概述
KahaDB是目前默认的存储方式,可用于任何场景,提高了性能和恢复能力。消息存储使用一个事
务日志和仅仅用一个索引文件来存储它所有的地址。
KahaDB是一个专门针对消息持久化的解决方案,它对典型的消息使用模式进行了优化。在Kaha
中,数据被追加到data logs中。当不再需要log文件中的数据的时候,log文件会被丢弃
KahaDB基本配置例子
<persistenceAdapter>
<kahaDB directory="{activemq.data}/kahadb"/>
</persistenceAdapter>
可用的属性有:
1: director: KahaDB存放的路径,默认值 activemq-data
2: indexWriteBatchsize:批量写入磁盘的索引page数量,默认值1000
3: indexCachesize:内存中缓存索引page的数量,默认值10000
4: enableIndexWriteAsync:是否异步写出索引,默认 false
5: journalMaxFileLength:设置每个消息data1og的大小,默认是32MB
6: enableJournalDiskSyncs:设置是否保证每个没有事务的内容,被同步写入磁盘,JMS持久化的时候需
要,默认为true
7: cleanupInterval: 在检查到不再使用的消息后,在具体删除消息前的时间,默认30000
8: checkpointInterval: checkpoint的间隔时间,默认5000
9: ignoreMissingJournalfiles:是否忽略丢失的消息日志文件,默认fa1se
10: checkForCorruptJournalFiles:在启动的时候,将会验证消息文件是否损坏,默认fa1se
11: checksumJournalFiles:是否为每个消息日志文件提供 checksum,默认 false
12: archiveDatalogs:是否移动文件到特定的路径,而不是删除它们,默认fa1se
13: directoryArchive:定义消息已经被消费过后,移动 data log到的路径,默认nu11
14: databaseLockedWailayDelay:获得数据库锁的等待时间( used by shared master/ slave),默认
10000
15: maxAsyncJobs:设置最大的可以存储的异步消息队列,默认值10000,可以和 concurrent
MessageProducers设置成一样的值
16: concurrentStoreAndDispatchTransactions:是否分发消息到客户端,同时事务存储消息,默认
true
17: concurrentstoreAndDispatchTopics:是否分发 Topic消息到客户端,同时进行存储,默认true
18: concurrentStoreAndDispatchQueues:是否分发 queue消息到客户端,同时进行存储,默认true
在Java中内嵌使用 Broker,使用 KahaDB的例子
package com.mq.test.activeMQ; import java.io.File; import org.apache.activemq.broker.BrokerService;
import org.apache.activemq.store.kahadb.KahaDBStore; public class KahaDB {
BrokerService createEmbeddedBroker() throws Exception {
BrokerService broker =new BrokerService();
File dataFileDir =new File( "target/amg-in-action/kahadb");
KahaDBStore kaha= new KahaDBStore();
kaha.setDirectory (dataFileDir);
// Using a bi gger journal file
kaha.setJournalMaxFileLength(*);
// small batch means more frequent and smaller writes
kaha.setIndexWriteBatchSize();
// do the Andex write in a separate thread
kaha.setEnableIndexWriteAsync(true);
broker.setPersistenceAdapter(kaha);
//create a transport connector
broker.addConnector("tcp://localhost:61616");
//start the broker
broker.start();
return broker;
}
}
AMQ Message Store概述
AMQ Message Store是 ActiveMQ5.0缺省的持久化存储,它是一个基于文件、事
务存储设计为快速消息存储的一个结构,该结构是以流的形式来进行消息交互的。
这种方式中, Messages被保存到 data logs中,同时被 reference store进行索
引以提高存取速度。 Date logs由一些单独的 data log文件组成,缺省的文件大小是
32M,如果某个消息的大小超过了 data log文件的大小,那么可以修改配置以增加
data log文件的大小。如果某个 data log文件中所有的消息都被成功消费了,那么这
个 data log文件将会被标记,以便在下一轮的清理中被删除或者归档。
AMQ Message Store配置示例
<broker brokerName=broker persistent=true useShutdownHook="false">
<persistenceAdapter>
<amqPersistenceAdapter directory="${activemq.basel/data" maxFileLength="32mb"/>
</persistenceAdapter>
</broker>
使用JDBC来持久化消息
ActiveMQ支持使用JDBC来持久化消息,预定义的表如下:
1:消息表,缺省表名为 ACTIVEMQ_MSGS, queue和 topic都存在里面i,结构如下:
图2-1
2: ACTIVEMQ_ACKS表存储持久订阅的信息和最后一个持久订阅接收的消息ID,结构如下
图2-2
3:锁定表,缺省表名为 ACTIVEMQ_LOCK,用来确保在某一时刻,只能有一个
ActiveMQ broker实例来访问数据库,结构如下:
图2-3
■使用JDBC来持久化消息的配置示例
图2-4
代码如下：
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- START SNIPPET: example -->
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd"> <!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean> <!-- Allows accessing the server log -->
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean> <!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}"> <destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see: http://activemq.apache.org/slow-consumer-handling.html -->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit=""/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy> <!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see: http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext> <!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see: http://activemq.apache.org/persistence.html
-->
<persistenceAdapter>
<!--
<kahaDB directory="${activemq.data}/kahadb"/>
-->
<jdbcPersistenceAdapter dataSource="#mysql-ds" createTablesOnStartup="true"/>
</persistenceAdapter> <!--
The systemUsage controls the maximum amount of space the broker will
use before disabling caching and/or slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage> <!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see: http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5673?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors> <!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks> </broker>
<bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost:3306/activemq?relaxAutoCommit=true"/>
<property name="username" value="root"/>
<property name="password" value=""/>
<property name="poolPreparedStatements" value="true"/>
</bean>
<!--
Enable web consoles, REST and Ajax APIs and demos
The web consoles requires by default login, you can disable this in the jetty.xml file Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
-->
<import resource="jetty.xml"/> </beans>
<!-- END SNIPPET: example -->
问题：报org.apache.commons.dbcp.BasicDataSource 找不到
解决：导入jar报lib中mysql-connector-java-5.1.35.jar
JDBC Message Store with ActiveMQ Journal
这种方式克服了 JDBC Store的不足,用快速的缓存写入技术,大大提高了性能。配置示例如下:
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- START SNIPPET: example -->
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd"> <!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean> <!-- Allows accessing the server log -->
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean> <!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}"> <destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to prevent
slow topic consumers to block producers and affect other consumers
by limiting the number of messages that are retained
For more information, see: http://activemq.apache.org/slow-consumer-handling.html -->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit=""/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy> <!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see: http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext> <!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see: http://activemq.apache.org/persistence.html
-->
<!--
<persistenceAdapter> <kahaDB directory="${activemq.data}/kahadb"/> <jdbcPersistenceAdapter dataSource="#mysql-ds" createTablesOnStartup="true"/> </persistenceAdapter>
-->
<persistenceFactory>
<journalPersistenceAdapterFactory
journalLogFiles=""
journalLogFileSize=""
useJournal="true"
useQuickJournal="true"
dataSource="#mysql-ds"
dataDirectory="activemq-data" />
</persistenceFactory>
<!--
The systemUsage controls the maximum amount of space the broker will
use before disabling caching and/or slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage> <!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see: http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<!-- DOS protection, limit concurrent connections to and frame size to 100MB -->
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="amqp" uri="amqp://0.0.0.0:5673?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
<transportConnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors> <!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks> </broker>
<bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost:3306/activemq?relaxAutoCommit=true"/>
<property name="username" value="root"/>
<property name="password" value=""/>
<property name="poolPreparedStatements" value="true"/>
</bean>
<!--
Enable web consoles, REST and Ajax APIs and demos
The web consoles requires by default login, you can disable this in the jetty.xml file Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
-->
<import resource="jetty.xml"/> </beans>
<!-- END SNIPPET: example -->
JDBC Store和 JDBC Message Store with ActiveMQ Journal的区别
1: Jdbc with journal的性能优表jdbc
2:Jdbc用于 master/ slave模式的数据库分享
3: Jdbc with journal不能用于 master/ slave模式
4:一般情况下,推荐使用 jdbc with journal
Memory Message Store
内存游息存储主要是存储所有的持久化的消息在内存中,这里没有动态的缓存存在,所以
你必须注意设置你的 broker所在的JM和内存限制
Memory Message Store配置示例
<beans>
<broker brokerName="test-broker" persistent= "false" xmlns="http://activemq.apacheorg/schema/core">
<transportConnectors>
<ransport Connector uri="tcp: //localhost: 61635"/>
</transportConnectors>
<broker>
</beans>
在Java中内嵌使用 Broker,使用 Memory的例子
package test.mq.topic1; import java.net.URI; import org.apache.activemq.broker.BrokerFactory;
import org.apache.activemq.broker.BrokerService; public class InnerBroker {
public static void main(String[] args) throws Exception {
BrokerService broker=new BrokerService();
broker.setPersistent(false);
broker.setUseJmx(true);
broker.addConnector("tcp://localhost:61616");
broker.start(); }
}
分布式-信息方式-ActiveMQ的消息存储持久化的更多相关文章
- 分布式-信息方式-ActiveMQ基础
ActiveMQ简介 ActiveMQ是什么ActiveMQ是Apache推出的,一款开源全支持JMS.1和J2EE1.4范的JMS Provider实现的信息中间件.(message oriente ...
- 分布式-信息方式-ActiveMQ的Destination高级特性3
虚拟destination用来创建逻辑destination,客户端可以通过它来生产和消费消息,它会把消息映射到物理destination. ActiveMQ支持2种方式: 1:虚拟主题(Virtua ...
- 分布式-信息方式-ActiveMQ的Message dispatch高级特性之(指针) Message cursors
Message dispatch高级特性之 Message cursors概述 ActiveMQ发送持久消息的典型处现方式是:当消息的消费者准备就绪时,消息发送系统把存储的 消息 ...
- 分布式-信息方式-ActiveMQ的Destination高级特性2
使用filtered destinations,在xml配置如下: <destinationInterceptors> <virtualDestinationInterceptor& ...
- 分布式-信息方式-ActiveMQ的Destination高级特性1
ActiveMQ的Destination高级特性 Destination高级特性----->Composite Destinations 组合队列Composite Destinations : ...
- 分布式-信息方式-ActiveMQ的集群
ActiveMQ的集群Queue consumer clusters ActiveMQ支持 Consumer对消息高可靠性的负载平衡消费,如果一个 Consumer死掉,该消 ...
- 分布式-信息方式-ActiveMQ静态网络连接的容错
容错的链接Failover Protocol 前面讲述的都是client配置链接到指定的 broker上.但是,如果 Broker的链接失败怎么办呢?此时, Client有两个选项:要么立刻死掉,要么 ...
- 分布式-信息方式-ActiveMQ的动态网络链接
ActiveMQ的动态网络链接多播协议 multicast ActiveMQ使用 Multicast协议将一个 Service和其他的 Broker的 Service连接起来,IPmulticast是 ...
- 分布式-信息方式-ActiveMQ静态网络连接信息回流功能
“丢失”的消息 有这样的场景, broker1和 broker2通过 netwoskconnector连接,一些 consumers连接到 broker1,消费 broker2上的消息.消息先被 br ...
随机推荐
- 使用Keras基于RCNN类模型的卫星/遥感地图图像语义分割
遥感数据集 1. UC Merced Land-Use Data Set 图像像素大小为256*256,总包含21类场景图像,每一类有100张,共2100张. http://weegee.vision ...
- JavaScript制作九九乘法表
<!DOCTYPE html> <html> <head> <meta charset="UTF-8"> <title> ...
- 07 Python中zip(),map(),filter(),reduce()用法
一. zip() zip() 函数用于将可迭代的对象作为参数,将对象中对应的元素打包成一个个元组,然后返回由这些元组组成的列表. 如果各个迭代器的元素个数不一致,则返回列表长度与最短的对象相同,利用 ...
- SpringBoot:spring boot使用Druid和监控配置
Druid是Java语言中最好的数据库连接池,并且能够提供强大的监控和扩展功能. Spring Boot默认的数据源是:org.apache.tomcat.jdbc.pool.DataSource 业 ...
- 转载：Cesium的Property机制总结
转自:https://www.jianshu.com/p/f0b47997224c 前言 Cesium官方教程中有一篇叫<空间数据可视化>(Visualizing Spatial Data ...
- IDEA修改Maven全局配置
在使用过程中发现,IDEA每次新建一个Project ,这个maven配置都会初始化默认的. 这里需要设置下全局配置: File -> Other Settings -> Settings ...
- Linux6上安装MySQL
MySQL安装包下载:https://www.mysql.com/downloads/ 然后选择: 把下载好的安装包传到服务器上的指定目录,然后解压: [root@master mysql]# tar ...
- IT经理工作职责
IT经理工作职责: 1. 管理公司信息技术以及计算机系统. 2. 确保公司信息技术是可访问的并且配备了现有的可用的硬件和软件. 3. 监控并且维护公司信息技术并确保能够得到最大化的使用 ...
- ps aux详解（进程状态说明）
linux上进程有5种状态: 1. 运行(正在运行或在运行队列中等待) 2. 中断(休眠中, 受阻, 在等待某个条件的形成或接受到信号) 3. 不可中断(收到信号不唤醒和不可运行, 进程必须等待直到有 ...
- Linux下如何查看CPU型号、个数、核数、逻辑CPU数、位数、发行版本、内核信息、内存、服务器生产厂家
[原文链接]:http://blog.csdn.net/mdx20072419/article/details/7767809 http://blog.chinaunix.net/uid-224252 ...