Tomcat APR & Linux Optimization

一、简介

APR(Apache portable Run-time libraries)模式：简单理解，就是从操作系统级别解决异步IO问题，大幅度的提高服务器的处理和响应性能，也是Tomcat运行高并发应用的首选模式。

二、启用apr模式步聚

1、安装系统必要库和工具

yum install apr-devel

yum install openssl-devel

yum install gcc

yum install make

openssl库要求在0.9.7以上版本，APR要求在1.2以上版本

2、安装apr动态库

进入tomcat的bin目录，解压tomcat-native.tar.gz文件，并进入tomcat-native-1.2.7-src/native目录，执行./configure && make && make install 命令，动态库默认安装在/usr/local/apr/lib目录下

3、配置APR本地库到系统共享库搜索路径中

编辑$TOMCAT_HOME/bin/catalina.sh文件，在虚拟机启动参数JAVA_OPTS中添加java.library.path参数，指定apr库的路径

JAVA_OPTS="$JAVA_OPTS -Djava.library.path=/usr/local/apr/lib"

修改 server.xml 中的协议为 APR Protocol

<Connector port="8080" protocol="org.apache.coyote.http11.Http11AprProtocol"

connectionTimeout="20000"

redirectPort="8443" />

如果不想启用SSL，将server.xml中apr模式下ssl关闭即可:

<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" />

重启Tomcat, 日志显示

Aug 29, 2010 3:47:32 PM org.apache.catalina.core.AprLifecycleListener init

INFO: Loaded APR based Apache Tomcat Native library 1.1.20.

在最后的端口信息方面也可以看到http-apr-8080

⚠注意

Tomcat 7.x版本从7.0.30开始就支持APR, 之后版本的都可以使用APR模式

三、Linux Optimization

由于我们采用了linux服务器，所以优化内核参数也是一个非常重要的工作。以下为优化参考:

修改/etc/sysctl.cnf文件，在最后追加如下内容：

    net.core.netdev_max_backlog = 32768 

    net.core.somaxconn = 32768 

    net.core.wmem_default = 8388608 

    net.core.rmem_default = 8388608 

    net.core.rmem_max = 16777216 

    net.core.wmem_max = 16777216 

    net.ipv4.ip_local_port_range = 1024 65000 

    net.ipv4.route.gc_timeout = 100 

    net.ipv4.tcp_fin_timeout = 30 

    net.ipv4.tcp_keepalive_time = 1200 

    net.ipv4.tcp_timestamps = 0 

    net.ipv4.tcp_synack_retries = 2 

    net.ipv4.tcp_syn_retries = 2 

    net.ipv4.tcp_tw_recycle = 1 

    net.ipv4.tcp_tw_reuse = 1 

    net.ipv4.tcp_mem = 94500000 915000000 927000000 

    net.ipv4.tcp_max_orphans = 3276800 

    net.ipv4.tcp_max_syn_backlog = 65536

保存退出，执行sysctl -p生效

四、配置模式

APR

<Connector executor="tomcatThreadPool" port="8080" protocol="org.apache.coyote.http11.Http11AprProtocol"

     connectionTimeout="20000"

     redirectPort="8443"

     enableLookups="false"

     maxPostSize="10485760"

     URIEncoding="UTF-8"

     acceptCount="600"

     acceptorThreadCount="48"

     disableUploadTimeout="true"

     maxConnections="800"

     SSLEnabled="false"/>

<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"

        maxThreads="600" maxIdleTime="20000" minSpareThreads="400" prestartminSpareThreads="true" />

AJP

<Connector port="8009" protocol="AJP/1.3" maxThreads="1000" minSpareThreads="100" maxSpareThreads="500"

        acceptCount="700" connectionTimeout="20000" redirectPort="8443" />

2017-01-06 对原先基线和配置修改

机器配置为

CPU: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.6GHz 24core

内存: 128G

JDK Ver: 1.7.0_80

Tomcat Ver: 8.5.6

⚠注意使用Tomcat8时会出现如下Warning

警告 [localhost-startStop-1] org.apache.catalina.webresources.Cache.getResource Unable to add the resource at [/WEB-INF/classes/] to the cache because there was insufficient free space available after evicting expired cache entries - consider increasing the maximum size of the cache

解决办法在conf/conext.xml中加入如下配置

<Context>

    <WatchedResource>WEB-INF/web.xml</WatchedResource>

    <WatchedResource>${catalina.base}/conf/web.xml</WatchedResource>

    <Resources cachingAllowed="true" cacheMaxSize="100000" />

</Context>

在更换Tomcat之后首次调整ThreadPool采用NIO Pattern

<Connector executor="tomcatThreadPool" port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"

     connectionTimeout="20000"

     redirectPort="8443"

     enableLookups="false"

     maxPostSize="10485760"

     URIEncoding="UTF-8"

     acceptCount="600"

     acceptorThreadCount="48"

     disableUploadTimeout="true"

     maxConnections="800"

     SSLEnabled="false"/>

<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"

        maxThreads="600" maxIdleTime="20000" minSpareThreads="400" prestartminSpareThreads="true" />

acceptorThreadCount 数量为CPU Core的2倍

OSP Client Interface Conf Param: CentOS5.8、Timeout 20ms

验测线程50并发

验测线程200并发

Redis Response Interface Conf Param: CentOS5.8、Connection Redis

验测线程50并发

验测线程200并发

2017-01-10 切换服务器版本为CentOS6.6、Tomcat版本和ThreadPool配置参数不变，重新压测数据如下

OSP Client Interface Conf Param: CentOS6.6、Timeout 20ms

验测线程50并发

验测线程200并发

Redis Response Interface Conf Param: CentOS6.6、Connection Redis

验测线程50并发

验测线程200并发

此次调整操作系统版本6.6之后同比之前5.8版本中的压测结果均有30%左右的提升，但调用USP的OSP接口环比原生链接Redis的接口的QPS相差还在45%以上，前者低于后者。

2017-01-12

%TOMCAT_HOME%/bin/catalina.sh添加JVM参数

JAVA_OPTS="-server

                -Xms4096m

                -Xmx4096m

                -Xmn2048m

                -XX:MaxDirectMemorySize=4096m

                -XX:PermSize=256m

                -XX:MaxPermSize=512m

                -XX:ReservedCodeCacheSize=240M

                -XX:+UseCodeCacheFlushing"

OSP Client Interface

验测线程200并发

Redis Response Interface

验测线程200并发

两个接口QPS均有10%的提升，但环比差距仍为50%，原生访问Redis的接口好于访问OSP的接口

⚠注意

相同CPU: Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.6GHz 24core

相同OS Ver 和 Kernel: Linux version 2.6.32-504.el6.x86_64 (mockbuild@c6b9.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Wed Oct 15 04:27:16 UTC 2014

两个接口在上述条件下内存128G压测性能都好于32G有3倍之多