性能调优案例分享:jvm crash的原因 2
3.core dump分析
有了core dump文件,接下来要做的就是通过命令去解析此文件,定位具体问题了,主要有以下三个命令:
(1)先执行gdb $JAVA_HOME$/bin/java core-java-16427-1325846515,再执行bt,输出结果如下:
Loaded symbols for /opt/.../oracle/lib/libnnz10.so
Reading symbols from /opt/.../install/oracle/10.2.0.3/lib/libociei.so...(no debugging symbols found)...done.
Loaded symbols for /opt/.../oracle/lib/libociei.so
Reading symbols from /usr/lib/gconv/GB18030.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/gconv/GB18030.so
#0 0x4662051e in ttcrd2r ()
from /opt/.../oracle/lib/libclntsh.so.10.1
(gdb) bt
#0 0x4662051e in ttcrd2r () from /opt/.../oracle/lib/libclntsh.so.10.1
#1 0x46617212 in ttcrbur () from /opt/.../oracle/lib/libclntsh.so.10.1
#2 0x466176e6 in ttcbur () from /opt/.../oracle/lib/libclntsh.so.10.1
#3 0x465ba8c7 in ttcdrv () from /opt/.../oracle/lib/libclntsh.so.10.1
#4 0x464a1bc8 in nioqwa () from /opt/.../oracle/lib/libclntsh.so.10.1
#5 0x4630d618 in upirtrc () from /opt/.../oracle/lib/libclntsh.so.10.1
#6 0x462825f6 in kpurcsc () from /opt/.../oracle/lib/libclntsh.so.10.1
#7 0x46236a0d in kpuexecv8 () from /opt/.../oracle/lib/libclntsh.so.10.1
#8 0x46238ec4 in kpuexec () from /opt/.../oracle/lib/libclntsh.so.10.1
#9 0x463121b6 in OCIStmtExecute () from /opt/.../oracle/lib/libclntsh.so.10.1
#10 0x46e725aa in Java_oracle_jdbc_driver_T2CConnection_t2cClearAllApplicationContext ()
from /opt/.../install/oracle/10.2.0.3/lib/libocijdbc10.so
#11 0x46e7580f in Java_oracle_jdbc_driver_T2CStatement_t2cDefineExecuteFetch ()
from /opt/.../install/oracle/10.2.0.3/lib/libocijdbc10.so
#12 0xb125a8b8 in ?? ()
#13 0x47e01bc8 in ?? ()
#14 0x47d46bb0 in ?? ()
#15 0x47d46c40 in ?? ()
#16 0x4babf774 in ?? ()
#17 0x00000000 in ?? ()
通过这个debug命令可以查看到应用crash时正在做什么,这里是在获取jdbc连接,但这个并不代表问题就出在这里,有可能这只是个结果,起因还需要进一步排查。
(2)光有上述信息还不足以定位出问题在哪,可以通过命令jstack $JAVA_HOME$/bin/java core-java-16427-1325846515获取应用crash时的线程堆栈,输出结果如下:
Thread 11586: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() @bci=2, line=474 (Compiled frame)
- edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingDeque.takeFirst() @bci=20, line=389 (Compiled frame)
- edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingDeque.take() @bci=1, line=577 (Compiled frame)
- edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=674 (Compiled frame)
Error occurred during stack walking:
Thread 10924: (state = IN_NATIVE)
- oracle.jdbc.driver.T2CStatement.t2cDefineExecuteFetch(oracle.jdbc.driver.OracleStatement, long, int, int, int, int, boolean, boolean, byte[], int, byte, int, int, short[], int, byte[], char[], int, int, short[], byte[], int, int, boolean, boolean, oracle.jdbc.driver.Accessor[], byte[][][], long[], byte[], int, char[], int, short[], int) @bci=0 (Interpreted frame)
- oracle.jdbc.driver.T2CPreparedStatement.doDefineExecuteFetch() @bci=190, line=878 (Interpreted frame)
- oracle.jdbc.driver.T2CPreparedStatement.executeForRows(boolean) @bci=34, line=760 (Interpreted frame)
- oracle.jdbc.driver.OracleStatement.executeMaybeDescribe() @bci=154, line=1062 (Interpreted frame)
- oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout() @bci=102, line=1126 (Interpreted frame)
- oracle.jdbc.driver.OraclePreparedStatement.executeInternal() @bci=94, line=3339 (Interpreted frame)
- oracle.jdbc.driver.OraclePreparedStatement.execute() @bci=17, line=3445 (Interpreted frame)
- org.jboss.resource.adapter.jdbc.CachedPreparedStatement.execute() @bci=4, line=216 (Interpreted frame)
- org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.execute() @bci=12, line=209 (Compiled frame)
- com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(com.ibatis.sqlmap.engine.scope.RequestScope, java.sql.Connection, java.lang.String, java.lang.Object[], int, int, com.ibatis.sqlmap.engine.mapping.statement.RowHandlerCallback) @bci=135, line=180 (Compiled frame)
- com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(com.ibatis.sqlmap.engine.scope.RequestScope, java.sql.Connection, java.lang.String, java.lang.Object[], int, int, com.ibatis.sqlmap.engine.mapping.statement.RowHandlerCallback) @bci=15, line=205 (Interpreted frame)
- com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(com.ibatis.sqlmap.engine.scope.RequestScope, java.sql.Connection, java.lang.Object, java.lang.Object, com.ibatis.sqlmap.client.event.RowHandler, int, int) @bci=169, line=173 (Compiled frame)
- com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryForObject(com.ibatis.sqlmap.engine.scope.RequestScope, com.ibatis.sqlmap.engine.transaction.Transaction, java.lang.Object, java.lang.Object) @bci=28, line=104 (Compiled frame)
- com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(com.ibatis.sqlmap.engine.scope.SessionScope, java.lang.String, java.lang.Object, java.lang.Object) @bci=58, line=561 (Compiled frame)
- com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryForObject(com.ibatis.sqlmap.engine.scope.SessionScope, java.lang.String, java.lang.Object) @bci=5, line=536 (Interpreted frame)
- com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryForObject(java.lang.String, java.lang.Object) @bci=10, line=93 (Interpreted frame)
- org.springframework.orm.ibatis.SqlMapClientTemplate$1.doInSqlMapClient(com.ibatis.sqlmap.client.SqlMapExecutor) @bci=9, line=273 (Interpreted frame)
- org.springframework.orm.ibatis.SqlMapClientTemplate.execute(org.springframework.orm.ibatis.SqlMapClientCallback) @bci=242, line=209 (Compiled frame)
- org.springframework.orm.ibatis.SqlMapClientTemplate.queryForObject(java.lang.String, java.lang.Object) @bci=11, line=271 (Compiled frame)
......
- org.springframework.web.servlet.DispatcherServlet.doService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=219, line=809 (Interpreted frame)
- org.springframework.web.servlet.FrameworkServlet.processRequest(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=10, line=571 (Compiled frame)
- org.springframework.web.servlet.FrameworkServlet.doPost(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=3, line=511 (Interpreted frame)
- javax.servlet.http.HttpServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=139, line=717 (Interpreted frame)
- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=810 (Compiled frame)
- org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=173 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.invoke(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=396, line=672 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.processRequest(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=64, line=463 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.doForward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=469, line=398 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.forward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=55, line=301 (Interpreted frame)
- org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(java.util.Map, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=244, line=240 (Interpreted frame)
- org.springframework.web.servlet.view.AbstractView.render(java.util.Map, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=157, line=252 (Interpreted frame)
- org.springframework.web.servlet.DispatcherServlet.render(org.springframework.web.servlet.ModelAndView, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=225, line=1173 (Compiled frame)
- org.springframework.web.servlet.DispatcherServlet.doService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=219, line=809 (Interpreted frame)
- org.springframework.web.servlet.FrameworkServlet.processRequest(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=10, line=571 (Compiled frame)
- org.springframework.web.servlet.FrameworkServlet.doPost(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=3, line=511 (Interpreted frame)
- javax.servlet.http.HttpServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=139, line=717 (Interpreted frame)
- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=810 (Compiled frame)
- org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=173 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.invoke(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=396, line=672 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.processRequest(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=64, line=463 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.doForward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=469, line=398 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.forward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=55, line=301 (Interpreted frame)
- org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(java.util.Map, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=244, line=240 (Interpreted frame)
- org.springframework.web.servlet.view.AbstractView.render(java.util.Map, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=157, line=252 (Interpreted frame)
- org.springframework.web.servlet.DispatcherServlet.render(org.springframework.web.servlet.ModelAndView, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=225, line=1173 (Compiled frame)
- org.springframework.web.servlet.DispatcherServlet.doService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=219, line=809 (Interpreted frame)
- org.springframework.web.servlet.FrameworkServlet.processRequest(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=10, line=571 (Compiled frame)
- org.springframework.web.servlet.FrameworkServlet.doPost(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=3, line=511 (Interpreted frame)
- javax.servlet.http.HttpServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=139, line=717 (Interpreted frame)
- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=810 (Compiled frame)
- org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=173 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.invoke(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=396, line=672 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.processRequest(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=64, line=463 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.doForward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=469, line=398 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.forward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=55, line=301 (Interpreted frame)
- org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(java.util.Map, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=244, line=240 (Interpreted frame)
- org.springframework.web.servlet.view.AbstractView.render(java.util.Map, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=157, line=252 (Interpreted frame)
- org.springframework.web.servlet.DispatcherServlet.render(org.springframework.web.servlet.ModelAndView, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=225, line=1173 (Compiled frame)
- org.springframework.web.servlet.DispatcherServlet.doService(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=219, line=809 (Interpreted frame)
- org.springframework.web.servlet.FrameworkServlet.processRequest(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=10, line=571 (Compiled frame)
- org.springframework.web.servlet.FrameworkServlet.doPost(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=3, line=511 (Interpreted frame)
- javax.servlet.http.HttpServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=139, line=717 (Interpreted frame)
- javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=30, line=810 (Compiled frame)
- org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=101, line=173 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.invoke(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=396, line=672 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.processRequest(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=64, line=463 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.doForward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=469, line=398 (Interpreted frame)
- org.apache.catalina.core.ApplicationDispatcher.forward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=55, line=301 (Interpreted frame)
......
通过这个线程堆栈可以看出问题根本原因是一直在forward,代码出现死循环,总的forward的次数有100多次,直到线程堆栈溢出。这也能说明前 面通过gdb命令查看到的获取jdbc连接仅仅是最近的一次循环正在执行的操作而已,不是真正的起因。这个线程的死循环forward同样影响到了其他的 线程,导致出现Error occurred during stack walking错误,线程一直处于BLOCKED状态,总之线程堆栈对分析问题原因是非常有帮助的。
(3).可以通过命令jmap $JAVA_HOME$/bin/java core-java-16427-1325846515查看到应用的内存使用,结果如下:
using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 1342177280 (1280.0MB)
NewSize = 134217728 (128.0MB)
MaxNewSize = 134217728 (128.0MB)
OldSize = 402653184 (384.0MB)
NewRatio = 15
SurvivorRatio = 1024
PermSize = 100663296 (96.0MB)
MaxPermSize = 100663296 (96.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 134152192 (127.9375MB)
used = 22525464 (21.481956481933594MB)
free = 111626728 (106.4555435180664MB)
16.790977220856742% used
Eden Space:
capacity = 134086656 (127.875MB)
used = 22525464 (21.481956481933594MB)
free = 111561192 (106.3930435180664MB)
16.799183954591275% used
From Space:
capacity = 65536 (0.0625MB)
used = 0 (0.0MB)
free = 65536 (0.0625MB)
0.0% used
To Space:
capacity = 65536 (0.0625MB)
used = 0 (0.0MB)
free = 65536 (0.0625MB)
0.0% used
concurrent mark-sweep generation:
capacity = 1207959552 (1152.0MB)
used = 764300224 (728.8934936523438MB)
free = 443659328 (423.10650634765625MB)
Exception in thread "main" java.lang.OutOfMemoryError: requested 4096 bytes for jbyte in /BUILD_AREA/jdk1.5.0_08/hotspot/src/share/vm/prims/jni.cpp. Out of swap space?
最后一行出现了OOM,这是因为dump出来的文件太大,32位机器支持的最大内存也就2G,所以在解析时出现OOM问题,可以放到64位服务器上进行解析。
4.常见的core dump原因
造成coredump的原因很多,常见的有内存访问越界,非法指针,堆栈溢出等等,也可以通过发送操作系统信号中断应用,比如kill -SIGSEGV <pid>,这是一个无效存储信号。如果想手动生成core dump文件,可以通过gdb -p <pid>生成。
性能调优案例分享:jvm crash的原因 2的更多相关文章
- 性能调优案例分享:jvm crash的原因 1
性能调优案例分享:jvm crash的原因 poptest是国内唯一一家培养测试开发工程师的培训机构,以学员能胜任自动化测试,性能测试,测试工具开发等工作为目标.如果对课程感兴趣,请大家咨询qq: ...
- 性能调优案例分享:Mysql的cpu过高
性能调优案例分享:Mysql的cpu过高 问题:一个系统,Mysql数据库,数据量变大之后.mysql的cpu占用率很高,一个测试端访问服务器时mysql的cpu占用率为15% ,6个测试端连服务 ...
- hbase性能调优案例
hbase性能调优案例 1.人员-角色 人员有多个角色 角色优先级 角色有多个人员 人员 删除添加角色 角色 可以添加删除人员 人员 角色 删除添加 设计思路 person表 ...
- JVM性能调优2:JVM性能调优参数整理
序号 参数名 说明 JDK 默认值 使用过 1 JVM执行模式 2 -client-server 设置该JVM运行与Client 或者Server Hotspot模式,这两种模式从本质上来说是在JVM ...
- 一文带你深入了解JVM性能调优以及对JVM调优的全面总结
目录 JVM调优 概念 基本垃圾回收算法 垃圾回收面临的问题 分代垃圾回收详述1 分代垃圾回收详述2 典型配置举例1 典型配置举例2 新一代的垃圾回收算法 调优方法 反思 一.JVM调优的一些概念 数 ...
- Oracle 性能调优案例(代码级别)
业务案例一: 业务:千万记录表中查询出50条符合条件的记录. 现象:oracle部署时跨机器,业务取得数据耗时10ms.造成业务性能不达标. 为了突出主题,对于异常分支,均已省略. 对于通常写法, o ...
- SQL Server ->> 性能调优案例之 -- 包含递归查询的视图导致整个查询语句性能下降
有个语句最近性能下降很厉害,原本1秒就可以查询完毕的事情现在居然需要3-4分钟. 首先我的做法是先快速找出导致整个语句下降的元凶.在这个例子里面查询语句有3个JOIN字句,我通过删除某一个JOIN节点 ...
- JVM | 第1部分:自动内存管理与性能调优《深入理解 Java 虚拟机》
目录 前言 1. 自动内存管理 1.1 JVM运行时数据区 1.2 Java 内存结构 1.3 HotSpot 虚拟机创建对象 1.4 HotSpot 虚拟机的对象内存布局 1.5 访问对象 2. 垃 ...
- 如何合理的规划一次jvm性能调优
https://blog.csdn.net/miracle_8/article/details/78347172 摘要: JVM性能调优涉及到方方面面的取舍,往往是牵一发而动全身,需要全盘考虑各方面的 ...
随机推荐
- nodejs中异步
nodejs中的异步 1 nodejs 中的异步存在吗? 现在有点 javascript 基础的人都在听说过 nodejs ,而只要与 javascript 打交到人都会用或者是将要使用 nodejs ...
- 【排序算法】快速排序算法 Java实现
快速排序是C.R.A.Hoare于1962年提出的一种划分交换排序.它采用了一种分治的策略,通常称其为分治法(Divide-and-ConquerMethod). 基本思想 先从数组中找出一个数作为基 ...
- (译文)Python中的staticmethod与classmethod
原文是stackoverflow的一则高票回答,原文链接 可能之前也有人翻译过,但是刚好自己也有疑惑,所以搬运一下,个人水平有限所以可能翻译存在误差,欢迎指正(如侵删). 尽管classmethod和 ...
- Thread类常用方法
Thread类构造方法: 1.Thread(): 2.Thread(String name): 3.Thread(Runable r): 4.Thread(Runable r, String name ...
- 在Vue中通过自定义指令获取元素
vue.js 是数据绑定的框架,大部分情况下我们都不需要直接操作 DOM Element,但在某些时候,我们还是有获取DOM Element的需求的: 在 vue.js 中,获取某个DOM Eleme ...
- lxd容器之GPU发现和加载
lxd gpu设备发现: // /dev/nvidia[0-9]+ type nvidiaGpuCards struct { path string major int minor int id st ...
- 【Java基础】HashMap工作原理
HashMap Hash table based implementation of the Map interface. This implementation provides all of th ...
- 初识Windous程序
文本框Label MaxLength 设置输入文本最大字符 Multiline 表示是否输入多行文本 passwodechar 指示在文本框显示的字符,而不是实际内容 ReadeOnly 表示是否可 ...
- “幸福企业”定义-参观“MES项目”有感
作为公司的员工,总是想在一个自己满意的企业里面发展.作为企业主,虽不能天天将“回报社会”挂在嘴上,但凡是有抱负的,还是希望自己的部下“以厂为家的”.然而劳资双方的矛盾总是让双方感觉互有亏欠.这种不信任 ...
- C#中运算符的应用
c#中的运算符(只说我自己熟悉和常用的)一.算数运算:加 +减 -乘 *除 / (整型的除法是不算小数点的,也就是说结果是整数,余数舍去了,求余数看下面)求余数 % 加减乘好理解,整型的除法和求余数因 ...