记一次CPU飙升的问题排查
1.背景
通过公司监控工具监控，发现公司某个应用cpu利用率达到120%，也就是说这个应用自己单独占用一个cpu使用，为何占用这么高？让我们一起排查一下
2.开始排查
(1) 第一步获取cpu过高应用的进程信息 (top 命令监控)
(2)通过进程获取占用cpu过高的线程（Top -H -p ）
(3)通过jstack命令导出线程的堆栈信息
"http-bio-7051-exec-28" #33168 daemon prio=5 os_prio=0 tid=0x00007fe343f77000 nid=0xaa1 runnable [0x00007fe329287000]
java.lang.Thread.State: RUNNABLE
at java.lang.StringBuilder.append(StringBuilder.java:214)
at sun.util.calendar.CalendarUtils.sprintf0d(CalendarUtils.java:171)
at java.util.Date.toString(Date.java:1040)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at com.jiuyescm.oms.wm.odo.action.service.impl.CommonOmsOrderSendAction.buildAppointKey(CommonOmsOrderSendAction.java:72)
at com.jiuyescm.oms.wm.odo.controller.OmsDo2BController.queryStorage(OmsDo2BController.java:1007)
at sun.reflect.GeneratedMethodAccessor2011.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.bstek.dorado.data.method.MethodAutoMatchingUtils.invokeMethod(MethodAutoMatchingUtils.java:602)
at com.bstek.dorado.data.method.MethodAutoMatchingUtils.invokeMethod(MethodAutoMatchingUtils.java:695)
at com.bstek.dorado.data.provider.manager.DataProviderInterceptorInvoker.invokeInterceptorByParamType(DataProviderInterceptorInvoker.java:542)
at com.bstek.dorado.data.provider.manager.DataProviderInterceptorInvoker.invoke(DataProviderInterceptorInvoker.java:157)
at com.bstek.dorado.util.proxy.ChainedMethodInvocation.proceed(ChainedMethodInvocation.java:73)
at com.bstek.dorado.data.provider.DataProviderGetResultMethodInterceptor.invokeGetResult(DataProviderGetResultMethodInterceptor.java:29)
at com.bstek.dorado.data.provider.AbstractDataProviderGetResultMethodInterceptor.invoke(AbstractDataProviderGetResultMethodInterceptor.java:55)
at com.bstek.dorado.util.proxy.MethodInterceptorDispatcher.invoke(MethodInterceptorDispatcher.java:241)
at com.bstek.dorado.data.provider.DirectDataProvider_$$_jvst349_17.getResult(DirectDataProvider_$$_jvst349_17.java)
at com.bstek.dorado.view.service.LoadDataServiceProcessor.doExecute(LoadDataServiceProcessor.java:258)
at com.bstek.dorado.view.service.DataServiceProcessorSupport.execute(DataServiceProcessorSupport.java:133)
at com.bstek.dorado.view.resolver.ViewServiceInvoker.invoke(ViewServiceInvoker.java:32)
at com.bstek.dorado.view.resolver.ViewServiceInvoker_$$_jvst349_16._d5invoke(ViewServiceInvoker_$$_jvst349_16.java)
at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.bstek.dorado.util.proxy.MethodInterceptorDispatcher$FinalJavassistMethodInterceptor.invoke(MethodInterceptorDispatcher.java:96)
at com.bstek.dorado.util.proxy.ChainedMethodInvocation.proceed(ChainedMethodInvocation.java:73)
at com.bstek.dorado.util.proxy.MethodInterceptorDispatcher$FinalAopallianceMethodInterceptor.invoke(MethodInterceptorDispatcher.java:67)
at com.bstek.dorado.util.proxy.ChainedMethodInvocation.proceed(ChainedMethodInvocation.java:73)
at com.jiuyescm.cfm.dorado.interceptor.SessionValidationRemoteServiceMethodInterceptor.invoke(SessionValidationRemoteServiceMethodInterceptor.java:33)
at com.bstek.dorado.view.service.AbstractRemoteServiceMethodInterceptor.invoke(AbstractRemoteServiceMethodInterceptor.java:30)
at com.bstek.dorado.util.proxy.ChainedMethodInvocation.proceed(ChainedMethodInvocation.java:73)
at com.bstek.bdf2.core.exception.interceptor.AjaxMethodInterceptor.invoke(AjaxMethodInterceptor.java:26)
at com.bstek.dorado.util.proxy.MethodInterceptorDispatcher.invoke(MethodInterceptorDispatcher.java:170)
at com.bstek.dorado.util.proxy.MethodInterceptorDispatcher.invoke(MethodInterceptorDispatcher.java:241)
at com.bstek.dorado.view.resolver.ViewServiceInvoker_$$_jvst349_16.invoke(ViewServiceInvoker_$$_jvst349_16.java)
at com.bstek.dorado.view.resolver.ViewServiceResolver.processTask(ViewServiceResolver.java:154)
at com.bstek.dorado.view.resolver.ViewServiceResolver.execute(ViewServiceResolver.java:244)
at com.bstek.dorado.web.resolver.AbstractTextualResolver.doHandleRequest(AbstractTextualResolver.java:128)
at com.bstek.dorado.web.resolver.WebContextSupportedController.handleRequestInternal(WebContextSupportedController.java:70)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:154)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:945)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:876)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:931)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:833)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:807)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.alibaba.druid.support.http.WebStatFilter.doFilter(WebStatFilter.java:123)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:108)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at com.bstek.bdf2.core.security.filter.ControllerFilter.doFilter(ControllerFilter.java:64)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:100)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at com.bstek.bdf2.core.security.filter.PreAuthenticatedProcessingFilter.doFilter(PreAuthenticatedProcessingFilter.java:41)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at com.bstek.bdf2.core.security.filter.ContextFilter.doFilter(ContextFilter.java:36)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at com.r.tomcat.session.management.RequestSessionHandlerValve.invoke(RequestSessionHandlerValve.java:30)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
- locked <0x0000000724439528> (a org.apache.tomcat.util.net.SocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
从以上日志中找出最魁祸首
at java.lang.StringBuilder.append(StringBuilder.java:214)
at sun.util.calendar.CalendarUtils.sprintf0d(CalendarUtils.java:171)
at java.util.Date.toString(Date.java:1040)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at com.jiuyescm.oms.wm.odo.action.service.impl.CommonOmsOrderSendAction.buildAppointKey(CommonOmsOrderSendAction.java:72)
at com.jiuyescm.oms.wm.odo.controller.OmsDo2BController.queryStorage(OmsDo2BController.java:1007)
(4)结合代码寻找真相
3.总结
跟我们的堆栈完全可以对应的上，性能耗费在循环调用字符串拼接这里。在这里要说一下字符串使用操作符"+"拼接的问题，如果你用”+”来连接固定长度的字符串，在性能上会有问题，但是如果你是在循环中来”+”多个串的话，性能将指数倍的下降。假设有一个字符串，我们将对这个字符串做大量循环拼接操作，使用”+”的话将得到最低的性能。网上有很多对比的文章随便一找一大把，如果不信自己可以写个demo调用1000次然后看自己cpu的情况。
记一次CPU飙升的问题排查的更多相关文章
- 面试连环炮系列（八）：服务器CPU飙升100%怎么排查
服务器CPU飙升100%怎么排查 执行"top"命令,查看当前进程CPU占用的实时情况,PID列是进程号,确定是哪个应用程序的问题. 如果是Java应用导致的,怎么定位故障原因 执 ...
- 线上CPU飙升100%问题排查
本文转载自线上CPU飙升100%问题排查 引子 对于互联网公司,线上CPU飙升的问题很常见(例如某个活动开始,流量突然飙升时),按照本文的步骤排查,基本1分钟即可搞定!特此整理排查方法一篇,供大家参考 ...
- 记一次CPU飙升BUG
图文地址:https://mp.weixin.qq.com/s?__biz=Mzg3NjEzODQ4NQ==&mid=2247483690&idx=1&sn=7c926f400 ...
- 线上CPU飙升100%问题排查，一篇足矣
一.引子 对于互联网公司,线上CPU飙升的问题很常见(例如某个活动开始,流量突然飙升时),按照本文的步骤排查,基本1分钟即可搞定!特此整理排查方法一篇,供大家参考讨论提高. 二.问题复现 线上系统突然 ...
- 记一次CPU使用100%问题排查
需求 前端同事说测试环境的服务接口查起来很慢,很不稳定,不是个别接口,而是大量接口. 情况分析 由于是在测试环境联调,没有多少用户量.第一步:先去服务器看看资源的使用情况.使用top命令,查看cpu的 ...
- 【原创】记一次MySQL大表高并发写入引发CPU飙升的排障过程
目录 一．故障现象... 1 二．初步分析... 2 三．排障过程... 2 1．排查是否QPS或insert并发请求上升导致问题发生... 2 2．排查是否锁资源等待或block导致了insert变 ...
- CPU飙升排查
怎么排查CPU飙升 线上有些系统,本来跑的好好的,突然有一天就会出现报警,CPU使用率飙升,然后重启之后就好了.例如,多线程操作一个线程不安全的list往往就会出现这种现象.那么怎么定位到具体的代码范 ...
- 【转】Java程序CPU飙升问题排查方法
windows环境下cpu飙升问题 线上某台runtime机器(windows Server)cpu报警,这种情况初步就是代码里面死循环了,先把机器下线了保证不再有新的任务分配进来,然而cpu使用依然 ...
- 记一次Linux server偶发CPU飙升问题的跟进与解决
背景 进入6月后,随着一个主要功能版本api的上线,服务端的QPS翻了一倍,平时服务器的CPU使用稳定在30%上下,高峰期则在60%上下,但是偶尔会有单台机器出现持续数分钟突然飙到90%以上,导致大量 ...
- Java服务器内存过高&CPU过高问题排查
一.内存过高 1.内存过高一般有两种情况:内存溢出和内存泄漏 (1)内存溢出:程序分配的内存超出物理机的内存大小,导致无法继续分配内存,出现OOM报错 (2)内存泄漏:不再使用的对象一直占据着内存不释 ...
随机推荐
- 神经网络之卷积篇：详解边缘检测示例（Edge detection example）
详解边缘检测示例 卷积运算是卷积神经网络最基本的组成部分,使用边缘检测作为入门样例.在这个博客中,会看到卷积是如何进行运算的. 在之前的博客中,说过神经网络的前几层是如何检测边缘的,然后,后面的层有可 ...
- VUE—CLI学习
https://cli.vuejs.org/zh/guide/creating-a-project.html#vue-create 需要的自己来看吧 关于旧版本 Vue CLI 的包名称由 vue-c ...
- SMU Spring 2023 Trial Contest Round 1
A. Prepend and Append 用ans记录n的值,然后双指针从前后判断是否一个为0一个为1,是的话则ans-2,否则退出循环即可. #include<bits/stdc++.h&g ...
- 树莓派CM4(三): 定制自己的树莓派镜像
1. 镜像下载 使用树莓派最新的镜像Raspberry Pi OS Lite,内核版本6.6 下载链接 https://downloads.raspberrypi.com/raspios_lite_a ...
- [CSP-S 2023] 消消乐 & CF1223F 题解
LG9753 CF1223F 我们称一个字符串是可消除的,当且仅当可以对这个字符串进行若干次操作,使之成为一个空字符串.其中每次操作可以从字符串中删除两个相邻的相同字符,操作后剩余字符串会拼接在一起. ...
- games101 作业1及作业2分析及解决 详解透视矩阵
games101 作业1及作业2分析及解决 去年的时候把games101的课程以及作业完成,但是整个过程比较粗略,也借助了不少外界的力量(doge),于是最近准备抽几天集中再把作业(1-7)过一遍,常 ...
- JavaScript设计模式样例十七 —— 迭代器模式
迭代器模式(Itrator Pattern) 定义:用于顺序访问集合对象的元素,不需要知道集合对象的底层表示.目的:提供一种方法顺序访问一个聚合对象中各个元素, 而又无须暴露该对象的内部表示.场景:$ ...
- 折腾 Quickwit，Rust 编写的分布式搜索引擎-官方教程
快速上手 在本快速入门指南中,我们将安装 Quickwit,创建一个索引,添加文档,最后执行搜索查询.本指南中使用的所有 Quickwit 命令都在 CLI 参考文档 中进行了记录. https:// ...
- 安装nvm，并通过nvm安装nodejs
转载请注明出处: 1.安装nvm 打开终端,然后运行以下命令来下载并安装nvm: curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39 ...
- AI驱动的PlantUML：快速生成专业级UML图表
**对于程序员来说,编写验收文档中的各种UML图是最让人头疼的事情之一,相信各位读者对此深有体会.** 本文将探讨如何利用AI驱动的PlantUML来快速生成专业级别的UML图表,从而减轻这一负担. ...