5 Ways to Use Log Data to Analyze System Performance--reference
Recently we looked across some of the most common behaviors that our community of 25,000 users looked for in their logs with a particular focus on web server logs. In fact our research identified the top 15 web server tags and alerts created by our customers – you can read more about these from in ourcommunity insights section – and you can also easily create tags or alerts based on the patterns to identify these behaviours in your systems.
This week we are focusing on performance analysis using log data. Again we looked across our community of over 25,000 users and identified 5 ways in which people use log data to analyze system performance. As always customer data was anonymized and privacy protected. Over the course of the next week we will be diving into each of these area’s in more detail and will feature customers first hand accounts of how they are using logs to help identify and resolve such issues in their systems.
Our research looked at more than 200k patterns from across our Community to identify important events in their log data. With a particular focus on performance related issues we identified the following 5 areas as trending and common across our user base.:
1. Slow Response Times:Response times are one of the most common and useful performance measures that are available from your log data. They give you an immediate understanding of how long a request is taking to be returned. For example web server logs can give you insight into how long a request takes to return a response to a client device. This can include time taken for the different components behind your web server (application servers, DBs) to process the request so it can give an immediate view as to how well your application is performing. Recording response times from the client device/broswer can give you an even more complete picture since it also captures page load time in the app/browser as well as network latency.
A good rule of thumb when measuring response times is to follow the 3 response time limitsas outlined by Jakob Nielsen in his publication on ‘Usability Engineering’ back in 1993 that is still relevant today. In short 0.1 second is about the limit for having the user feel that the system is reacting instantaneously, 1.0 second is about the limit for the user’s flow of thought to stay uninterrupted, and 10 seconds is about the limit for keeping the user’s attention focused on the dialogue.
Slow response time patterns almost always follow the pattern below:
- response_time>X
Where response_time is the field value representing the server or client’s response and ‘X’ is a threshold, which if exceeded, you want the event to be highlighted or a notification to be sent so that you and your team are aware that somebody is having a poor user experience.
2. Memory Issues and Garbage Collection: Outofmemory errors can be pretty catastrophic when they occur as they often result in the application crashing due to lack of resources. Thus you want to know about these when they occur and creating tags and generating notifications via alerts when these events occur is always recommended.
However a leading indicator of outofmemory issues can be your garbage collection behavior, thus tracking this and getting notified if heap used vs free heap space is over a particular threshold, or if garbage collection is taking a long time can be particularly useful and can often point you in the direction of memory leaks. Identifying a memory leak before an out of memory exception can be the difference between a major system outage and a simple server restart until the issue is patched.
Furthermore slow or long garbage collection can also be one of the reasons for user’s experiencing slow application behavior as during garbage collection your system can slow down or in some situations it blocks until garbage collection is complete (e.g. with ‘stop the world’ garbage collection).
Below are some examples of common patterns used to identify some of the memory related issues outlined above:
- Out of memory
- exceeds memory limit
- memory leak detected
- java.lang.OutOfMemoryError
- System.OutOfMemoryException
- memwatch:leak: Ended heapDiff
- GC AND stats
3. Deadlocks and Threading Issues
Deadlocks can occur in many shapes and sizes and can have pretty bad effects when they occur – everywhere from bringing your system to a complete halt to simply slowing it down. In short, a deadlock is a situation in which two or more competing actions are each waiting for the other to finish, and thus neither ever does. For example, we say that a set of processes or threads is deadlocked when each thread is waiting for an event that only another process in the set can cause.
Not surprisingly deadlocks feature as one of our top 5 performance related issues that our users write patterns to detect in their systems.
Most deadlock patterns simply contain the keyword ‘deadlock’, but some of the common patterns follow the following structure:
- ‘deadlock’
- ‘Deadlock found when trying to get lock’
- ‘Unexpected error while processing request: deadlock;’
4. High Resource Usage (CPU/Disk/ Network)
In many cases a slow down in system performance may not be as a result of any major software flaw, but can be a simple case of the load on your system increasing, yet not having increased resources available to deal with this. Tracking resource usage can allow you to see when you require additional capacity such that you can kick off more server instances for example.
Example patterns used when analysing resource usage:
- metric=/CPUUtilization/ AND minimum>X
- cpu>X
- disk>X
- disk is at or near capacity
- not enough space on the disk
- java.io.IOException: No space left on device
- insufficient bandwidth
5. Database Issues and Slow Queries
Knowing when a query failed can be useful as it allows you to identify situations when a request may have returned without the relevant data and thus helps you identify when users are not getting the data they need. However more subtle issues can be when a user is getting the correct results but the results are taking a long time to return and while technically the system may be fine and bug free a slow user experience may be hurting your top line.
Tracking slow queries allows you to track how your DB queries are performing. Setting acceptable thresholds for query time and reporting on anything that exceeds these thresholds can help you quickly identify when your users experience is being effected.
Example patterns:
- SqlException
- SQL Timeout
- Long query
- Slow query
- WARNING: Query took longer than X
- Query_time > X
As always let us know if you think we have left out any important issues that you like to track in your logs. To start tracking your own system performance, create a free accountand include these patterns listed above to automatically create tags and alerts relevant for your system.
Published at DZone with permission of Trevor Parsons, author and DZone MVB. (source)
http://java.dzone.com/articles/5-ways-use-log-data-analyze?mz=110215-high-perf
5 Ways to Use Log Data to Analyze System Performance--reference的更多相关文章
- 学习笔记:Analyze MySQL Performance及慢日志的开启
Table of Contents Analyze MySQL PerformanceTuningSlow queries and Slowlog Brought to you by Rick Jam ...
- 错误描述:请求“System.Data.SqlClient.SqlClientPermission, System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089”类型的权限已失败
错误描述:请求“System.Data.SqlClient.SqlClientPermission, System.Data, Version=2.0.0.0, Culture=neutral, Pu ...
- System.Data.Dbtype转换为System.Data.SqlDbType
最近在做一些OM Mapping的准备工作,新学了一招. 如果要将System.Data.Dbtype转换为System.Data.SqlDbType,以前以为要写Switch Case语句.其实有很 ...
- 【译】Using .NET for Apache Spark to Analyze Log Data
.NET for Spark可用于处理成批数据.实时流.机器学习和ad-hoc查询.在这篇博客文章中,我们将探讨如何使用.NET for Spark执行一个非常流行的大数据任务,即日志分析. 1 什么 ...
- Note: OBLIVIATE: A Data Oblivious File System for Intel SGX
OBLIVIATE redesigned ORAM for SGX filesystem operations for confuse access patterns to protect user ...
- 错误 1 “System.Data.DataRow.DataRow(System.Data.DataRowBuilder)”不可访问,因为它受保护级别限制
new DataRow 的方式: DataTable pDataTable = new DataTable(); DataRow pRow = new DataRow(); 正确的方式: DataRo ...
- Chapter 20: Diagnostics
WHAT'S IN THIS CHAPTER?n Code contractsn Tracingn Event loggingn Performance monitoringWROX.COM CODE ...
- SQL Server ErrorLog
SQL Server 使用ErrorLog记录SQL Server启动和运行过程中的信息,具体信息参考:<SQLSERVER errorlog讲解>.通常来说,ErrorLog是指SQL ...
- GoldenGate配置(二)之双向复制配置
GoldenGate配置(二)之双向复制配置 环境: Item Source System Target System Platform Red Hat Enterprise Linux Serve ...
随机推荐
- javascript 字符串转数字
//把str转换为数字的方式,想起很久以前的一个面试题,说字符转数字的方式有哪些,现在想了想 var str1='4.88',str2='4.8xx'; console.log(parseInt(st ...
- spring 属性配置细节
1.使用构造器注入属性值可以指定参数的位置和参数的类型!以区分重载的构造器.例如:<constructor-arg value="" type="java.lang ...
- 【网络流24题】No.11(航空路线问题 最长不相交路径 最大费用流)
[题意] 给定一张航空图, 图中顶点代表城市, 边代表 2 城市间的直通航线. 现要求找出一条满足下述限制条件的且途经城市最多的旅行路线.(1) 从最西端城市出发,单向从西向东途经若干城市到达最东端城 ...
- 使用solrj进行DIH操作
背景说明:在一个项目中需要将Mongodb中的数据导入到solr中完成搜索.在solr中Mysql数据库有对应的DIH包,可以通过配置sql语句完成数据的导入.Mongodb下也有开源的工具用来实现数 ...
- Qt的事件模型(5种使用办法,通常重新实现event handler即可。只有定义控件才需要管理信号的发射)
Qt的事件模型 1.事件的概念 应用程序对象将系统消息接收为 Qt 事件.应用程序可以按照不同的粒度对事件加以监控.过滤并做出响应. 在 Qt 中,事件是指从 QEvent继承 的对象.Qt将事件发送 ...
- springboot + devtools(热部署)
技术介绍 devtools:是boot的一个热部署工具,当我们修改了classpath下的文件(包括类文件.属性文件.页面等)时,会重新启动应用(由于其采用的双类加载器机制,这个启动会非常快,如果发现 ...
- 【HDOJ】1166 敌兵布阵
线段树. #include <stdio.h> #define maxn 55555 ]; void PushUP(int rt) { sums[rt] = sums[rt<< ...
- Angular.js vs Ember.js
Angular.js 拥抱 HTML/CSS Misko Hevery(Angular.js的开发者之一)回答了这一问题,他的主要观点如下: 在HTML中加入太多逻辑不是好做法.Angular.js只 ...
- 使用 Visual Studio 生成通用的 XAML 应用程序 (Windows Phone 和 Windows 通用程序)
在Build会议上,我们发布了新的版本---Windows Phone 8.1. Windows 8.1 平台.作为开发人员,这意味着您现在可以生成 XAML 和 HTML 的通用程序,并通过分享大量 ...
- eclipse设置项目发布到tomcat webaap下
之前用myeclipse,自动deploy到web app下.目前的项目换了eclipse,项目被发布到本地,通过一个链接到tomcat中,很不习惯. 设置的地方很隐晦,在tomcat中,而不是在pr ...