Ten ways to improve the performance of large tables in MySQL--转载
原文地址:http://www.tocker.ca/2013/10/24/improving-the-performance-of-large-tables-in-mysql.html
Today I wanted to take a look at improving the performance of tables that cause performance problems based largely on their size. Some of this advice also applies to databases that are large in-aggregate over many tables, but I always find the individually large table a special-case that is problematic.
What you will normally find is that the speed that the table can be modified will trend down as the size increases. Here is what I am going to call the typical B+Tree index performance over time:
So we should expect degradation of performance due to the structure of the index, but there are actually some ways that we can try and stretch out the curve, and not degrade as quickly.
Ten potential ways to reduce large table impact:
Make sure to use InnoDB instead of MyISAM. MyISAM can be faster at inserts to the end of a table, but it has both table locking (limiting updates and deletes) and uses a single lock to protect the key buffer when loading data to/from disk, resulting in contention. It also does not have the change buffering feature described just below.
InnoDB has change buffering (previously called the insert buffer), which is a feature to delay building secondary indexes that are not unique, and merge writes. It's further described by Facebook here. It's not shown in the graph above, but it can boost insert performance by quite a lot, and it's enabled by default. It was greatly improved in MySQL 5.5, so it is time to upgrade if you haven't.
Partitioning may reduce the size of indexes, effectively reducing the table
into many smaller tables. It also reduces internal index->lockcontention, something that has been greatly improved in the MySQL 5.7.2 DMR.Use innodb page compression. For some workloads (particularly those with lots of char/varchar/text data types) compression will allow the data to be more compact, stretching out that performance curve for longer. It may also allow you to more easily justify SSDs which are typically smaller in capacity. InnoDB page compression was improved a lot in MySQL 5.6, courtesy of Facebook providing a series of patches.
Sort and bulk load data into tables. Inserting in order will result in fewer page splits (which will perform worse on tables not in memory), and the bulk loading is not specifically related to the table size, but it will help reduce redo log pressure.
Remove any unnecessary indexes on the table, paying particular attention to UNIQUE indexes as these disable change buffering. Don't use a UNIQUE index if you have no reason for that constraint; prefer a regular INDEX.
Related to the points 5 & 6, the type of primary key also matters. It is much better to use either an
INT
orBIGINT
datatype than say aGUID
, which will have a curve that degrades much faster. Having no PRIMARY KEY will also affect performance negatively.If bulk loading a fresh table, delay creating any indexes besides the PRIMARY KEY. If you create them once all data is loaded, then InnoDB is able to apply a pre-sort and bulk load process which is both faster and results in typically more compact indexes. This optimization became true in MySQL 5.5.
More memory can actually help here too. I frequently see people under spec memory on new database servers compared to what it actually costs these days. Simple advice: If
SHOW ENGINE INNODB STATUS
shows any reads/s underBUFFER POOL AND MEMORY
and the number of Free buffers (also underBUFFER POOL AND MEMORY
) is zero, you could benefit from more (assuming you have sizedinnodb_buffer_pool_size
correctly on your server. See here.)As well as memory, SSDs can help too. Much of the performance drop shown on the curve can be attributed to additional IO which is created as the table gets bigger. While a hard drive can do 200 operations per second (IOPS), a typical SSD will do 20K+
Ten ways to improve the performance of large tables in MySQL--转载的更多相关文章
- Five Invaluable Techniques to Improve Regex Performance
Regular expressions are powerful, but with great power comes great responsibility. Because of the wa ...
- 8 ways to improve ASP.NET Web API performance
ASP.NET Web API is a great piece of technology. Writing Web API is so easy that many developers don’ ...
- Effective Modern C++ 42 Specific Ways to Improve Your Use of C++11 and C++14
Item 1: Understand template type deduction. Item 2: Understand auto type deduction. Item 3: Understa ...
- to improve sqlite performance
INSERT is really slow - I can only do few dozen INSERTs per second http://www.sqlite.org/faq.html#q1 ...
- build a real-time analytics dashboard to visualize the number of orders getting shipped every minute to improve the performance of their logistics for an e-commerce portal
https://cloudxlab.com/blog/real-time-analytics-dashboard-with-apache-spark-kafka/
- LMAX Disruptor – High Performance, Low Latency and Simple Too 转载
原文地址:http://www.symphonious.net/2011/07/11/lmax-disruptor-high-performance-low-latency-and-simple-to ...
- Packet for query is too large(1767212 > 1048576)mysql在存储图片时提示图片过大
原网址:http://blog.csdn.net/bigbird2012/article/details/6304417 错误现象:Packet for query is too large(1767 ...
- Why MySQL could be slow with large tables ?
https://www.percona.com/blog/2006/06/09/why-mysql-could-be-slow-with-large-tables/
- 关于数据库报Packet for query is too large (1986748 > 1048576)(mysql写入数据过大)的解决办法
方法2 (很妥协,很纠结的办法) 进入mysql server 在mysql 命令行中运行 set global max_allowed_packet = 2*1024*1024*10 然后关闭掉这此 ...
随机推荐
- SYSU 6356 Dispatching
Dispatching Time Limit: 3000ms Memory Limit: 262144KB This problem will be judged on SYSU. Original ...
- CSUOJ 1549 Navigition Problem
1549: Navigition Problem Time Limit: 1 Sec Memory Limit: 256 MBSubmit: 65 Solved: 12 Description N ...
- LightOJ 1070 Algebraic Problem (推导+矩阵高速幂)
题目链接:problem=1070">LightOJ 1070 Algebraic Problem 题意:已知a+b和ab的值求a^n+b^n.结果模2^64. 思路: 1.找递推式 ...
- POJ 2111 DP+记录路径
题意: 思路: 类似滑雪 //By SiriusRen #include <cstdio> #include <cstring> #include <algorithm& ...
- POJ 1474 Video Surveillance 半平面交/多边形核是否存在
http://poj.org/problem?id=1474 解法同POJ 1279 A一送一 缺点是还是O(n^2) ...nlogn的过几天补上... /********************* ...
- readonly&&declare&&unset &&export&&env环境变量
readonly命令用于定义只读shell变量和shell函数.readonly命令的选项-p可以输出显示系统中所有定义的只读变量. 选项 -f:定义只读函数: -a:定义只读数组变量: -p:显示系 ...
- MVC—实现ajax+mvc异步获取数据
之前写过ajax和一般处理程序的结合实现前后台的数据交换的博客,如今做系统用到了MVC,同一时候也用到了异步获取数据. ajax+一般处理程序与MVC+ajax原理是一样的在"URL&quo ...
- 通过一个案例彻底读懂10046 trace--字节级深入破解
转载请注明出处:http://blog.csdn.net/guoyjoe/article/details/37840583 2014.7.23晚20:30 Oracle support组猫大师分享&l ...
- hdoj-1870-愚人节的礼物(栈)
愚人节的礼物 Time Limit: 5000/1000 MS (Java/Others) Memory Limit: 32768/32768 K (Java/Others) Total Sub ...
- 轻松学习Linux之Shell的常用过滤器
下载高清视频: http://down.51cto.com/data/157818 大小3:MB 时长: 7分钟 更多内容见: Linux爱好者的圣诞大餐-轻松学习Linux系列多媒体 ...