Avoiding PostgreSQL database corruption
TL;DR: Don't ever set fsync=off, don't kill -9 the postmaster then deletepostmaster.pid, don't run PostgreSQL on network file systems.
Reports of database corruption on the PostgreSQL mailing list are uncommon, but hardly rare. While a few data corruption issues have been found in PostgreSQL over the years, the vast majority of issues are caused by:
- Administrator action;
- Misconfiguration; or
- Bad hardware
A recent mailing list post asked what can be done to reduce the chance of corruption and keep data safe.
If you think you have a corrupt PostgreSQL database, stop the database server and take a complete copy of the data directory now. See the Corruption page on the wiki. Then ask for help on the pgsql-general mailing list, or for critical/urgent issues contact a profesional support provider.
Do not attempt to fix the problem before taking a complete copy of the entire data directory. You might make the problem much worse, turning a recoverable problem into unrecoverable data loss.
Here's my advice for avoiding DB corruption issues, with some general PostgreSQL administration advice thrown in for good measure:
Read the manual
Read the reliability section of the manual in detail.
Many of the issues people on the mailing lists ask about could be avoided if they'd only read the manual. The PostgreSQL manual is well written, comprehensive, and will teach you about SQL and databases in general, not just PostgreSQL. Reading it cover-to-cover helped me learn more about database systems than any one other thing I've ever done. Read it.
PostgreSQL is easy enough to use, so you don't have to read the manual, but you'll get a lot more out of the system if you do. You'll be able to tune it for better performance, know how to avoid a few pitfalls, and know how to use more of its features. You'd know to avoid usingSIGKILL
(kill -9
) to terminate the postmaster, for example.
Updates
Bug fixes are no good if you don't apply them. Keep up to date with the latest PostgreSQL patch releases. Don't be one of those people still running 9.0.0 when 9.0.10 is out. The team releases updates for a reason. Minor point releases require minimal downtime as there's no need to dump and reload; just stop the DB, install the new binaries, and start the DB. Even this downtime can be avoided with failover and failback using something like repmgr.
As per the versioning policy they're extremely conservative about what gets back-patched into released major versions so upgrades are very safe. Read the versioning policy. Just read the release notes for each version in between yours and the new version, then install it.
For more on this topic read the FAQ entry A bug I'm encountering is fixed in a newer minor release of PostgreSQL, but I don't want to upgrade. Can I get a patch for just this issue?
Keep an eye on the end of life dates. Plan to upgrade to a new major version well before your version goes EOL.
Administrator action
Don't kill -9
(SIGKILL
) any PostgreSQL processes. It should be fine, but it's still not smart. kill -9
should be your last resort; first use pg_cancel_backend
thenpg_terminate_backend
(which is effectively a kill -15
). If you kill -9
a backend the whole DB will terminate and restart, breaking all connections and forcing crash recovery.
Never delete postmaster.pid
.
(Corruption should not actually occur unless you kill -9
the postmaster but leave otherpostgres
processes running, then delete postmaster.pid
and start a new postmaster. If you insist on unlocking the gun safe, loading the shotgun, aiming it at your foot and pulling the trigger we can't stop you. Don't do that.)
Don't delete anything in the data directory, with the sole exception of the contents ofpg_log
if your logs are within your data directory. Everything else is vital and should never be messed with directly by the administrator. Do not delete anything from pg_clog
orpg_xlog
, these are vital components of the database system.
Do not run your database with fsync = off
if it contains any data you care about. Turning off the fsync configuration parameter effectively gives PostgreSQL permission to avoid slow syncs at the risk of severe and unrecoverable data corruption if the server crashes or loses power. If it's possible to recover such a database at all it'll take many hours of time from a serious (and expensive) expert. These days there's rarely a reason to turn fsync = off
anyway; you can achieve many of the same benefits without the corruption risks usingasynchronous commit and a commit delay.
The system catalogs (pg_catalog.*
) are there for looking, not for touching. Do not everINSERT
, UPDATE
or DELETE
in the system catalog tables unless you are absolutelycertain you know what you are doing. Only a superuser or (for some catalogs) the database owner can mess directly with the catalogs, and you shouldn't be running as either day-to-day.
If you need pg_resetxlog
then you already have a corruption problem. pg_resetxlog
is a last resort for getting a damaged database going again, not a semi-routine operation like MySQL's myisamchk
. If you have to pg_resetxlog
a DB, you need to work out what went wrong and why, then dump the DB(s), re-initdb, and restore the dumps. Don't just keep on using the damaged database.
Backups, replication and PITR
The backups section of the manual is good. Read it. You should also look at the Barmantool, which helps automate many of these tasks.
Maintain rolling backup dumps with proper ageing. For example, keep one a day for the last 7 days, then one a week for the last 4 weeks, then one a month for the rest of the year, then one a year. Backups won't stop corruption or damage happening, but they'll help you recover with a minimum of pain. In my view good old pg_dump is the tool of choice here; I tend to produce one -Fc
dump per database and a pg_dumpall --globals-only
for users/roles/etc.
Use warm standby with log shipping and/or replication to maintain a live copy of the DB, so you don't have that painful window between your last backup and when the failure occurred. Do not rely solely on replication, take dumps too.
If you want point-in-time recovery, keep a few days or weeks worth of WAL archives and a basebackup around. That'll help you recover from those "oops I meant DROP TABLE unimportant;
not DROP TABLE vital_financial_records;
" issues with a minimum of data loss.
Barman can help with warm standby and PITR.
If taking filesystem-level backups of the data directory, you don't have to stop the DB if you use pg_basebackup
, or use pg_start_backup
and pg_stop_backup
correctly. Seethe manual. You do have to include the whole data directory. Do not omit folders you think are unimportant from the backup. People who left pg_xlog
or pg_clog
out of their backup as "unimportant" periodically turn up asking for help on the mailing list. Don't be one of them.
Remember to test your backups, don't just assume they're fine.
Hardware
Plug-pull test your system when you're testing it before going live. Put it under load with something like pgbench, then literally pull the power plug out so your server loses power and shuts down uncleanly. If your database doesn't come back up fine you have hardware, OS or configuration problems. Do this repeatedly while under load, until you're confident it'll come up reliably.
Use good quality hardware with proper cooling and a good quality power supply. If possible, ECC RAM is a nice extra.
Never, ever, ever use cheap SSDs. Use good quality hard drives or (after proper testing) high end SSDs. Read the SSD reviews periodically posted on this mailing list if considering using SSDs. Make sure the SSD has power-loss protection for its write cache - a supercapacitor/ultracapacitor, low-power capacitor array, secondary battery, or other write-cache protection. Always do repeated plug-pull testing when using SSDs, don't just trust the specifications.
Avoid RAID 5, mostly because the performance is terrible, but also because I've seen corruption issues with rebuilds from parity on failing disks. If you're using spinning disks RAID 10 is a better option for a database.
If you have a hardware RAID controller with a battery backed cache (BBU), test the BBU regularly. A battery backup that doesn't work is no good, and a RAID controller in write-back caching mode without a working BBU is guaranteed to severely corrupt your data if the system loses power.
Use a good quality hardware RAID controller with a battery backup cache unit if you're using spinning disks in RAID. This is as much for performance as reliability; a BBU will make an immense difference to database performance. Be aware that if your RAID controller fails you must be able to get a hold of a compatible one before you can read your array. If you don't add a battery backed cache you won't gain much over Linux's md
software RAID, which has the advantage that arrays are portable from machine to machine. Do not use cheap on-motherboard/in-BIOS "RAID" systems, they combine the worst features of hardware and software RAID.
If you're going to have a UPS (you shouldn't need one as your system should be crash-safe), don't waste your money on a cheap one. Get a good online double-conversion unit that does proper power filtering. Cheap UPSs are just a battery with a fast switch, they provide no power filtering and what little surge protection they offer is done with a component that wears out over time and load, becoming totally ineffective. Since your system should be crash-safe a cheap UPS will do nothing for corruption protection, it'll only help with uptime.
I shouldn't have to say this, but ... don't run your database off a USB flash drive ("USB memory key" or "USB stick").
Software and OS
Use a solid, reliable file system. zfs-on-linux, btrfs, etc are not the right choices for a database you care about. Never, ever, ever use FAT32.
If on Windows, do not run an anti-virus program on your
database server. Nobody should be using it for other things or running programs on it anyway. Antivirus software intercepts system calls in all sorts of fun and interesting ways. Many of these are subject to exciting race conditions and bugs when software like PostgreSQL has several processes all reading from and writing to the same files. Antivirus software may also try to "clean" the database files, causing severe corruption. If you absolutely must run antivirus software, add exclusions for the PostgreSQL data directory and PostgreSQL install directory. If your AV software supports process exclusions, add an exclusion for all programs in the PostgreSQL bin
directory, especially postgres.exe
.
Make sure your OS is disabling your hard drive's write cache or properly flushing it. These days this is a pretty safe bet unless you're using a cheap SSD that ignores flush requests. Plug-pull testing should catch write-cache issues, as should the tools mentioned in the reliability documentation linked to above.
You may also wish to consider reading these wiki articles:
参考:
http://blog.ringerc.id.au/2012/10/avoiding-postgresql-database-corruption.html
Avoiding PostgreSQL database corruption的更多相关文章
- How to drop a PostgreSQL database if there are active connections to it?
1.PostgreSQL 9.1 and below: SELECT pg_terminate_backend(pg_stat_activity.procpid) FROM pg_stat_activ ...
- Database Corruption ->> Fix Database In Suspect State
昨天在工作中遇到一个情况,就是Development环境中的某台服务器上的某个数据库进入了Suspect状态.以前看书倒是知道说这个状态,不过实际工作当中从来没有遇到过.那么一些背景情况是这样的. 环 ...
- How to change owner of PostgreSql database?
ALTER DATABASE name OWNER TO new_owner;
- 安装PostgreSQL数据库 ,Database Cluster 失败!
在安装PG数据库的过程中,会选择安装目录以及数据存放目录和端口,并需要选择Local,如果全部使用默认,并且设置好自己的密码后开始安装,前期进展还比较顺利,到了安装Database Cluster时, ...
- How to speed up insertion performance in PostgreSQL
Disable any triggers on the table Drop indexes before starting the import, re-create them afterwards ...
- DataBase异常状态:Recovery Pending,Suspect,估计Recovery的剩余时间
一,RECOVERY PENDING状态 今天修改了SQL Server的Service Account的密码,然后重启SQL Server的Service,发现有db处于Recovery Pendi ...
- CentOS 6安装PostgreSQL
https://zh.wikipedia.org/wiki/PostgreSQL PostgreSQL是自由的对象-关系型数据库服务器(数据库管理系统),在灵活的BSD-风格许可证下发行.它在其他开放 ...
- kali linux 系列教程之metasploit 连接postgresql可能遇见的问题
kali linux 系列教程之metasploit 连接postgresql可能遇见的问题 文/玄魂 目录 kali linux 下metasploit 连接postgresql可能遇见的问题. ...
- PostgreSQL Performance Monitoring Tools
PostgreSQL Performance Monitoring Tools https://github.com/CloudServer/postgresql-perf-tools This pa ...
随机推荐
- treap模版代码
treap模版暂存. 以后修改整理. #include<cstdio> #include<iostream> #include <time.h> #include& ...
- Python学习路程day7
多态 class Animal: def __init__(self, name): # Constructor of the class self.name = name def talk(self ...
- 什么php?
PHP是一种开源的通用计算机脚本语言,尤其适用于网络开发并可嵌入HTML中使用.PHP的语法借鉴吸收了C语言.Java和Perl等流行计算机语言的特点,易于一般程序员学习.PHP的主要目标是允许网络开 ...
- UITouch的用法
UITouch一般无法直接获取,是通过UIView的touchesBegan等函数获得. //这四个方法是UIResponder中得方法 // Generally, all responders wh ...
- PHP图像处理之在原图像处理
处理原有的图像 图片处理,缩放,裁剪,翻转,旋转,透明,锐化等图片操作 一.创建图片资源 imagecreatetruecolor(width,hei ...
- 简单的多线程(活用OD解决运行时错误)
代码源自<VC++深入详解>第15章 “多线程”,位于第563页 - 566 页. 程序的目的是展示多线程运行的效果: #include <windows.h> #includ ...
- Magento:Paypal付款不成功返回后不要清空购物车产品的解决方案
经常遇到这个问题,当我们使用第三方支付工具Gateway如paypal支付的时候,如果用户付款不成功或者取消了订单再返回网站时,发现购物车里面的产品已经被清空了,如果是客户主动cancel的还好,但是 ...
- dedecms织梦
dedecms的安装,其修改连接数据库文件是:common.inc.php
- (理论篇)温故而知新_PHP入门基础教程
简单的回顾一下基础知识 1.嵌入方法: 类似ASP的<%,PHP可以是<?php或者是<?,结束符号是?>,当然您也可以自己指定. 2.引用文件: 引用文件的方法有两种:req ...
- jq中如何阻止元素的默认行为?
阻止网页元素的默认行为: event.preventDefault(); 或者:return false;