PostreSQL崩溃试验全记录

磨砺技术珠矶，践行数据之道，追求卓越价值

回到上一级页面： PostgreSQL基础知识与基本操作索引页     回到顶级页面：PostgreSQL索引页

[作者 高健@博客园  luckyjackgao@gmail.com]

由于客户提到，运行某些大运算量批处理时，PostgreSQL突发性使用大量资源的问题，

进行了如下的调查，发现确实发生了崩溃。PostgreSQL 需要资源控制方案啊。

现在正在考虑是否必须要用 os 级别的限制方案：

过程如下：

测试环境：

内存：大约1024MB

postgresql.conf 设定：

使用缺省值：checkpoint_segments =3  shard_buffers=32MB

这些是有意为之，就是想看看数据量大、shared_buffer小的状况：

建立表(一条记录大约1024字节)：

postgres=# create table test01(id integer, val char(1024));

向表中插入大量数据(总共插入2400MB)

postgres=# insert into test01 values(generate_series(1,2457600),repeat( chr(int4(random()*26)+65),1024));

插入操作会花费一些时间，此时用ps 命令看PostgreSQL个进程对内存使用状况：‘

[root@server ~]# ps aux | grep post

root        0.0  0.0     pts/    S    :   : su - postgres

postgres    0.0  0.0      pts/    S+   :   : -bash

postgres    0.0  0.2    pts/    S    :   : /usr/local/pgsql/bin/postgres -D /gao/data

postgres    0.4  3.0   ?        Ss   :   : postgres: writer process                  

postgres    0.2  0.1    ?        Ds   :   : postgres: wal writer process              

postgres    0.0  0.0    ?        Ss   :   : postgres: autovacuum launcher process     

postgres    0.0  0.0      ?        Ss   :   : postgres: stats collector process         

root        0.0  0.0     pts/    S    :   : su - postgres

postgres    0.0  0.0      pts/    S    :   : -bash

postgres    0.0  0.0      pts/    S+   :   : ./psql

postgres   14.8 80.2   ?      Ds   :   : postgres: postgres postgres [local] INSERT

root        0.0  0.0      pts/    S+   :   : grep post

可以发现，INSERT操作正在消耗80%以上的内存。

再等片刻，发现psql端传来反馈：

WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
The connection to the server was lost. Attempting reset: Failed.
!> 

此时看看Log，可以看到：background writer (3321) 已经被干掉了，所有连接被重置。

LOG:  autovacuum launcher started
LOG:  database system is ready to accept connections
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( second apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( second apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( second apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  checkpoints are occurring too frequently ( seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".
LOG:  background writer process (PID ) was terminated by signal : Killed
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
LOG:  all server processes terminated; reinitializing
FATAL:  the database system is in recovery mode
LOG:  database system was interrupted; last known up at -- :: CST
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  consistent recovery state reached at /B7657BD0
LOG:  redo starts at /B60FE2B8
LOG:  unexpected pageaddr /B044C000 in log file , segment , offset
LOG:  redo done at /B844B940
LOG:  autovacuum launcher started
LOG:  database system is ready to accept connections

各个进程都重新生成了：

[root@server ~]# ps aux | grep post
root        0.0  0.0     pts/    S    :   : su - postgres
postgres    0.0  0.0      pts/    S+   :   : -bash
postgres    0.0  0.5    pts/    S    :   : /usr/local/pgsql/bin/postgres -D /gao/data
root        0.0  0.0     pts/    S    :   : su - postgres
postgres    0.0  0.0      pts/    S    :   : -bash
postgres    0.0  0.0     pts/    S+   :   : ./psql
postgres    0.0  0.0     ?        Ss   :   : postgres: writer process
postgres    0.0  0.0     ?        Ss   :   : postgres: wal writer process
postgres    0.0  0.1    ?        Ss   :   : postgres: autovacuum launcher process
postgres    0.0  0.0      ?        Ss   :   : postgres: stats collector process
root        0.0  0.0      pts/    R+   :   : grep post
[root@server ~]# 

回到psql端再看，发现连接已经失效了：

!> \
Invalid command \. Try \? for help.
!> \dt;
You are currently not connected to a database.
!> 

根据向社区提问，据说是因为OS级别的OOM错误发生,所以杀死了Postmaster进程。

总之，此种情况表明，如果没有对资源消费总量的控制，那么突发性的用户访问会导致崩溃的。

[作者 高健@博客园  luckyjackgao@gmail.com]

回到上一级页面： PostgreSQL基础知识与基本操作索引页     回到顶级页面：PostgreSQL索引页

磨砺技术珠矶，践行数据之道，追求卓越价值