使用cgroups来控制内存使用

磨砺技术珠矶，践行数据之道，追求卓越价值

回到上一级页面：PostgreSQL内部结构与源代码研究索引页    回到顶级页面：PostgreSQL索引页

[作者 高健@博客园  luckyjackgao@gmail.com]

首先学习网上例子，进行体验性的试验：

首先不限制内存使用来进行下载：

[root@cent6 Desktop]# free -m
             total       used       free     shared    buffers     cached
Mem:
-/+ buffers/cache:
Swap:
[root@cent6 Desktop]# 

然后，再下载约700M:

wget http://centos.arcticnetwork.ca/6.4/isos/x86_64/CentOS-6.4-x86_64-LiveCD.iso

然后看内存使用情况：

[root@cent6 Desktop]# free -m
             total       used       free     shared    buffers     cached
Mem:
-/+ buffers/cache:
Swap:
[root@cent6 Desktop]# 

确实是用掉了700多M内存。

然后，重新启动，限制内存使用：

[root@cent6 Desktop]# service cgconfig status
Stopped

#mount -t cgroup -o memory memcg /cgroup 

# mkdir /cgroup/GroupA
# echo 10M > /cgroup/GroupA/memory.limit_in_bytes
# echo $$ > /cgroup/GroupA/tasks 

然后，再看内存状况：

[root@cent6 Desktop]# free -m
             total       used       free     shared    buffers     cached
Mem:
-/+ buffers/cache:
Swap:
[root@cent6 Desktop]#

再下载约700M:

wget http://centos.arcticnetwork.ca/6.4/isos/x86_64/CentOS-6.4-x86_64-LiveCD.iso

再看内存使用前后对比:

[root@cent6 Desktop]# free -m
             total       used       free     shared    buffers     cached
Mem:
-/+ buffers/cache:
Swap:
[root@cent6 Desktop]#

可以知道，大约的内存使用量为 1525-1494=31M。不过free命令观察到的结果是有误差的，程序执行时间长，free就是一个不断累减的值，由于当前shell被限制使用内存最大10M，那么基数很小的情况下，时间越长，误差越大。

下面，看看对PostgreSQL能否产生良好的限制：

再此之前，通过系统设定来看看对postgres用户进行wget操作时的内存的控制：

[postgres@cent6 Desktop]$ cat /etc/cgconfig.conf
#
#  Copyright IBM Corporation. 2007
#
#  Authors:    Balbir Singh <balbir@linux.vnet.ibm.com>
#  This program is free software; you can redistribute it and/or modify it
#  under the terms of version 2.1 of the GNU Lesser General Public License
#  as published by the Free Software Foundation.
#
#  This program is distributed in the hope that it would be useful, but
#  WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See man cgconfig.conf for further details.
#
# By default, mount all controllers to /cgroup/<controller>

mount {
    cpuset    = /cgroup/cpuset;
    cpu    = /cgroup/cpu;
    cpuacct    = /cgroup/cpuacct;
    memory    = /cgroup/memory;
    devices    = /cgroup/devices;
    freezer    = /cgroup/freezer;
    net_cls    = /cgroup/net_cls;
    blkio    = /cgroup/blkio;
}

group test1 {
    perm {
          task{
              uid=postgres;
              gid=postgres;
          }

          admin{
             uid=root;
             gid=root;
          }

    } memory {
       memory.limit_in_bytes=30M;
    }
}

[postgres@cent6 Desktop]$ 

还有一个文件，cgrules.conf，也很重要:

[postgres@cent6 Desktop]$ cat /etc/cgrules.conf
# /etc/cgrules.conf
#
#Each line describes a rule for a user in the forms:
#
#<user>            <controllers>        <destination>
#<user>:<process name>    <controllers>        <destination>
#
#Where:
# <user> can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for any user or group.
#        - The %, which is equivalent to "ditto". This is useful for
#          multiline rules where different cgroups need to be specified
#          for various hierarchies for a single user.
#
# <process name> is optional and it can be:
#     - a process name
#     - a full command path of a process
#
# <controller> can be:
#      - comma separated controller names (no spaces)
#      - * (for all mounted controllers)
#
# <destination> can be:
#      - path with-in the controller hierarchy (ex. pgrp1/gid1/uid1)
#
# Note:
# - It currently has rules based on uids, gids and process name.
#
# - Don't put overlapping rules. First rule which matches the criteria
#   will be executed.
#
# - Multiline rules can be specified for specifying different cgroups
#   for multiple hierarchies. In the example below, user "peter" has
#   specified 2 line rule. First line says put peter's task in test1/
#   dir for "cpu" controller and second line says put peter's tasks in
#   test2/ dir for memory controller. Make a note of "%" sign in second line.
#   This is an indication that it is continuation of previous rule.
#
#
#<user>      <controllers>      <destination>
#
#john          cpu        usergroup/faculty/john/
#john:cp       cpu        usergroup/faculty/john/cp
#@student      cpu,memory    usergroup/student/
#peter           cpu        test1/
#%           memory        test2/
#@root            *        admingroup/
#*        *        default/
# End of file
 postgres      memory           test1/
#
[postgres@cent6 Desktop]$ 

在root用户，设置如下两个服务随系统启动：

chkconfig cgconfig  on

chkconfig cgred on

然后重新启动系统后，用postgres用户进行登录，进行检验：

[postgres@cent6 Desktop]$ free -m
             total       used       free     shared    buffers     cached
Mem:          2006        381       1625          0         25        134
-/+ buffers/cache:        221       1785

[postgres@cent6 Desktop]$ wget http://centos.arcticnetwork.ca/6.4/isos/x86_64/CentOS-6.4-x86_64-LiveCD.iso

执行完毕后，看内存状况，成功。

[postgres@cent6 Desktop]$ free -m
             total       used       free     shared    buffers     cached
Mem:          2006        393       1613          0         28        141
-/+ buffers/cache:        224       1782
Swap:         4031         67       3964
[postgres@cent6 Desktop]$ 

下面看对postgresql中执行sql 的限制如何：

步骤1:   对/etc/cgconfig.conf 文件和 /etc/cgrules.conf 文件的设置如前所述。

步骤2:   运行前查看内存状况：

[postgres@cent6 Desktop]$ free -m

             total       used       free     shared    buffers     cached

Mem:          2006        384       1622          0         26        138

-/+ buffers/cache:        219       1787

Swap:         4031         87       3944

[postgres@cent6 Desktop]$ 

步骤3: 开始处理大量数据(约600MB)

postgres=# select count(*) from test01;
 count
-------
     0
(1 row)

postgres=# insert into test01 values(generate_series(1,614400),repeat( chr(int4(random()*26)+65),1024));

运行刚刚开始，就出现了如下的错误：

The connection to the server was lost. Attempting reset: Failed.
!> 

这和之前碰到的崩溃情形一致。

 

PostgreSQL的log本身是这样的： 

[postgres@cent6 pgsql]$ LOG:  database system was shut down at -- :: CST
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
LOG:  server process (PID ) was terminated by signal : Killed
DETAIL:  Failed process was running: insert into test01 values(generate_series(,),repeat( chr(int4(random()*)+),));
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
FATAL:  the database system is in recovery mode
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up at -- :: CST
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at /9E807C90
LOG:  unexpected pageaddr /946BE000 in log file , segment , offset
LOG:  redo done at /9F6BDB50
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

通过dmesg命令，可以看到，发生了Out of Memory错误，这次是 cgroup out of memory

[postgres@cent6 Desktop]$ dmesg | grep post
[ ]                                       postgres
[ ]                                        postgres
[ ]                                        postgres
[ ]                                        postgres
[ ]                                        postgres
[ ]                                        postgres
[ ]                                       postgres
[ ]                                       postgres
[ ]                                        postgres
[ ]                                        postgres
[ ]                                        postgres
[ ]                                        postgres
[ ]                                        postgres
[ ]                                       postgres
[ ]                                       postgres
[ ]                                        postgres
[ ]                                        postgres
[ ]                                       postgres
[ ]                                       postgres
[ ]                                        postgres
[ ]                                      postgres
Memory cgroup out of memory: Kill process  (postgres) score  or sacrifice child
Killed process , UID , (postgres) total-vm:292300kB, anon-rss:8432kB, file-rss:3712kB
[postgres@cent6 Desktop]$ 

我怀疑自己的内存开得过小了，影响到一些基本的运行。PostgreSQL本身也需要一些资源(shared_buffers、wal_buffers都需要用一些内存)

所以我调整了参数   memory.limit_in_bytes=300M ，再次运行：
前述的sql问处理1200MB数据，成功结束，内存没有过多增长。

[作者 高健@博客园  luckyjackgao@gmail.com]

回到上一级页面：PostgreSQL内部结构与源代码研究索引页    回到顶级页面：PostgreSQL索引页

磨砺技术珠矶，践行数据之道，追求卓越价值