RHCA rh442 008 oom 故障触发器 内存溢出 swap numa tmpfs shm

OOM out-of-memory killer

进程被莫名其妙杀死

所有内存加swap为活动

zone_normal 没空间 （目前64G时代可以不考虑这个）

溢出就死机

内存溢出前，去杀死进程

因此需要oom

[root@servera tuned]# pidof vsftpd
828
[root@servera tuned]# cd /proc/828/
[root@servera 828]# ls
attr             cpuset   limits      net            personality  smaps_rollup  timerslack_ns
autogroup        cwd      loginuid    ns             projid_map   stack         uid_map
auxv             environ  map_files   numa_maps      root         stat          wchan
cgroup           exe      maps        oom_adj        sched        statm
clear_refs       fd       mem         oom_score      schedstat    status
cmdline          fdinfo   mountinfo   oom_score_adj  sessionid    syscall
comm             gid_map  mounts      pagemap        setgroups    task
coredump_filter  io       mountstats  patch_state    smaps        timers
[root@servera 828]# cat oom_score
0
[root@servera 828]# cat oom_adj
0
[root@servera 828]# cat oom_score_adj
0
[root@servera 828]#

针对于每个进程，系统会分配一个oom值，默认为0，取值范围 -1000 ~ 1000，数字越小，被杀的概率就越小

[root@servera 828]# echo -1000 > oom_score_adj
[root@servera 828]# cat oom_score_adj
-1000
[root@servera 828]# cat oom_adj
-17

oom_adj随着oom_score_adj 变化 (两个调的东西都是一样，他们互相影响变化。)

/proc/PID/oom_score_adj -1000~1000

/proc/PID/oom_score -17~-15 表示进程被杀概率 2^-17 --- 2^15

先比小数字，然后再比大数字

默认都是平等的，内存溢出，先杀谁。都平等

yum -y install kernel-doc
[root@servera Documentation]# ls -lR | grep sysrq
-r--r--r--. 1 root root  12491 Mar 13  2019 sysrq.rst
-r--r--r--. 1 root root  35958 Mar 13  2019 sysrq.html
-r--r--r--. 1 root root 12491 Mar 13  2019 sysrq.rst.txt
[root@servera Documentation]# pwd
/usr/share/doc/kernel-doc-4.18.0/Documentation
[root@servera Documentation]# 

[root@servera admin-guide]# vi sysrq.rst
[root@servera admin-guide]# pwd
/usr/share/doc/kernel-doc-4.18.0/Documentation/admin-guide
[root@servera admin-guide]#

模拟系统死机，崩溃，溢出

sysrq.rst



正常关机，系统会执行一遍sync

[root@servera admin-guide]# echo f > /proc/sysrq-trigger
[root@servera admin-guide]# systemctl status vsftpd.service
● vsftpd.service - Vsftpd ftp daemon
Loaded: loaded (/usr/lib/systemd/system/vsftpd.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/vsftpd.service.d
        └─01-limits.conf
Active: failed (Result: signal) since Mon 2022-07-04 19:21:43 CST; 2min 22s ago
Process: 827 ExecStart=/usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf (code=exited, status=0/SUCCESS)
Main PID: 828 (code=killed, signal=KILL)

Jul  4 19:21:43 servera kernel: Out of memory: Kill process 828 (vsftpd) score 999 or sacrifice child
Jul  4 19:21:43 servera kernel: Killed process 828 (vsftpd) total-vm:26952kB, anon-rss:28kB, file-rss:0kB, shmem-rss:0kB
Jul  4 19:21:43 servera kernel: oom_reaper: reaped process 828 (vsftpd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

查看日志

sysrq.rst 模拟各种系统故障

系统认为你是内存消耗的大户，把你杀了。

不想被杀死，那你就把那个值调大一点 oom_score_adj

Out of memory: Kill process 785 (firewalld) score 2 or sacrifice child
Jul  4 19:31:42 servera kernel: Killed process 785 (firewalld) total-vm:454412kB, anon-rss:112kB, file-rss:76kB, shmem-rss:0kB
Jul  4 19:31:42 servera kernel: oom_reaper: reaped process 785 (firewalld), now anon-rss:0kB, file-rss:8kB, shmem-rss:0kB
Jul  4 19:31:42 servera systemd[1]: firewalld.service: Main process exited, code=killed, status=9/KILL

我把vsftpd调到-1000，那么系统不杀vsftpd。转而杀死防火墙

/proc/sysrq-trigger 这是很有用的触发器

启动服务就可以 设置oom_score_adj 设置被内存杀手杀死的权值

写在tuned脚本，开机生效，或者手动profile生效

[root@servera supermao12]# tail -n 4 tuned.conf
[my_script2]
type=script
script=vsftpd.sh

[root@servera supermao12]# cat vsftpd.sh
#!/bin/bash
echo 909 > /proc/$(pidof vsftpd)/oom_score_adj

第二种方法，启动服务时生效

[root@servera system]# cat vsftpd.service
Unit]
Description=Vsftpd ftp daemon
After=network.target

[Service]
Type=forking
ExecStart=/usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
ExecStartPost=/usr/local/bin/vsftpd.sh

[Install]
WantedBy=multi-user.target

[root@servera system]# cat /usr/local/bin/vsftpd.sh
#!/bin/bash
echo 1000 > /proc/$(pidof vsftpd)/oom_score_adj

内存溢出：所有内存用光

内存泄漏：当运行一个应用程序，可能需要100MB内存，但关闭改应用程序时，内存不能被完全释放

windows 内存释放的不行，必须关机。

linux 做的好一些三年，不重启，依然保证性能

yum -y install valgrind

[root@servera ~]# valgrind --tool=memcheck ls
==2739== Memcheck, a memory error detector
==2739== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2739== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==2739== Command: ls
==2739==
anaconda-ks.cfg  bigmem  cache-lab-8.0	cache-lab-8.0.tar.gz  oom_score_adj~  original-ks.cfg  sar1.data  sar-disk.log
==2739==
==2739== HEAP SUMMARY:
==2739==     in use at exit: 22,016 bytes in 15 blocks
==2739==   total heap usage: 50 allocs, 35 frees, 59,971 bytes allocated
==2739==
==2739== LEAK SUMMARY:
==2739==    definitely lost: 0 bytes in 0 blocks
==2739==    indirectly lost: 0 bytes in 0 blocks
==2739==      possibly lost: 0 bytes in 0 blocks
==2739==    still reachable: 22,016 bytes in 15 blocks
==2739==         suppressed: 0 bytes in 0 blocks
==2739== Rerun with --leak-check=full to see details of leaked memory
==2739==
==2739== For counts of detected and suppressed errors, rerun with: -v
==2739== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
[root@servera ~]#

如果ls每次都泄露一点内存，那就完蛋了，好在lack为0

[root@servera ~]# cp /usr/local/bin/bigmem .
[root@servera ~]# valgrind --tool=memcheck ./bigmem 512M
==2863== Memcheck, a memory error detector
==2863== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2863== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==2863== Command: ./bigmem 512M
==2863==
Process PID: 2863
Allocating 512 MiB of resident memory (in 1 MiB chunks)...
Done
Press <Enter> to exit
==2863==
==2863== HEAP SUMMARY:
==2863==     in use at exit: 536,870,912 bytes in 512 blocks
==2863==   total heap usage: 514 allocs, 2 frees, 536,872,960 bytes allocated
==2863==
==2863== LEAK SUMMARY:
==2863==    definitely lost: 531,628,032 bytes in 507 blocks
==2863==    indirectly lost: 0 bytes in 0 blocks
==2863==      possibly lost: 5,242,880 bytes in 5 blocks
==2863==    still reachable: 0 bytes in 0 blocks
==2863==         suppressed: 0 bytes in 0 blocks
==2863== Rerun with --leak-check=full to see details of leaked memory
==2863==
==2863== For counts of detected and suppressed errors, rerun with: -v
==2863== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
[root@servera ~]#

definitely lost: 531,628,032 bytes in 507 blocks

bigmem得执行第一条copy到其他文件夹下

重启系统，就可以释放

windows两年不重启，就很慢

oom-kill is not a fix for memory leaks

内存泄漏，kill机制无法解决 （kill保护系统自救而已，内存不够还泄漏，kill就不行了）

应用程序设计有问题

swap

当内存不够用的时候，会把一部分资源放到swap里去

swap空间最好放在固态里

swap最大才合适



用分区不用文件

[root@servera system]# mkswap /dev/vdd1
Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
no label, UUID=9a5b7ced-ec93-4e77-a54d-4706b59afef0
[root@servera system]# mkswap /dev/vdc1
Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
no label, UUID=f4791546-5242-4ca1-be7a-604f928a5985
[root@servera system]# vi /etc/fstab
[root@servera system]# free -m
            total        used        free      shared  buff/cache   available
Mem:           1829         188        1478          16         161        1482
Swap:             0           0           0
[root@servera system]# vi /etc/fstab
UUID=f4791546-5242-4ca1-be7a-604f928a5985    swap swap default 0 0
UUID=9a5b7ced-ec93-4e77-a54d-4706b59afef0    swap swap default 0 0
[root@servera system]# swapon -a
[root@servera system]# free -m
            total        used        free      shared  buff/cache   available
Mem:           1829         189        1477          16         161        1481
Swap:          2047           0        2047
[root@servera system]# swapon -s
Filename				Type		Size	Used	Priority
/dev/vdc1                              	partition	1048572	0	-2
/dev/vdd1                              	partition	1048572	0	-3
[root@servera system]#

数字越大优秀越高

UUID=f4791546-5242-4ca1-be7a-604f928a5985    swap swap default,pri=1 0 0
UUID=9a5b7ced-ec93-4e77-a54d-4706b59afef0    swap swap default,pri=2 0 0

[root@servera system]# swapon -s
Filename				Type		Size	Used	Priority
/dev/vdc1                              	partition	1048572	0	1
/dev/vdd1                              	partition	1048572	0	2

当我bigmem2000M时，swap优先级为2的开始运行，1的没用

[root@servera ~]# watch -n 1 'swapon -s'
Filename                                Type            Size    Used    Priority
/dev/vdc1                               partition	1048572 0	1
/dev/vdd1                               partition	1048572 532480  2

[root@servera ~]# watch -n 1 'swapon -s'
Filename                                Type            Size    Used    Priority
/dev/vdc1                               partition	1048572 0	1
/dev/vdd1                               partition	1048572 532480  1

Filename                                Type            Size    Used    Priority
/dev/vdc1                               partition	1048572 250632  1
/dev/vdd1                               partition	1048572 250944  1
均衡的使用，减少每个硬盘的压力
没swap要不申请不了2000 要不kill杀进程 要不死机
[root@servera system]# swapoff -a  这个命令很慢是因为，他得把swap的东西放回内存

kill也会杀死内存消耗很高的家伙

256G的内存一旦紧张，那么2Gswap远远不够。瞬间很高时，把暂时不用的放到swap里来

create up to 32 swap
最多32个swap设备。（可以实现均衡 但32有些夸张）

NUMA

非一致性内存访问

正常是通过前端总线来交换，前端总线是瓶颈

[root@servera system]# yum -y install numactl
[root@servera system]# numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1
node 0 size: 1829 MB
node 0 free: 1400 MB
node distances:
node   0
0:  10

[root@workstation ~]# lab memtuning-numa start

[root@servera grub2]# numastat -c

Per-node numastat info (in MBs):
                Node 0 Node 1 Node 2 Node 3 Total
                ------ ------ ------ ------ -----
Numa_Hit           929   1429     27     28  2413
Numa_Miss          185      0   1873   1216  3273
Numa_Foreign         0   3273      0      0  3273
Interleave_Hit      26     26     26     26   104
Local_Node         905   1425      1      1  2333
Other_Node         209      4   1899   1242  3354

[root@servera ~]# numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1
node 0 size: 478 MB
node 0 free: 305 MB
node 1 cpus: 0 1
node 1 size: 343 MB
node 1 free: 118 MB
node 2 cpus: 0 1
node 2 size: 503 MB
node 2 free: 499 MB
node 3 cpus: 0 1
node 3 size: 502 MB
node 3 free: 497 MB
node distances:
node   0   1   2   3
0:  10  10  10  10
1:  10  10  10  10
2:  10  10  10  10
3:  10  10  10  10

numa 他们之间的距离不可能都是10

这个是假的

真的，可以通过这个来判断numa之间的距离

红帽的lab命令改了grub，生成了grub2文件。并重启

[root@foundation0 boot]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="resume=UUID=c3393fce-c67b-4d64-962c-bfc6f9f0fbfa rhgb quiet rd.shell=0"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

[root@servera ~]# bigmem 1204M

[root@servera grub2]# numastat  -c bigmem

Per-node process memory usage (in MBs) for PID 1638 (bigmem)
        Node 0 Node 1 Node 2 Node 3 Total
        ------ ------ ------ ------ -----
Huge          0      0      0      0     0
Heap          0      0      0      0     0
Stack         0      0      0      0     0
Private     182     96    467    465  1210
-------  ------ ------ ------ ------ -----
Total       182     97    467    465  1210

numa生效了，这个内存在四个numa里

[root@servera grub2]# numactl --membind 1 -- bigmem 128M

[root@servera ~]# numastat -c bigmem

Per-node process memory usage (in MBs) for PID 1686 (bigmem)
        Node 0 Node 1 Node 2 Node 3 Total
        ------ ------ ------ ------ -----
Huge          0      0      0      0     0
Heap          0      0      0      0     0
Stack         0      0      0      0     0
Private       0    130      0      0   130
-------  ------ ------ ------ ------ -----
Total         0    130      0      0   130
[root@servera ~]# 

[root@servera grub2]# numactl --membind 1,2 -- bigmem 128M
指定numa在1,3里运行

1500多兆的内存分给4个numa

每个numa没多大

你非要分500在一个numanode，那么会失败

倾向于在numanode1

[root@servera grub2]# numactl --preferred=1 -- bigmem 210M
Process PID: 1746
Allocating 210 MiB of resident memory (in 1 MiB chunks)...

[root@servera ~]# numastat -c bigmem

Per-node process memory usage (in MBs) for PID 1746 (bigmem)
        Node 0 Node 1 Node 2 Node 3 Total
        ------ ------ ------ ------ -----
Huge          0      0      0      0     0
Heap          0      0      0      0     0
Stack         0      0      0      0     0
Private       0    141     72      0   212
-------  ------ ------ ------ ------ -----
Total         0    141     72      0   212

星型结构，numa之间距离相等

numactl --cpunodebind=2 --prefered=2 -- mygrogram

cpu绑定2 内存倾向使用2

numactl --cpunodebind=2 --membind=2,3 -- mygrogram

cpu绑定2，内存使用2,3

numactl --interleave all -- mydatabase

numaoff 使用所有numa资源

numa结构自动调整

选中这个功能，虚拟机运行时，运行在一个numanode里

tmpfs

[root@servera shm]# dd if=/dev/zero of=test1 bs=1M count=512 oflag=direct
dd: failed to open 'test1': Invalid argument
[root@servera shm]# dd if=/dev/zero of=test1 bs=1M count=512
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 0.468107 s, 1.1 GB/s
[root@servera shm]# ls
test1
[root@servera shm]# 

[root@servera grub2]# free -m
            total        used        free      shared  buff/cache   available
Mem:           1828         183        1007         528         636         927
Swap:             0           0           0
[root@servera grub2]#

缓存并大

且sysctl vm.drop_caches=3 清理不掉缓存

tmpfs虚拟内存文件系统

访问速度更快

把内存当做磁盘来使用

提升速度很快

cache服务器



第一次去internet里读，随后存到缓存

之后就一直去缓存里读

只加速，可靠性不高，但是它只是做缓存呢，丢缓存不要紧

[root@servera shm]# yum -y install squid

[root@servera shm]# vim /etc/squid/squid.conf
:65
#cache_dir ufs /var/spool/squid 100 16 256

100M 16个一级目录 256个二级目录
[root@servera shm]# mount --bind /dev/shm/  /var/spool/squid/
[root@servera shm]# cd /var/spool/squid/
[root@servera squid]# ls
test1

/etc/fstab

/dev/shn /var/spool/squid none default,bind 0 0

将一个已被挂载目录挂载给另一个目录

[root@servera ~]# mount --bind /dev/shm/  /var/spool/squid/
[root@servera ~]# cd /var/spool/squid/
[root@servera squid]# ls
[root@servera squid]# ls
[root@servera squid]# systemctl start squid.service
[root@servera squid]# ls
squid-cf__metadata.shm  squid-cf__queues.shm  squid-cf__readers.shm

重要

mkdir /data /redhat
mount --bind /data /redhat

有时候就是需要通过cp拷贝，这个就可以通过挂载

tmpfs默认为内存的一半

如何改变呢

nodev  /dev/shm  tmpfs defaults,size=1400M 0 0
[root@servera data]# mount -o remount /dev/shm/
[root@servera data]# df -hT
Filesystem     Type      Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs  891M     0  891M   0% /dev
tmpfs          tmpfs     1.4G   12K  1.4G   1% /dev/shm

cgroup与tmpfs都是内存中机制

shm

shm: shared memory

[root@servera data]# free -m
            total        used        free      shared  buff/cache   available
Mem:           1828         199        1395          16         232        1369
Swap:          2047           0        2047

shared共享的内存

多个进程用同一个内存

进程间通讯

[root@servera data]# sysctl -a | grep shm
kernel.shm_next_id = -1
kernel.shm_rmid_forced = 0
kernel.shmall = 18446744073692774399   整个系统只能设置多少内存用于共享内存  单位：页     shmmax  *  shmmni  < shmall
kernel.shmmax = 18446744073692774399   每个内存段最最多申请多少内存 单位：字节    一个进程最多申请多少内存
kernel.shmmni = 4096   系统中最多允许的共享内存段的数据 最多允许多少个进程来申请共享内存    单位：个数
vm.hugetlb_shm_group = 0

一个进程为一个内存段

给数据库用，正常的应用也会使用

man proc

/shmall

/proc/sys/kernel/shmall (since Linux 2.2)
            This file contains the system-wide limit on the total number of pages of System V shared memory.

多个进程运行同一个资源

shm案例

一个机器运行了500多天

服务器内存只有32G 用了26G 6G可用

swap32G swap也用了18G

DB2数据库用了40G内存

pmap 11345

哪个文件占用了太多内存



shm 共享内存

共享内存给到oracle，系统中只要好资源的都可以用

[root@servera data]# bc
bc 1.07.1
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006, 2008, 2012-2017 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
16777216*4096/1024/1024/1024
64

设置高了shmall oracle跑起来就会消耗很多资源，不用oracle，它也会一直持续消耗资源

多少内存设置为共享内存给oracle来用？

共享内存16G给oracle

正常内存16G给其他应用

降低资源调整shmall

** vi 忽略大小写 set ignorecase**

man systemd.service