场景描述:
收到zabbix监控报警,晋中生产机器出现磁盘空间不足报警。
远程到该服务器,排查占员工磁盘空间的原因,发现tomcat日志过多,于是清除3天前的日志。
日志清理后,发现磁盘空间没有释放,如下是处理过程和原因分析:
日志清理后,查看磁盘空间:
[root@jzreport2 ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_report2-LogVol01
ext4 83G 50G 29G 64% /
tmpfs tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 ext4 283M 41M 227M 16% /boot
原因应该是已经删除的文件仍然被某个进程占用,所以导致磁盘空间得不到释放,如下命令可以查看已经删除的空间爱你被哪些进程占用。
[root@jzreport2 ~]# lsof |grep -i delete
java 4825 root 13r REG 253,0 416582455 5377302 /geelyapp/tomcat-ime/logs/catalina-2016-07-26.out (deleted)
java 4825 root 15r REG 253,0 3815677 5377477 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-22.txt (deleted)
java 4825 root 16r REG 253,0 7384392 5377242 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-30.txt (deleted)
java 4825 root 18r REG 253,0 364016974 5377293 /geelyapp/tomcat-ime/logs/catalina-2016-07-28.out (deleted)
java 4825 root 24r REG 253,0 19699678 5377464 /geelyapp/tomcat-ime/logs/catalina-2016-07-22.out (deleted)
java 4825 root 27r REG 253,0 368883739 5377240 /geelyapp/tomcat-ime/logs/catalina-2016-07-30.out (deleted)
java 4825 root 28r REG 253,0 358485718 5377247 /geelyapp/tomcat-ime/logs/catalina-2016-07-31.out (deleted)
java 4825 root 31r REG 253,0 7239318 5377338 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-28.txt (deleted)
java 4825 root 32r REG 253,0 5689600 5382395 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-24.txt (deleted)
java 4825 root 33r REG 253,0 355019854 5377250 /geelyapp/tomcat-ime/logs/catalina-2016-08-01.out (deleted)
java 4825 root 35r REG 253,0 2988975349 5377294 /geelyapp/tomcat-ime/logs/catalina-2016-07-25.out (deleted)
java 4825 root 37r REG 253,0 5417099 5377320 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-27.txt (deleted)
java 4825 root 39r REG 253,0 7357142 5377249 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-31.txt (deleted)
java 4825 root 40r REG 253,0 371975503 5377318 /geelyapp/tomcat-ime/logs/catalina-2016-07-27.out (deleted)
java 4825 root 41r REG 253,0 5413317330 5382387 /geelyapp/tomcat-ime/logs/catalina-2016-07-23.out (deleted)
java 4825 root 42r REG 253,0 353928941 5377258 /geelyapp/tomcat-ime/logs/catalina-2016-08-02.out (deleted)
java 4825 root 44r REG 253,0 19902702 5382393 /geelyapp/tomcat-ime/logs/catalina-2016-07-24.out (deleted)
java 4825 root 45r REG 253,0 7333147 5377252 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-01.txt (deleted)
java 4825 root 46r REG 253,0 358733756 5377265 /geelyapp/tomcat-ime/logs/catalina-2016-08-03.out (deleted)
java 4825 root 48r REG 253,0 371727923 5377271 /geelyapp/tomcat-ime/logs/catalina-2016-08-04.out (deleted)
java 4825 root 50r REG 253,0 7328014 5377260 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-02.txt (deleted)
java 4825 root 51r REG 253,0 385891702 5377279 /geelyapp/tomcat-ime/logs/catalina-2016-08-05.out (deleted)
java 4825 root 54r REG 253,0 7336642 5377267 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-03.txt (deleted)
java 4825 root 55r REG 253,0 7446032 5377307 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-06.txt (deleted)
java 4825 root 61r REG 253,0 7375397 5377274 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-04.txt (deleted)
java 4825 root 62r REG 253,0 419590833 5377332 /geelyapp/tomcat-ime/logs/catalina-2016-08-08.out (deleted)
java 4825 root 63r REG 253,0 7534494 5377304 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-26.txt (deleted)
java 4825 root 64r REG 253,0 6619016 5382398 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-25.txt (deleted)
java 4825 root 65r REG 253,0 5664294 5382389 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-23.txt (deleted)
java 4825 root 66r REG 253,0 7408877 5377282 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-05.txt (deleted)
java 4825 root 67r REG 253,0 7379473 5377350 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-29.txt (deleted)
java 4825 root 69r REG 253,0 379479937 5377347 /geelyapp/tomcat-ime/logs/catalina-2016-07-29.out (deleted)
java 4825 root 70r REG 253,0 382905142 5377342 /geelyapp/tomcat-ime/logs/catalina-2016-08-09.out (deleted)
java 4825 root 72r REG 253,0 441397404 5377364 /geelyapp/tomcat-ime/logs/catalina-2016-08-11.out (deleted)
java 4825 root 73r REG 253,0 393485017 5377295 /geelyapp/tomcat-ime/logs/catalina-2016-08-06.out (deleted)
java 4825 root 76r REG 253,0 7386336 5377330 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-07.txt (deleted)
java 4825 root 77r REG 253,0 371199837 5377323 /geelyapp/tomcat-ime/logs/catalina-2016-08-07.out (deleted)
java 4825 root 79r REG 253,0 7475422 5377336 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-08.txt (deleted)
java 4825 root 81r REG 253,0 360021471 5377379 /geelyapp/tomcat-ime/logs/catalina-2016-08-13.out (deleted)
java 4825 root 83r REG 253,0 7401499 5377344 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-09.txt (deleted)
java 4825 root 84r REG 253,0 7421931 5377360 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-10.txt (deleted)
java 4825 root 86r REG 253,0 367824636 5377389 /geelyapp/tomcat-ime/logs/catalina-2016-08-15.out (deleted)
java 4825 root 88r REG 253,0 7594223 5377367 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-11.txt (deleted)
java 4825 root 89r REG 253,0 373697996 5377358 /geelyapp/tomcat-ime/logs/catalina-2016-08-10.out (deleted)
java 4825 root 90r REG 253,0 7218216 5377377 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-12.txt (deleted)
java 4825 root 94r REG 253,0 7207574 5377381 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-13.txt (deleted)
java 4825 root 96r REG 253,0 363153109 5377375 /geelyapp/tomcat-ime/logs/catalina-2016-08-12.out (deleted)
java 4825 root 97r REG 253,0 6953046 5377386 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-14.txt (deleted)
java 4825 root 100r REG 253,0 7443747 5377390 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-15.txt (deleted)
java 4825 root 102r REG 253,0 301548658 5377387 /geelyapp/tomcat-ime/logs/catalina-2016-08-14.out (deleted)
java 4825 root 104r REG 253,0 8306681 5377453 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-16.txt (deleted)
java 4825 root 105r REG 253,0 530882613 5377597 /geelyapp/tomcat-ime/logs/catalina-2016-08-19.out (deleted)
java 4825 root 110r REG 253,0 8246802 5377581 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-17.txt (deleted)
java 4825 root 111r REG 253,0 608724746 5377450 /geelyapp/tomcat-ime/logs/catalina-2016-08-16.out (deleted)
java 4825 root 113r REG 253,0 8433313 5377592 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-18.txt (deleted)
java 4825 root 114r REG 253,0 8173165 5377599 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-19.txt (deleted)
java 4825 root 117r REG 253,0 8061150 5377605 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-20.txt (deleted)
java 4825 root 119r REG 253,0 7948469 5377611 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-21.txt (deleted)
java 4825 root 121r REG 253,0 612650394 5377589 /geelyapp/tomcat-ime/logs/catalina-2016-08-18.out (deleted)
java 4825 root 127r REG 253,0 627820458 5377580 /geelyapp/tomcat-ime/logs/catalina-2016-08-17.out (deleted)
java 4825 root 142r REG 253,0 475492585 5377603 /geelyapp/tomcat-ime/logs/catalina-2016-08-20.out (deleted)
zabbix_ag 15644 zabbix 3wW REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15646 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15647 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15648 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15649 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15650 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
catalina. 25220 root 0u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
cronolog 25221 root 2u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
java 25232 root 0u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
以上可以看出已经删除的tomcat日志被进程4825所占用:(本次的4825进程是logstash进程)
重启4825进程后,空间释放:
[root@jzreport2 logstash-2.1.1]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_report2-LogVol01
ext4 83G 32G 47G 41% /
tmpfs tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 ext4 283M 41M 227M 16% /boot
系统的日志采集在logstach中配置的是*号匹配日期,tomcat日志删除后,可能会出现空间不释放问题,可以参考上述处理过程进行处理。
- linux删除文件空间不释放问题解决
目录 场景描述 原因 解决方法 在线清空文件 场景描述 某天,收到Prometheus报警,生产中某台机器出现磁盘空间不足报警,该台服务器是mysql其中一台从库,远程登录到该服务后,排查磁盘空间的原 ...
- linux 删除文件空间未释放问题
现象:我们测试环境上,导出数据文件时,由于作业报错,重复导出,空间使用到达100%,按理说,导出的文件时在相同的路径下,文件名也是一致的,会自动替换. 那么之前导出的文件会被删除,问题就出现在删除这一 ...
- linux删除文件未释放空间问题处理
linux删除文件未释放空间问题处理 或者 /根分区满了 (我的根分区是/dev/sda1,/dev/sda1满了) http://blog.csdn.net/donghustone/article/ ...
- 【转】Linux删除文件未释放空间问题处理
linux里的文件被删除后,空间没有被释放是因为在Linux系统中,通过rm或者文件管理器删除文件将会从文件系统的目录结构上解除链接(unlink).然而如果文件是被打开的(有一个进程正在使用),那么 ...
- Linux 删除文件未释放空间问题处理,下清空或删除大文件
linux里的文件被删除后,空间没有被释放是因为在Linux系统中,通过rm或者文件管理器删除文件将会从文件系统的目录结构上解除链接(unlink).然而如果文件是被打开的(有一个进程正在使用),那么 ...
- linux删除文件未释放
https://access.redhat.com/solutions/2316 $ /usr/sbin/lsof | grep deleted ora 25575 data 33u REG 65,6 ...
- Linux文件删除空间未释放
当系统空间使用量过大需要清理空间或者清理某个文件时,有时会出现执行了删除命令之后磁盘空间并没有释放,很多人首次遇到该情况时会比较困惑,在考虑是不是像windows系统的回收站一样,删除只是逻辑删除到回 ...
- 解决linux删除文件后空间没有释放问题
linux删除文件后沒有释放空间 今天发现一台服务器的home空间满了,于是要清空没用的文件,当我删除文件后,发现可用空间沒有变化 os:centos4.7 现象: 发现当前磁盘空间使用情况: [ro ...
- Linux 删除文件后空间不释放【原创】
删除MySQL备份文件后,查找文件所在目录发现文件已经备删除了,但是空间没有释放,还是83% 解决方法: lsof|grep -i delete 发现进程还在,杀掉进程 kill -9 5377 再次 ...
随机推荐
- vue element-ui typescript tree报错 === Property 'getCheckedNodes' does not exist on type 'Element | Element[] | Vue | Vue[]'.
import { Tree } from 'element-ui' const ref = <Tree>this.$refs.tree ref.getCheckedNodes()
- php类常量
类常量类常量可以使用define定义,也可用const定义,但是在类的内部,只允许用const定义,类常量不能更新,也不能删除类常量通常是大写的,两个单词之间用下滑线连接,如MY_NATION类常量在 ...
- WUSTOJ 1287: B304(Java:355ms,C:8ms)
题目:
- Go语言学习笔记(7)——函数和方法
Go语言中同时有函数和方法! 函数: go程序必须要包含一个main函数.main函数不能有任何参数和返回值! 1. 定义方法示例: func max(num1, num2 int) int { // ...
- Nginx学习笔记(二):Nginx 连接处理 / 负载均衡
Connection 在 Nginx 中,connection 就是对 TCP 连接的封装,其中包括连接的 socket,读写事件 Nginx 处理连接流程: 解析配置文件,得到需要监听的端口和I ...
- Swarm系列7--存储介绍
存储介绍 1. 存储使用 与docker一样,在使用swarm服务级别的时候可以定义服务的存储需求, docker存储介绍参考: Docker之应用数据管理(volume/bind mount/tmp ...
- EDK2开发环境搭建
EDK2开发环境搭建 来源 https://blog.csdn.net/rikeyone/article/details/80759724 EDK2全称为“uEFI Development Kit”, ...
- Express无法解析POST请求的JSON参数
在用Express和MongoDB搭建后端开发环境时,可能会利用测试工具发送带有JSON格式数据的POST请求,那么这时就要利用request.body获取json数据,但此时可能遇到json无法解析 ...
- Async await 解析
Async 定义:使异步函数以同步函数的形式书写(Generator函数语法糖) 原理:将Generator函数和自动执行器spawn包装在一个函数里 形式:将Generator函数的*替换成asyn ...
- EditText编辑框
<?xml version="1.0" encoding="utf-8"?> <LinearLayout xmlns:android=&quo ...