场景描述:
收到zabbix监控报警,晋中生产机器出现磁盘空间不足报警。
远程到该服务器,排查占员工磁盘空间的原因,发现tomcat日志过多,于是清除3天前的日志。
日志清理后,发现磁盘空间没有释放,如下是处理过程和原因分析:
日志清理后,查看磁盘空间:
[root@jzreport2 ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_report2-LogVol01
ext4 83G 50G 29G 64% /
tmpfs tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 ext4 283M 41M 227M 16% /boot
原因应该是已经删除的文件仍然被某个进程占用,所以导致磁盘空间得不到释放,如下命令可以查看已经删除的空间爱你被哪些进程占用。
[root@jzreport2 ~]# lsof |grep -i delete
java 4825 root 13r REG 253,0 416582455 5377302 /geelyapp/tomcat-ime/logs/catalina-2016-07-26.out (deleted)
java 4825 root 15r REG 253,0 3815677 5377477 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-22.txt (deleted)
java 4825 root 16r REG 253,0 7384392 5377242 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-30.txt (deleted)
java 4825 root 18r REG 253,0 364016974 5377293 /geelyapp/tomcat-ime/logs/catalina-2016-07-28.out (deleted)
java 4825 root 24r REG 253,0 19699678 5377464 /geelyapp/tomcat-ime/logs/catalina-2016-07-22.out (deleted)
java 4825 root 27r REG 253,0 368883739 5377240 /geelyapp/tomcat-ime/logs/catalina-2016-07-30.out (deleted)
java 4825 root 28r REG 253,0 358485718 5377247 /geelyapp/tomcat-ime/logs/catalina-2016-07-31.out (deleted)
java 4825 root 31r REG 253,0 7239318 5377338 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-28.txt (deleted)
java 4825 root 32r REG 253,0 5689600 5382395 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-24.txt (deleted)
java 4825 root 33r REG 253,0 355019854 5377250 /geelyapp/tomcat-ime/logs/catalina-2016-08-01.out (deleted)
java 4825 root 35r REG 253,0 2988975349 5377294 /geelyapp/tomcat-ime/logs/catalina-2016-07-25.out (deleted)
java 4825 root 37r REG 253,0 5417099 5377320 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-27.txt (deleted)
java 4825 root 39r REG 253,0 7357142 5377249 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-31.txt (deleted)
java 4825 root 40r REG 253,0 371975503 5377318 /geelyapp/tomcat-ime/logs/catalina-2016-07-27.out (deleted)
java 4825 root 41r REG 253,0 5413317330 5382387 /geelyapp/tomcat-ime/logs/catalina-2016-07-23.out (deleted)
java 4825 root 42r REG 253,0 353928941 5377258 /geelyapp/tomcat-ime/logs/catalina-2016-08-02.out (deleted)
java 4825 root 44r REG 253,0 19902702 5382393 /geelyapp/tomcat-ime/logs/catalina-2016-07-24.out (deleted)
java 4825 root 45r REG 253,0 7333147 5377252 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-01.txt (deleted)
java 4825 root 46r REG 253,0 358733756 5377265 /geelyapp/tomcat-ime/logs/catalina-2016-08-03.out (deleted)
java 4825 root 48r REG 253,0 371727923 5377271 /geelyapp/tomcat-ime/logs/catalina-2016-08-04.out (deleted)
java 4825 root 50r REG 253,0 7328014 5377260 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-02.txt (deleted)
java 4825 root 51r REG 253,0 385891702 5377279 /geelyapp/tomcat-ime/logs/catalina-2016-08-05.out (deleted)
java 4825 root 54r REG 253,0 7336642 5377267 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-03.txt (deleted)
java 4825 root 55r REG 253,0 7446032 5377307 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-06.txt (deleted)
java 4825 root 61r REG 253,0 7375397 5377274 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-04.txt (deleted)
java 4825 root 62r REG 253,0 419590833 5377332 /geelyapp/tomcat-ime/logs/catalina-2016-08-08.out (deleted)
java 4825 root 63r REG 253,0 7534494 5377304 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-26.txt (deleted)
java 4825 root 64r REG 253,0 6619016 5382398 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-25.txt (deleted)
java 4825 root 65r REG 253,0 5664294 5382389 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-23.txt (deleted)
java 4825 root 66r REG 253,0 7408877 5377282 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-05.txt (deleted)
java 4825 root 67r REG 253,0 7379473 5377350 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-29.txt (deleted)
java 4825 root 69r REG 253,0 379479937 5377347 /geelyapp/tomcat-ime/logs/catalina-2016-07-29.out (deleted)
java 4825 root 70r REG 253,0 382905142 5377342 /geelyapp/tomcat-ime/logs/catalina-2016-08-09.out (deleted)
java 4825 root 72r REG 253,0 441397404 5377364 /geelyapp/tomcat-ime/logs/catalina-2016-08-11.out (deleted)
java 4825 root 73r REG 253,0 393485017 5377295 /geelyapp/tomcat-ime/logs/catalina-2016-08-06.out (deleted)
java 4825 root 76r REG 253,0 7386336 5377330 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-07.txt (deleted)
java 4825 root 77r REG 253,0 371199837 5377323 /geelyapp/tomcat-ime/logs/catalina-2016-08-07.out (deleted)
java 4825 root 79r REG 253,0 7475422 5377336 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-08.txt (deleted)
java 4825 root 81r REG 253,0 360021471 5377379 /geelyapp/tomcat-ime/logs/catalina-2016-08-13.out (deleted)
java 4825 root 83r REG 253,0 7401499 5377344 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-09.txt (deleted)
java 4825 root 84r REG 253,0 7421931 5377360 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-10.txt (deleted)
java 4825 root 86r REG 253,0 367824636 5377389 /geelyapp/tomcat-ime/logs/catalina-2016-08-15.out (deleted)
java 4825 root 88r REG 253,0 7594223 5377367 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-11.txt (deleted)
java 4825 root 89r REG 253,0 373697996 5377358 /geelyapp/tomcat-ime/logs/catalina-2016-08-10.out (deleted)
java 4825 root 90r REG 253,0 7218216 5377377 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-12.txt (deleted)
java 4825 root 94r REG 253,0 7207574 5377381 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-13.txt (deleted)
java 4825 root 96r REG 253,0 363153109 5377375 /geelyapp/tomcat-ime/logs/catalina-2016-08-12.out (deleted)
java 4825 root 97r REG 253,0 6953046 5377386 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-14.txt (deleted)
java 4825 root 100r REG 253,0 7443747 5377390 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-15.txt (deleted)
java 4825 root 102r REG 253,0 301548658 5377387 /geelyapp/tomcat-ime/logs/catalina-2016-08-14.out (deleted)
java 4825 root 104r REG 253,0 8306681 5377453 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-16.txt (deleted)
java 4825 root 105r REG 253,0 530882613 5377597 /geelyapp/tomcat-ime/logs/catalina-2016-08-19.out (deleted)
java 4825 root 110r REG 253,0 8246802 5377581 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-17.txt (deleted)
java 4825 root 111r REG 253,0 608724746 5377450 /geelyapp/tomcat-ime/logs/catalina-2016-08-16.out (deleted)
java 4825 root 113r REG 253,0 8433313 5377592 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-18.txt (deleted)
java 4825 root 114r REG 253,0 8173165 5377599 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-19.txt (deleted)
java 4825 root 117r REG 253,0 8061150 5377605 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-20.txt (deleted)
java 4825 root 119r REG 253,0 7948469 5377611 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-21.txt (deleted)
java 4825 root 121r REG 253,0 612650394 5377589 /geelyapp/tomcat-ime/logs/catalina-2016-08-18.out (deleted)
java 4825 root 127r REG 253,0 627820458 5377580 /geelyapp/tomcat-ime/logs/catalina-2016-08-17.out (deleted)
java 4825 root 142r REG 253,0 475492585 5377603 /geelyapp/tomcat-ime/logs/catalina-2016-08-20.out (deleted)
zabbix_ag 15644 zabbix 3wW REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15646 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15647 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15648 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15649 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15650 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
catalina. 25220 root 0u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
cronolog 25221 root 2u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
java 25232 root 0u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
以上可以看出已经删除的tomcat日志被进程4825所占用:(本次的4825进程是logstash进程)
重启4825进程后,空间释放:
[root@jzreport2 logstash-2.1.1]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_report2-LogVol01
ext4 83G 32G 47G 41% /
tmpfs tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 ext4 283M 41M 227M 16% /boot
系统的日志采集在logstach中配置的是*号匹配日期,tomcat日志删除后,可能会出现空间不释放问题,可以参考上述处理过程进行处理。
- linux删除文件空间不释放问题解决
目录 场景描述 原因 解决方法 在线清空文件 场景描述 某天,收到Prometheus报警,生产中某台机器出现磁盘空间不足报警,该台服务器是mysql其中一台从库,远程登录到该服务后,排查磁盘空间的原 ...
- linux 删除文件空间未释放问题
现象:我们测试环境上,导出数据文件时,由于作业报错,重复导出,空间使用到达100%,按理说,导出的文件时在相同的路径下,文件名也是一致的,会自动替换. 那么之前导出的文件会被删除,问题就出现在删除这一 ...
- linux删除文件未释放空间问题处理
linux删除文件未释放空间问题处理 或者 /根分区满了 (我的根分区是/dev/sda1,/dev/sda1满了) http://blog.csdn.net/donghustone/article/ ...
- 【转】Linux删除文件未释放空间问题处理
linux里的文件被删除后,空间没有被释放是因为在Linux系统中,通过rm或者文件管理器删除文件将会从文件系统的目录结构上解除链接(unlink).然而如果文件是被打开的(有一个进程正在使用),那么 ...
- Linux 删除文件未释放空间问题处理,下清空或删除大文件
linux里的文件被删除后,空间没有被释放是因为在Linux系统中,通过rm或者文件管理器删除文件将会从文件系统的目录结构上解除链接(unlink).然而如果文件是被打开的(有一个进程正在使用),那么 ...
- linux删除文件未释放
https://access.redhat.com/solutions/2316 $ /usr/sbin/lsof | grep deleted ora 25575 data 33u REG 65,6 ...
- Linux文件删除空间未释放
当系统空间使用量过大需要清理空间或者清理某个文件时,有时会出现执行了删除命令之后磁盘空间并没有释放,很多人首次遇到该情况时会比较困惑,在考虑是不是像windows系统的回收站一样,删除只是逻辑删除到回 ...
- 解决linux删除文件后空间没有释放问题
linux删除文件后沒有释放空间 今天发现一台服务器的home空间满了,于是要清空没用的文件,当我删除文件后,发现可用空间沒有变化 os:centos4.7 现象: 发现当前磁盘空间使用情况: [ro ...
- Linux 删除文件后空间不释放【原创】
删除MySQL备份文件后,查找文件所在目录发现文件已经备删除了,但是空间没有释放,还是83% 解决方法: lsof|grep -i delete 发现进程还在,杀掉进程 kill -9 5377 再次 ...
随机推荐
- php5.6安装及php-fpm优化配置
1,安装依赖包: yum install -y gcc gcc-c++ zlib zlib-devel pcre pcre-devel gd libjpeg libjpeg-devel libpn ...
- shell 批量重命名
原文链接https://www.cnblogs.com/sunmmi/articles/6709125.html shell 批量重命名 1.把文件名的第一字母批量改成a 方法一: 1 2 3 4 ...
- 乐字节Java变量与数据类型之一:Java编程规范,关键字与标识符
大家好,我是乐字节的小乐,这次要给大家带来的是Java变量与数据类型.本文是第一集:Java编程规范,关键字与标识符. 一.编程规范 任何地方的名字都需要见名知意: 代码适当缩进 书写过程成对编程 对 ...
- Z1. 广度优先搜索(BFS)解题思路
/** BFS 解题思路 特点:从某些特定的节点开始,感染相邻的节点; 被感染的节点,再感染其相邻的节点,以此类推. 题目常见于数据结构包括 二维数组.树.图 **/ /** 1). 二维数组特定节点 ...
- [转帖]18W喂不饱有必要买30W充电器吗?iPhone 11 Pro Max充电评测
18W喂不饱有必要买30W充电器吗?iPhone 11 Pro Max充电评测 https://www.cnbeta.com/articles/tech/895237.htm 改天买一个 设备玩一玩 ...
- oracle导入.dmp脚本
一.更改数据库管理员sys/system密码 1.运行到C盘根目录 2.输入:SET ORACLE_SID = 你的SID名称 3.输入:sqlplus /nolog 4.输入:connect /as ...
- 【51nod】1776 路径计数
[51nod]1776 路径计数 我们先把前两种数给排好,排好之后会有\(a + b + 1\)个空隙可以填数,我们计算有\(k\)个空隙两端都是相同字母的方案数 可以用枚举把第二种数分成几段插进去来 ...
- eclipse3.62开发第一个java程序HelloWorld
[学习笔记] 用eclipse3.62开发第一个java程序: 使用eclipse之前们需要先配置一下jdk.window/preference/java/installed JREs,详细请见视频. ...
- keras损失函数详解
以下信息均来自官网 ------------------------------------------------------------------------------------------ ...
- SAS学习笔记25 t检验(单个样本t检验、配对样本t检验、两个独立样本t检验及方差不齐时的t'检验)
根据研究设计和资料的性质有单个样本t检验.配对样本t检验.两个独立样本t检验以及在方差不齐时的t'检验 单样本t检验 单样本t检验(one-sample t-test)又称单样本均数t检验,适用于样本 ...