场景描述:
收到zabbix监控报警,晋中生产机器出现磁盘空间不足报警。
远程到该服务器,排查占员工磁盘空间的原因,发现tomcat日志过多,于是清除3天前的日志。
日志清理后,发现磁盘空间没有释放,如下是处理过程和原因分析:
日志清理后,查看磁盘空间:
[root@jzreport2 ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_report2-LogVol01
ext4 83G 50G 29G 64% /
tmpfs tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 ext4 283M 41M 227M 16% /boot
原因应该是已经删除的文件仍然被某个进程占用,所以导致磁盘空间得不到释放,如下命令可以查看已经删除的空间爱你被哪些进程占用。
[root@jzreport2 ~]# lsof |grep -i delete
java 4825 root 13r REG 253,0 416582455 5377302 /geelyapp/tomcat-ime/logs/catalina-2016-07-26.out (deleted)
java 4825 root 15r REG 253,0 3815677 5377477 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-22.txt (deleted)
java 4825 root 16r REG 253,0 7384392 5377242 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-30.txt (deleted)
java 4825 root 18r REG 253,0 364016974 5377293 /geelyapp/tomcat-ime/logs/catalina-2016-07-28.out (deleted)
java 4825 root 24r REG 253,0 19699678 5377464 /geelyapp/tomcat-ime/logs/catalina-2016-07-22.out (deleted)
java 4825 root 27r REG 253,0 368883739 5377240 /geelyapp/tomcat-ime/logs/catalina-2016-07-30.out (deleted)
java 4825 root 28r REG 253,0 358485718 5377247 /geelyapp/tomcat-ime/logs/catalina-2016-07-31.out (deleted)
java 4825 root 31r REG 253,0 7239318 5377338 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-28.txt (deleted)
java 4825 root 32r REG 253,0 5689600 5382395 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-24.txt (deleted)
java 4825 root 33r REG 253,0 355019854 5377250 /geelyapp/tomcat-ime/logs/catalina-2016-08-01.out (deleted)
java 4825 root 35r REG 253,0 2988975349 5377294 /geelyapp/tomcat-ime/logs/catalina-2016-07-25.out (deleted)
java 4825 root 37r REG 253,0 5417099 5377320 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-27.txt (deleted)
java 4825 root 39r REG 253,0 7357142 5377249 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-31.txt (deleted)
java 4825 root 40r REG 253,0 371975503 5377318 /geelyapp/tomcat-ime/logs/catalina-2016-07-27.out (deleted)
java 4825 root 41r REG 253,0 5413317330 5382387 /geelyapp/tomcat-ime/logs/catalina-2016-07-23.out (deleted)
java 4825 root 42r REG 253,0 353928941 5377258 /geelyapp/tomcat-ime/logs/catalina-2016-08-02.out (deleted)
java 4825 root 44r REG 253,0 19902702 5382393 /geelyapp/tomcat-ime/logs/catalina-2016-07-24.out (deleted)
java 4825 root 45r REG 253,0 7333147 5377252 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-01.txt (deleted)
java 4825 root 46r REG 253,0 358733756 5377265 /geelyapp/tomcat-ime/logs/catalina-2016-08-03.out (deleted)
java 4825 root 48r REG 253,0 371727923 5377271 /geelyapp/tomcat-ime/logs/catalina-2016-08-04.out (deleted)
java 4825 root 50r REG 253,0 7328014 5377260 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-02.txt (deleted)
java 4825 root 51r REG 253,0 385891702 5377279 /geelyapp/tomcat-ime/logs/catalina-2016-08-05.out (deleted)
java 4825 root 54r REG 253,0 7336642 5377267 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-03.txt (deleted)
java 4825 root 55r REG 253,0 7446032 5377307 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-06.txt (deleted)
java 4825 root 61r REG 253,0 7375397 5377274 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-04.txt (deleted)
java 4825 root 62r REG 253,0 419590833 5377332 /geelyapp/tomcat-ime/logs/catalina-2016-08-08.out (deleted)
java 4825 root 63r REG 253,0 7534494 5377304 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-26.txt (deleted)
java 4825 root 64r REG 253,0 6619016 5382398 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-25.txt (deleted)
java 4825 root 65r REG 253,0 5664294 5382389 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-23.txt (deleted)
java 4825 root 66r REG 253,0 7408877 5377282 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-05.txt (deleted)
java 4825 root 67r REG 253,0 7379473 5377350 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-29.txt (deleted)
java 4825 root 69r REG 253,0 379479937 5377347 /geelyapp/tomcat-ime/logs/catalina-2016-07-29.out (deleted)
java 4825 root 70r REG 253,0 382905142 5377342 /geelyapp/tomcat-ime/logs/catalina-2016-08-09.out (deleted)
java 4825 root 72r REG 253,0 441397404 5377364 /geelyapp/tomcat-ime/logs/catalina-2016-08-11.out (deleted)
java 4825 root 73r REG 253,0 393485017 5377295 /geelyapp/tomcat-ime/logs/catalina-2016-08-06.out (deleted)
java 4825 root 76r REG 253,0 7386336 5377330 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-07.txt (deleted)
java 4825 root 77r REG 253,0 371199837 5377323 /geelyapp/tomcat-ime/logs/catalina-2016-08-07.out (deleted)
java 4825 root 79r REG 253,0 7475422 5377336 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-08.txt (deleted)
java 4825 root 81r REG 253,0 360021471 5377379 /geelyapp/tomcat-ime/logs/catalina-2016-08-13.out (deleted)
java 4825 root 83r REG 253,0 7401499 5377344 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-09.txt (deleted)
java 4825 root 84r REG 253,0 7421931 5377360 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-10.txt (deleted)
java 4825 root 86r REG 253,0 367824636 5377389 /geelyapp/tomcat-ime/logs/catalina-2016-08-15.out (deleted)
java 4825 root 88r REG 253,0 7594223 5377367 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-11.txt (deleted)
java 4825 root 89r REG 253,0 373697996 5377358 /geelyapp/tomcat-ime/logs/catalina-2016-08-10.out (deleted)
java 4825 root 90r REG 253,0 7218216 5377377 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-12.txt (deleted)
java 4825 root 94r REG 253,0 7207574 5377381 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-13.txt (deleted)
java 4825 root 96r REG 253,0 363153109 5377375 /geelyapp/tomcat-ime/logs/catalina-2016-08-12.out (deleted)
java 4825 root 97r REG 253,0 6953046 5377386 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-14.txt (deleted)
java 4825 root 100r REG 253,0 7443747 5377390 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-15.txt (deleted)
java 4825 root 102r REG 253,0 301548658 5377387 /geelyapp/tomcat-ime/logs/catalina-2016-08-14.out (deleted)
java 4825 root 104r REG 253,0 8306681 5377453 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-16.txt (deleted)
java 4825 root 105r REG 253,0 530882613 5377597 /geelyapp/tomcat-ime/logs/catalina-2016-08-19.out (deleted)
java 4825 root 110r REG 253,0 8246802 5377581 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-17.txt (deleted)
java 4825 root 111r REG 253,0 608724746 5377450 /geelyapp/tomcat-ime/logs/catalina-2016-08-16.out (deleted)
java 4825 root 113r REG 253,0 8433313 5377592 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-18.txt (deleted)
java 4825 root 114r REG 253,0 8173165 5377599 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-19.txt (deleted)
java 4825 root 117r REG 253,0 8061150 5377605 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-20.txt (deleted)
java 4825 root 119r REG 253,0 7948469 5377611 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-21.txt (deleted)
java 4825 root 121r REG 253,0 612650394 5377589 /geelyapp/tomcat-ime/logs/catalina-2016-08-18.out (deleted)
java 4825 root 127r REG 253,0 627820458 5377580 /geelyapp/tomcat-ime/logs/catalina-2016-08-17.out (deleted)
java 4825 root 142r REG 253,0 475492585 5377603 /geelyapp/tomcat-ime/logs/catalina-2016-08-20.out (deleted)
zabbix_ag 15644 zabbix 3wW REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15646 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15647 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15648 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15649 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15650 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
catalina. 25220 root 0u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
cronolog 25221 root 2u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
java 25232 root 0u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
以上可以看出已经删除的tomcat日志被进程4825所占用:(本次的4825进程是logstash进程)
重启4825进程后,空间释放:
[root@jzreport2 logstash-2.1.1]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_report2-LogVol01
ext4 83G 32G 47G 41% /
tmpfs tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 ext4 283M 41M 227M 16% /boot
系统的日志采集在logstach中配置的是*号匹配日期,tomcat日志删除后,可能会出现空间不释放问题,可以参考上述处理过程进行处理。
- linux删除文件空间不释放问题解决
目录 场景描述 原因 解决方法 在线清空文件 场景描述 某天,收到Prometheus报警,生产中某台机器出现磁盘空间不足报警,该台服务器是mysql其中一台从库,远程登录到该服务后,排查磁盘空间的原 ...
- linux 删除文件空间未释放问题
现象:我们测试环境上,导出数据文件时,由于作业报错,重复导出,空间使用到达100%,按理说,导出的文件时在相同的路径下,文件名也是一致的,会自动替换. 那么之前导出的文件会被删除,问题就出现在删除这一 ...
- linux删除文件未释放空间问题处理
linux删除文件未释放空间问题处理 或者 /根分区满了 (我的根分区是/dev/sda1,/dev/sda1满了) http://blog.csdn.net/donghustone/article/ ...
- 【转】Linux删除文件未释放空间问题处理
linux里的文件被删除后,空间没有被释放是因为在Linux系统中,通过rm或者文件管理器删除文件将会从文件系统的目录结构上解除链接(unlink).然而如果文件是被打开的(有一个进程正在使用),那么 ...
- Linux 删除文件未释放空间问题处理,下清空或删除大文件
linux里的文件被删除后,空间没有被释放是因为在Linux系统中,通过rm或者文件管理器删除文件将会从文件系统的目录结构上解除链接(unlink).然而如果文件是被打开的(有一个进程正在使用),那么 ...
- linux删除文件未释放
https://access.redhat.com/solutions/2316 $ /usr/sbin/lsof | grep deleted ora 25575 data 33u REG 65,6 ...
- Linux文件删除空间未释放
当系统空间使用量过大需要清理空间或者清理某个文件时,有时会出现执行了删除命令之后磁盘空间并没有释放,很多人首次遇到该情况时会比较困惑,在考虑是不是像windows系统的回收站一样,删除只是逻辑删除到回 ...
- 解决linux删除文件后空间没有释放问题
linux删除文件后沒有释放空间 今天发现一台服务器的home空间满了,于是要清空没用的文件,当我删除文件后,发现可用空间沒有变化 os:centos4.7 现象: 发现当前磁盘空间使用情况: [ro ...
- Linux 删除文件后空间不释放【原创】
删除MySQL备份文件后,查找文件所在目录发现文件已经备删除了,但是空间没有释放,还是83% 解决方法: lsof|grep -i delete 发现进程还在,杀掉进程 kill -9 5377 再次 ...
随机推荐
- jquery的ajax设置为同步
在使用$.get或者$.post的时候,前面加上 $.ajaxSettings.async = false; 使用完之后再设置为异步 $.ajaxSettings.async = true; 而在使用 ...
- C++比起C来新增的拓展
命名空间 register 在C语言横行的时代,为了加快运行速度,一些关键变量会被放入寄存器中,程序代码请求编译器把变量存入寄存器,然而C语言版的寄存器变量无法通过地址获得register变量.c++ ...
- Java变量与数据类型之三:数据类型与转义字符
大家好,小乐又来了,上一篇:乐字节Java变量与数据类型之二:Java常量与变量, 今天接着讲Java变量与数据类型之三:数据类型与转义字符 一.数据类型 1.定义 Java是一种强类型语言,针对每种 ...
- Tomcat部署WEB应用方式
罗列在Tomcat部署web应用的几种方法,供以后翻阅,本博文以helloapp应用为例 Tomcat目录介绍 简单目录介绍如下 bin目录:包含tomcat启动/关闭等脚本,支持linux.wind ...
- (三)Servlet 知识点总结(来自那些年的笔记)
(史上最全知识汇总)转载请贴上原文链接! 作者:淮左白衣 写于 2018年4月15日20:14:55 如果,碰巧你打开了本篇博客,相信我,你想要的servlet知识,这里应该都能找到!! 目录 (史上 ...
- Python列表推导
一. 列表推导式 ord() 函数是 chr() 函数(对于8位的ASCII字符串)或 unichr() 函数(对于Unicode对象)的配对函数, 它以一个字符(长度为1的字符串)作为参数,返回 ...
- 在Yii2中集成Markdown编辑器
安装命令: composer require ijackua/yii2-lepture-markdown-editor-widget:dev-master 可能会遇到的问题 如果在下载依赖包的过程中出 ...
- 【转】Visual Studio Code必备插件
先ctrl+shift+p,弹出命令面板-选中Extensions:Install Extensions 或者直接点击左侧栏这个扩展按钮(Ctrl+Shift+X) 然后左侧栏就会显示出很多插件,如图 ...
- 怎样用sql语句复制表table1到表table2的同时复制主键
原文:怎样用sql语句复制表table1到表table2的同时复制主键 在从table1表复制到table2的时候,我们会用语句: select * into table2 from table1 但 ...
- SQL logic error no such module: fts5 解决方案
因项目原因,需要使用SQLite的全文索引,用到了最新的fts5模块 但在咱们.net framwork中却会提示“SQL logic error no such module: fts5”:找不到f ...