场景描述:
收到zabbix监控报警,晋中生产机器出现磁盘空间不足报警。
远程到该服务器,排查占员工磁盘空间的原因,发现tomcat日志过多,于是清除3天前的日志。
日志清理后,发现磁盘空间没有释放,如下是处理过程和原因分析:
日志清理后,查看磁盘空间:
[root@jzreport2 ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_report2-LogVol01
ext4 83G 50G 29G 64% /
tmpfs tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 ext4 283M 41M 227M 16% /boot
原因应该是已经删除的文件仍然被某个进程占用,所以导致磁盘空间得不到释放,如下命令可以查看已经删除的空间爱你被哪些进程占用。
[root@jzreport2 ~]# lsof |grep -i delete
java 4825 root 13r REG 253,0 416582455 5377302 /geelyapp/tomcat-ime/logs/catalina-2016-07-26.out (deleted)
java 4825 root 15r REG 253,0 3815677 5377477 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-22.txt (deleted)
java 4825 root 16r REG 253,0 7384392 5377242 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-30.txt (deleted)
java 4825 root 18r REG 253,0 364016974 5377293 /geelyapp/tomcat-ime/logs/catalina-2016-07-28.out (deleted)
java 4825 root 24r REG 253,0 19699678 5377464 /geelyapp/tomcat-ime/logs/catalina-2016-07-22.out (deleted)
java 4825 root 27r REG 253,0 368883739 5377240 /geelyapp/tomcat-ime/logs/catalina-2016-07-30.out (deleted)
java 4825 root 28r REG 253,0 358485718 5377247 /geelyapp/tomcat-ime/logs/catalina-2016-07-31.out (deleted)
java 4825 root 31r REG 253,0 7239318 5377338 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-28.txt (deleted)
java 4825 root 32r REG 253,0 5689600 5382395 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-24.txt (deleted)
java 4825 root 33r REG 253,0 355019854 5377250 /geelyapp/tomcat-ime/logs/catalina-2016-08-01.out (deleted)
java 4825 root 35r REG 253,0 2988975349 5377294 /geelyapp/tomcat-ime/logs/catalina-2016-07-25.out (deleted)
java 4825 root 37r REG 253,0 5417099 5377320 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-27.txt (deleted)
java 4825 root 39r REG 253,0 7357142 5377249 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-31.txt (deleted)
java 4825 root 40r REG 253,0 371975503 5377318 /geelyapp/tomcat-ime/logs/catalina-2016-07-27.out (deleted)
java 4825 root 41r REG 253,0 5413317330 5382387 /geelyapp/tomcat-ime/logs/catalina-2016-07-23.out (deleted)
java 4825 root 42r REG 253,0 353928941 5377258 /geelyapp/tomcat-ime/logs/catalina-2016-08-02.out (deleted)
java 4825 root 44r REG 253,0 19902702 5382393 /geelyapp/tomcat-ime/logs/catalina-2016-07-24.out (deleted)
java 4825 root 45r REG 253,0 7333147 5377252 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-01.txt (deleted)
java 4825 root 46r REG 253,0 358733756 5377265 /geelyapp/tomcat-ime/logs/catalina-2016-08-03.out (deleted)
java 4825 root 48r REG 253,0 371727923 5377271 /geelyapp/tomcat-ime/logs/catalina-2016-08-04.out (deleted)
java 4825 root 50r REG 253,0 7328014 5377260 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-02.txt (deleted)
java 4825 root 51r REG 253,0 385891702 5377279 /geelyapp/tomcat-ime/logs/catalina-2016-08-05.out (deleted)
java 4825 root 54r REG 253,0 7336642 5377267 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-03.txt (deleted)
java 4825 root 55r REG 253,0 7446032 5377307 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-06.txt (deleted)
java 4825 root 61r REG 253,0 7375397 5377274 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-04.txt (deleted)
java 4825 root 62r REG 253,0 419590833 5377332 /geelyapp/tomcat-ime/logs/catalina-2016-08-08.out (deleted)
java 4825 root 63r REG 253,0 7534494 5377304 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-26.txt (deleted)
java 4825 root 64r REG 253,0 6619016 5382398 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-25.txt (deleted)
java 4825 root 65r REG 253,0 5664294 5382389 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-23.txt (deleted)
java 4825 root 66r REG 253,0 7408877 5377282 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-05.txt (deleted)
java 4825 root 67r REG 253,0 7379473 5377350 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-07-29.txt (deleted)
java 4825 root 69r REG 253,0 379479937 5377347 /geelyapp/tomcat-ime/logs/catalina-2016-07-29.out (deleted)
java 4825 root 70r REG 253,0 382905142 5377342 /geelyapp/tomcat-ime/logs/catalina-2016-08-09.out (deleted)
java 4825 root 72r REG 253,0 441397404 5377364 /geelyapp/tomcat-ime/logs/catalina-2016-08-11.out (deleted)
java 4825 root 73r REG 253,0 393485017 5377295 /geelyapp/tomcat-ime/logs/catalina-2016-08-06.out (deleted)
java 4825 root 76r REG 253,0 7386336 5377330 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-07.txt (deleted)
java 4825 root 77r REG 253,0 371199837 5377323 /geelyapp/tomcat-ime/logs/catalina-2016-08-07.out (deleted)
java 4825 root 79r REG 253,0 7475422 5377336 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-08.txt (deleted)
java 4825 root 81r REG 253,0 360021471 5377379 /geelyapp/tomcat-ime/logs/catalina-2016-08-13.out (deleted)
java 4825 root 83r REG 253,0 7401499 5377344 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-09.txt (deleted)
java 4825 root 84r REG 253,0 7421931 5377360 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-10.txt (deleted)
java 4825 root 86r REG 253,0 367824636 5377389 /geelyapp/tomcat-ime/logs/catalina-2016-08-15.out (deleted)
java 4825 root 88r REG 253,0 7594223 5377367 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-11.txt (deleted)
java 4825 root 89r REG 253,0 373697996 5377358 /geelyapp/tomcat-ime/logs/catalina-2016-08-10.out (deleted)
java 4825 root 90r REG 253,0 7218216 5377377 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-12.txt (deleted)
java 4825 root 94r REG 253,0 7207574 5377381 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-13.txt (deleted)
java 4825 root 96r REG 253,0 363153109 5377375 /geelyapp/tomcat-ime/logs/catalina-2016-08-12.out (deleted)
java 4825 root 97r REG 253,0 6953046 5377386 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-14.txt (deleted)
java 4825 root 100r REG 253,0 7443747 5377390 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-15.txt (deleted)
java 4825 root 102r REG 253,0 301548658 5377387 /geelyapp/tomcat-ime/logs/catalina-2016-08-14.out (deleted)
java 4825 root 104r REG 253,0 8306681 5377453 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-16.txt (deleted)
java 4825 root 105r REG 253,0 530882613 5377597 /geelyapp/tomcat-ime/logs/catalina-2016-08-19.out (deleted)
java 4825 root 110r REG 253,0 8246802 5377581 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-17.txt (deleted)
java 4825 root 111r REG 253,0 608724746 5377450 /geelyapp/tomcat-ime/logs/catalina-2016-08-16.out (deleted)
java 4825 root 113r REG 253,0 8433313 5377592 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-18.txt (deleted)
java 4825 root 114r REG 253,0 8173165 5377599 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-19.txt (deleted)
java 4825 root 117r REG 253,0 8061150 5377605 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-20.txt (deleted)
java 4825 root 119r REG 253,0 7948469 5377611 /geelyapp/tomcat-ime/logs/localhost_access_log.2016-08-21.txt (deleted)
java 4825 root 121r REG 253,0 612650394 5377589 /geelyapp/tomcat-ime/logs/catalina-2016-08-18.out (deleted)
java 4825 root 127r REG 253,0 627820458 5377580 /geelyapp/tomcat-ime/logs/catalina-2016-08-17.out (deleted)
java 4825 root 142r REG 253,0 475492585 5377603 /geelyapp/tomcat-ime/logs/catalina-2016-08-20.out (deleted)
zabbix_ag 15644 zabbix 3wW REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15646 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15647 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15648 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15649 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
zabbix_ag 15650 zabbix 3w REG 253,0 5 2097154 /tmp/zabbix_agentd.pid (deleted)
catalina. 25220 root 0u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
cronolog 25221 root 2u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
java 25232 root 0u CHR 136,4 0t0 7 /dev/pts/4 (deleted)
以上可以看出已经删除的tomcat日志被进程4825所占用:(本次的4825进程是logstash进程)
重启4825进程后,空间释放:
[root@jzreport2 logstash-2.1.1]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_report2-LogVol01
ext4 83G 32G 47G 41% /
tmpfs tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 ext4 283M 41M 227M 16% /boot
系统的日志采集在logstach中配置的是*号匹配日期,tomcat日志删除后,可能会出现空间不释放问题,可以参考上述处理过程进行处理。
- linux删除文件空间不释放问题解决
目录 场景描述 原因 解决方法 在线清空文件 场景描述 某天,收到Prometheus报警,生产中某台机器出现磁盘空间不足报警,该台服务器是mysql其中一台从库,远程登录到该服务后,排查磁盘空间的原 ...
- linux 删除文件空间未释放问题
现象:我们测试环境上,导出数据文件时,由于作业报错,重复导出,空间使用到达100%,按理说,导出的文件时在相同的路径下,文件名也是一致的,会自动替换. 那么之前导出的文件会被删除,问题就出现在删除这一 ...
- linux删除文件未释放空间问题处理
linux删除文件未释放空间问题处理 或者 /根分区满了 (我的根分区是/dev/sda1,/dev/sda1满了) http://blog.csdn.net/donghustone/article/ ...
- 【转】Linux删除文件未释放空间问题处理
linux里的文件被删除后,空间没有被释放是因为在Linux系统中,通过rm或者文件管理器删除文件将会从文件系统的目录结构上解除链接(unlink).然而如果文件是被打开的(有一个进程正在使用),那么 ...
- Linux 删除文件未释放空间问题处理,下清空或删除大文件
linux里的文件被删除后,空间没有被释放是因为在Linux系统中,通过rm或者文件管理器删除文件将会从文件系统的目录结构上解除链接(unlink).然而如果文件是被打开的(有一个进程正在使用),那么 ...
- linux删除文件未释放
https://access.redhat.com/solutions/2316 $ /usr/sbin/lsof | grep deleted ora 25575 data 33u REG 65,6 ...
- Linux文件删除空间未释放
当系统空间使用量过大需要清理空间或者清理某个文件时,有时会出现执行了删除命令之后磁盘空间并没有释放,很多人首次遇到该情况时会比较困惑,在考虑是不是像windows系统的回收站一样,删除只是逻辑删除到回 ...
- 解决linux删除文件后空间没有释放问题
linux删除文件后沒有释放空间 今天发现一台服务器的home空间满了,于是要清空没用的文件,当我删除文件后,发现可用空间沒有变化 os:centos4.7 现象: 发现当前磁盘空间使用情况: [ro ...
- Linux 删除文件后空间不释放【原创】
删除MySQL备份文件后,查找文件所在目录发现文件已经备删除了,但是空间没有释放,还是83% 解决方法: lsof|grep -i delete 发现进程还在,杀掉进程 kill -9 5377 再次 ...
随机推荐
- 在Ubuntu中安装了MongoDB后无法启动mongod的问题
今天准备学习MongoDB,没想到下载之后服务器端启动不了,记录一下问题和处理过程 一.安装 在Ubuntu中安装还是很简单,直接:sudo apt install mongodb 二.启动 启动Mo ...
- java输入输出 -- I/O模型简述
一.简介 本文向大家介绍五种I/O模型.分别是阻塞I/O.非阻塞I/O.I/O复用.信号驱动式I/O.异步I/O等.内容参考<UNIX网络编程>,大家想进深入学习网络编程,建议去读读这本书 ...
- 数据结构 -- 链表(LinkedList)
链表是一种物理存储单元上非连续.非顺序的存储结构,数据元素的逻辑顺序是通过链表中的指针链接次序实现的.链表由一系列结点(链表中每一个元素称为结点)组成,结点可以在运行时动态生成. 每个结点包括两个部分 ...
- 关于c++模板非类型参数中指针和引用类型必须为全局或者静态变量的问题
之前在学习c++模板的时候,一直没留意到在非类型参数中对指针和引用有着一些限制,今早在复学模板的时候才注意到书上标明,指针和引用作为模板的非类型参数传递时必须要求是全局或者静态变量.其实不难想到,模板 ...
- centos7 通过yum安装mysql
但是CentOS7的yum源中默认好像是没有mysql的.为了解决这个问题,我们要先下载mysql的repo源. 1.下载mysql的repo源 $ wget http://repo.mysql.co ...
- HADOOP 与 jupyterlab 链接
首先 咱们先把jdk1.0.0_60.tar.gz 和 hadoop-2.7.2.tar.gz 的压缩包放到root根目录下的opt文件夹下 如图: 然后 ...
- S03_CH07_AXI_VDMA_OV5640摄像头采集系统
S03_CH07_AXI_VDMA_OV5640摄像头采集系统 7.1概述 本章内容和<S03_CH06_AXI_VDMA_OV7725摄像头采集系统>只是摄像头采用的分辨率不同,其他原理 ...
- java字节和字符的区别
字节: 1.bit=1 二进制数据0或1 2.byte=8bit 1个字节等于8位 存储空间的基本计量单位 3.一个英文字母=1byte=8bit 1个英文字母是1个字节,也就是8位 4.一个汉字 ...
- js 根据 数组条件 简单查询的方法临时保存
let array = [{ date: '2016-05-02', name: 'Ethan', status: 'success', total: '81' }, { date: '2016-05 ...
- Optional 理解
目录 Optional 理解 1. 含义 2. Optional 类中方法 3. Optional 对象不应该作为方法参数 Optional 理解 1. 含义 Optional 是一个容器对象,该容器 ...