mds0: Many clients (191) failing to respond to cache pressure
cephfs时我们产品依赖的主要分布式操作系统,但似乎很不给面子,压力测试的时候经常出问题。
背景
集群环境出现的问题: mds0: Many clients (191) failing to respond to cache pressure
背景:三个节点,100多个客户端mount,服务器可用内存仅剩100MB,ceph报错如下:
[root@node1 ceph]# ceph -s
cluster 1338affa-2d3d-416e-9251-4aa6e9c20eef
health HEALTH_WARN
mds0: Many clients (191) failing to respond to cache pressure
monmap e1: 3 mons at {node1=192.168.0.1:6789/0,node2=192.168.0.2:6789/0,node3=192.168.0.3:6789/0}
election epoch 22, quorum 0,1,2 node1,node2,node3
fsmap e924: 1/1/1 up {0=node1=up:active}, 2 up:standby
osdmap e71: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v48336: 576 pgs, 3 pools, 82382 MB data, 176 kobjects
162 GB used, 5963 GB / 6126 GB avail
576 active+clean
client io 0 B/s rd, 977 kB/s wr, 19 op/s rd, 116 op/s wr
至今问题也没有解决。(我的意思是说没有弄清楚Capacity的机制,如果抱着解决不了问题,就解决提出问题的人的思路,可以参考第三部分。)
mds日志如下:
2019-11-12 16:00:17.679876 7fa6a5040700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 34.236623 secs
2019-11-12 16:00:17.679914 7fa6a5040700 0 log_channel(cluster) log [WRN] : slow request 34.236623 seconds old, received at 2019-11-12 15:59:43.326917: client_request(client.154893:13683 open #1000005cb77 2019-11-12 15:59:43.293037) currently failed to xlock, waiting
2019-11-12 16:03:27.614474 7fa6a5040700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 34.350555 secs
2019-11-12 16:03:27.614523 7fa6a5040700 0 log_channel(cluster) log [WRN] : slow request 34.350555 seconds old, received at 2019-11-12 16:02:53.263857: client_request(client.155079:5446 open #1000003e360 2019-11-12 16:02:54.011037) currently failed to xlock, waiting
2019-11-12 16:03:57.615297 7fa6a5040700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 64.351379 secs
2019-11-12 16:03:57.615322 7fa6a5040700 0 log_channel(cluster) log [WRN] : slow request 64.351379 seconds old, received at 2019-11-12 16:02:53.263857: client_request(client.155079:5446 open #1000003e360 2019-11-12 16:02:54.011037) currently failed to xlock, waiting
2019-11-12 16:03:58.181330 7fa6a5040700 0 log_channel(cluster) log [WRN] : client.155079 isn't responding to mclientcaps(revoke), ino 1000003e360 pending pAsxLsXsxFcb issued pAsxLsXsxFsxcrwb, sent 64.458260 seconds ago
后续的努力
自己找环境重现,用的一个测试服务器,安装了一个Ubuntu系统,然后进行测试。惊喜的发现,同一个客户端不管我mount多少个目录,与后端的连接始终都只有那两个。
但重现过程中还是出现类似的问题了。
mds0: Client ubuntu:guest failing to respond to capability release
静置一段时间之后出现了如下错误:
[root@ceph741 ~]# ceph -s
cluster 1338affa-2d3d-416e-9251-4aa6e9c20eef
health HEALTH_WARN
mds0: Client ubuntu:guest failing to respond to capability release
mds0: Client ubuntu:guest failing to advance its oldest client/flush tid
monmap e2: 3 mons at {ceph741=192.168.15.112:6789/0,ceph742=192.168.15.113:6789/0,ceph743=192.168.15.114:6789/0}
election epoch 38, quorum 0,1,2 ceph741,ceph742,ceph743
fsmap e8989: 1/1/1 up {0=ceph743=up:active}, 2 up:standby
osdmap e67: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v847657: 576 pgs, 3 pools, 20803 MB data, 100907 objects
44454 MB used, 241 GB / 284 GB avail
576 active+clean
client io 59739 B/s rd, 3926 kB/s wr, 58 op/s rd, 770 op/s wr
临时的解决办法
临时的解决办法就是把出问题的客户端干掉。
步骤主要命令:
ceph tell mds.0 session ls
ceph tell mds.0 session evict id=249632
其中id是问题client的id。那么问题客户端比其他客户端哪里不同呢,实话说,我也不知道,大家可以看下:
参考:
https://www.jianshu.com/p/d1e0e32346ac
http://www.talkwithtrend.com/Article/242905
https://www.jianshu.com/p/fa49e40f6133
mds0: Many clients (191) failing to respond to cache pressure的更多相关文章
- MDS 多活配置
CephFS 介绍及使用经验分享 阅读 1179 收藏 2 2019-01-14 原文链接:www.jianshu.com WebRTC SFU中发送数据包的丢失反馈juejin.im 目录 Ceph ...
- cephfs测试中出现的问题
最近重新对cephfs进行性能测试. 测试步骤: (1) 选取一个特地版本的操作系统内核,挂载20000个客户端; (2) 用iozone中的fileop工具,在每隔挂载点上都跑一个fileop进程; ...
- cephfs删除报nospace的问题
ceph Vol 45 Issue 2 CephFS: No space left on device After upgrading to 10.2.3 we frequently see mess ...
- Java性能提示(全)
http://www.onjava.com/pub/a/onjava/2001/05/30/optimization.htmlComparing the performance of LinkedLi ...
- Chapter 6 — Improving ASP.NET Performance
https://msdn.microsoft.com/en-us/library/ff647787.aspx Retired Content This content is outdated and ...
- smb.conf - Samba组件的配置文件
总览 SYNOPSIS smb.conf是Samba组件的配置文件,包含Samba程序运行时的配置信息.smb.conf被设计成可由swat (8)程序来配置和管理.本文件包含了关于smb.conf的 ...
- SQL Server 内存相关博文
Don’t confuse error 823 and error 832 本文大意: 错误832: A page that should have been const ...
- Goal driven performance optimization
When your goal is to optimize application performance it is very important to understand what goal d ...
- 如何使用event 10049分析定位library cache lock and library cache pin
Oracle Library Cache 的 lock 与 pin 说明 一. 相关的基本概念 之前整理了一篇blog,讲了Library Cache 的机制,参考: Oracle Library c ...
随机推荐
- LeetCode 79. Word Search单词搜索 (C++)
题目: Given a 2D board and a word, find if the word exists in the grid. The word can be constructed fr ...
- postgres判断字符串是否为时间,数字
时间判断函数定义: -- FUNCTION: public.isdate(character varying) -- DROP FUNCTION public.isdate(character var ...
- 2016年蓝桥别A组模拟训练
1. 网友年龄 某君新认识一网友. 当问及年龄时,他的网友说: “我的年龄是个2位数,我比儿子大27岁, 如果把我的年龄的两位数字交换位置,刚好就是我儿子的年龄” 请你计算:网友的年龄一共有多少种可能 ...
- [LeetCode] 281. Zigzag Iterator 之字形迭代器
Given two 1d vectors, implement an iterator to return their elements alternately. Example: Input: v1 ...
- Gogs配置(本地安装篇-Debian)
知识储备: 用过MySQL等 了解Linux最基本的操作 git常用操作 关于ssh 本文参考:linux上安装gogs搭建个人仓库 下载 https://github.com/gogs/gogs/r ...
- Flink及Storm、Spark主流流框架比较
转自:http://www.sohu.com/a/142553677_804130 引言 随着大数据时代的来临,大数据产品层出不穷.我们最近也对一款业内非常火的大数据产品 - Apache Flink ...
- windows10安装ubuntu双系统教程(初稿)
windows10安装ubuntu双系统教程(绝对史上最详细) Win10 Ubuntu16.04/Ubuntu18.04双系统完美安装 Windows10+Ubuntu18.04双系统安装成功心得( ...
- CountdownLatch例子
CountdownLatch 一个线程或者多个线程等待其他线程完成了再接着往下执行 public class CountDownLatchTest { ); private static Random ...
- Balking模式
Balking模式讲的是如果现在不合适执行这个操作,或者没必要执行这个操作,就停止处理,直接返回 自动保存功能的实现逻辑一般都是隔一定时间自动执行存盘操作,存盘操作的前提是文件做过修改,如果文件没有执 ...
- HUT 排序训练赛 F - 水果
Problem's Link Mean: 略. analyse: 使用结构体排序. 首先,定义一个结构体,用来存放输入的数据,然后就是输入,注意:这儿有一个小细节,输入数字,然后紧跟着输入字符串,这时 ...