[ceph-users] ceph luminous + multi mds: slow request. behind on trimming, failedto authpin local pins

Burkhard Linke Burkhard.Linke at computational.bio.uni-giessen.de
Wed Dec 6 23:40:14 PST 2017


we have upgraded our cluster to luminous 12.2.2 and wanted to use a 
second MDS for HA purposes. Upgrade itself went well, setting up the 
second MDS from the former standby-replay configuration worked, too.

But upon load both MDS got stuck and need to be restarted. It starts 
with slow requests:

2017-12-06 20:26:25.756475 7fddc4424700  0 log_channel(cluster) log 
[WRN] : slow
  request 122.370227 seconds old, received at 2017-12-06 
20:24:23.386136: client_
request(client.15057265:2898 getattr pAsLsXsFs #0x100009de0f2 2017-12-06 
3.244096 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting

0x100009de0f2 is the inode id of the directory we mount as root on most 
clients. Running daemonperf for both MDS shows a rising number of 
journal segments, accompanied with the corresponding warnings in the 
ceph log. We also see other slow requests:

2017-12-06 20:26:25.756488 7fddc4424700  0 log_channel(cluster) log 
[WRN] : slow
  request 180.346068 seconds old, received at 2017-12-06 
20:23:25.410295: client_
request(client.15163105:549847914 getattr pAs #0x100009de0f2/sge-tmp 
20:23:25.406481 caller_uid=1426, caller_gid=1008{}) currently failed to 
local pins

This is a client accessing a sub directory of the mount point.

On the client side (various Ubuntu kernel using kernel based cephfs) 
this leads to CPU lockups if the problem is not fixed fast enough. The 
clients need a hard reboot to recover.

We have mitigated the problem by disabling the second MDS. The MDS 
related configuration is:

mds_replay_interval = 10
mds_cache_memory_limit = 10737418240

mds_beacon_grace = 60
mds_beacon_interval = 4
mds_session_timeout = 120

Data pool is on replicated HDD storage, meta data pool on replicated 
NVME storage. MDS are colocated with OSDs (12 HDD OSDs + 2 NVME OSDs, 
128 GB RAM).

The questions are:

- what is the minimum kernel version on clients required for multi mds 

- is the problem described above a known problem, e.g. a result of 
http://tracker.ceph.com/issues/21975 ?


Burkhard Linke

More information about the ceph-users mailing list