[ceph-users] CephFS 12.2.0 -> 12.2.1 change in inode caching behaviour

David C dcsysengineer at gmail.com
Mon Nov 27 04:44:49 PST 2017


Yep, that did it! Thanks, Zheng. I should read release notes more carefully!

On Fri, Nov 24, 2017 at 7:09 AM, Yan, Zheng <ukernel at gmail.com> wrote:

> On Thu, Nov 23, 2017 at 9:17 PM, David C <dcsysengineer at gmail.com> wrote:
> > Hi All
> >
> > I upgraded my 12.2.0 cluster to 12.2.1 a month or two back. I've noticed
> > that the number of inodes held in cache is only approx 1/5th of my
> > inode_max. This is a surprise to me as with 12.2.0, and before that
> Jewel,
> > after starting an MDS server, the cache would typically fill to the max
> > within 24 hours. I have ~8 million entries my file system, and most of
> this
> > is fairly hot data. I'm seeing quite frequent "failing to respond to
> cache
> > pressure" messages, I just have two Kernel clients accessing the
> filesystem.
> >
> > Are there some new defaults I need to change perhaps? Or potentially a
> bug?
> >
>
> we introduced config option 'mds_cache_memory_limit'.
>
>
> Regards
> Yan, Zheng
>
> > Output of perf dump mds:
> >
> >>> "mds": {
> >>>
> >>>         "request": 184132091,
> >>>
> >>>         "reply": 184132064,
> >>>
> >>>         "reply_latency": {
> >>>
> >>>             "avgcount": 184132064,
> >>>
> >>>             "sum": 125364.905594355,
> >>>
> >>>             "avgtime": 0.000680842
> >>>
> >>>         },
> >>>
> >>>         "forward": 0,
> >>>
> >>>         "dir_fetch": 9846671,
> >>>
> >>>         "dir_commit": 562495,
> >>>
> >>>         "dir_split": 0,
> >>>
> >>>         "dir_merge": 0,
> >>>
> >>>         "inode_max": 2500000,
> >>>
> >>>         "inodes": 444642,
> >>>
> >>>         "inodes_top": 185845,
> >>>
> >>>         "inodes_bottom": 127878,
> >>>
> >>>         "inodes_pin_tail": 130919,
> >>>
> >>>         "inodes_pinned": 179149,
> >>>
> >>>         "inodes_expired": 135604208,
> >>>
> >>>         "inodes_with_caps": 165900,
> >>>
> >>>         "caps": 165948,
> >>>
> >>>         "subtrees": 2,
> >>>
> >>>         "traverse": 187280168,
> >>>
> >>>         "traverse_hit": 185739606,
> >>>
> >>>         "traverse_forward": 0,
> >>>
> >>>         "traverse_discover": 0,
> >>>
> >>>         "traverse_dir_fetch": 118150,
> >>>
> >>>         "traverse_remote_ino": 8,
> >>>
> >>>         "traverse_lock": 60256,
> >>>
> >>>         "load_cent": 18413221445,
> >>>
> >>>         "q": 0,
> >>>
> >>>         "exported": 0,
> >>>
> >>>         "exported_inodes": 0,
> >>>
> >>>         "imported": 0,
> >>>
> >>>         "imported_inodes": 0
> >>>
> >>>     }
> >>
> >>
> >
> > A few extra details:
> >
> > Running two MDS servers, one is standby.
> > Both have mds_cache_size = 2500000.
> > CentOS 7.3 servers.
> > Kernel clients are CentOS 7.3 (3.10.0-514.2.2.el7.x86_64)
> >
> > Thanks,
> > David
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users at lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171127/51897deb/attachment.html>


More information about the ceph-users mailing list