[ceph-users] How To Scale Ceph for Large Numbers of Clients?

Zack Brenton zack at imposium.com
Wed Mar 6 13:18:32 PST 2019


Hello,

We're running Ceph on Kubernetes 1.12 using the Rook operator (
https://rook.io), but we've been struggling to scale applications mounting
CephFS volumes above 600 pods / 300 nodes. All our instances use the kernel
client and run kernel `4.19.23-coreos-r1`.

We've tried increasing the MDS memory limits, running multiple active MDS
pods, and running different versions of Ceph (up to the latest Luminous and
Mimic releases), but we run into MDS_SLOW_REQUEST errors at the same scale
regardless of the memory limits we set. See this GitHub issue for more info
on what we've tried up to this point:
https://github.com/rook/rook/issues/2590

I've written a simple load test that reads all the files in a given
directory on an interval. While running this test, I've noticed that the
`mds_co.bytes` value (from `ceph daemon mds.myfs-a dump_mempools | jq -c
'.mempool.by_pool.mds_co'`) increases each time files are read. Why is this
number increasing after the first iteration? If the same client is reading
the same cached files, why would the data in the cache change at all? What
is `mds_co.bytes` actually reporting?

My most important question is this: How do I configure Ceph to be able to
scale to large numbers of clients?

Thanks,
Zack
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20190306/fd39946e/attachment.html>


More information about the ceph-users mailing list