[ceph-users] bcache, dm-cache support

Kjetil Joergensen kjetil at medallia.com
Wed Oct 10 11:47:49 PDT 2018


We tested bcache, dm-cache/lvmcache, and one more which name eludes me with
PCIe NVME on top of large spinning rust drives behind a SAS3 expander - and
decided this were not for us.

This was probably jewel with filestore, and our primary reason for trying
to go down this path were that leveldb compaction were killing us, and
putting omap/leveldb and things on separate locations were "so-so"
supported (IIRC: some were explicitly supported, some you could do a bit of
symlink or mount trickery).

The caching worked - although, when we started doing power failure
survivability (power cycle the entire rig, wait for recovery, repeat), we
ended up with seriously corrupted the XFS filesystems on top of the cached
block device within a handful of power cycles). We did not test fully
disabling the spinning rust on-device cache (which were the leading
hypothesis of why this actually failed, potentially combined with ordering
of FLUSH+FUA ending up slightly funky combined with the rather asymmetric
commit latency). Just to rule out anything else, we did run the same
power-fail test regimen for days without the
nvme-over-spinning-rust-caching, without triggering the same filesystem

So yea - I'd recommend looking at i.e. bluestore and stick rocksdb, journal
and anything else performance critical on faster storage instead.

If you do decide to go down the dm-cache/lvmcache/(other cache) road - I'd
recommend throughly testing failure scenarios like i.e. power-loss so you
don't find out accidentally when you do have a multi-failure-domain outage.

- KJ

On Thu, Oct 4, 2018 at 3:42 AM Maged Mokhtar <mmokhtar at petasan.org> wrote:

> Hello all,
> Do  bcache and dm-cache work well with Ceph ? Is one recommended on the
> other ? Are there any issues ?
> There are a few posts in this list around them, but i could not
> determine if they are ready for mainstream use or not
> Appreciate any clarifications.  /Maged
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Kjetil Joergensen <kjetil at medallia.com>
SRE, Medallia Inc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181010/4a82218e/attachment.html>

More information about the ceph-users mailing list