[ceph-users] Erasure code ruleset for small cluster

Caspar Smit casparsmit at supernas.eu
Fri Feb 2 08:13:59 PST 2018

Hi all,

I'd like to setup a small cluster (5 nodes) using erasure coding. I would
like to use k=5 and m=3.
Normally you would need a minimum of 8 nodes (preferably 9 or more) for

Then i found this blog:

This sounded ideal to me so i started building a test setup using the 5+3

Changed the erasure ruleset to:

rule erasure_ruleset {
  ruleset X
  type erasure
  min_size 8
  max_size 8
  step take default
  step choose indep 4 type host
  step choose indep 2 type osd
  step emit

Created a pool and now every PG has 8 shards in 4 hosts with 2 shards each,

But then i tested a node failure, no problem again, all PG's stay active
(most undersized+degraded, but still active). Then after 10 minutes the
OSD's on the failed node were all marked as out, as expected.

I waited for the data to be recovered to the other (fifth) node but that
doesn't happen, there is no recovery whatsoever.

Only when i completely remove the down+out OSD's from the cluster the data
is recovered.

My guess is that the "step choose indep 4 type host" chooses 4 hosts
beforehand to store data on.

Would it be possible to do something like this:

Create a 5+3 EC profile, every hosts has a maximum of 2 shards (so 4 hosts
are needed), in case of node failure -> recover data from failed node to
fifth node.

Thank you in advance,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20180202/9afa52dd/attachment.html>

More information about the ceph-users mailing list