[ceph-users] Separation of public/cluster networks

Micha Krause micha at krausam.de
Wed Nov 15 04:58:48 PST 2017


Hi,

I've build a few clusters with separated public/cluster network, but I'm wondering if this is really
the way to go.

http://docs.ceph.com/docs/jewel/rados/configuration/network-config-ref

states 2 reasons:

1. There is more traffic in the backend, which could cause latencies in the public network.

  Is a low latency public network really an advantage, if my cluster network has high latency?

2. Security: evil users could cause damage in the cluster net.

  Couldn't you cause the same kind, or even more damage in the public network?


On the other hand, if one host looses it's cluster network, it will report random OSDs down over the
remaining public net. (yes I know about the "mon osd min down reporters" workaround)


Advantages of a single, shared network:

1. Hosts with network problems, that can't reach other OSDs, all so can't reach the mon. So our mon server doesn't get conflicting informations.

2. Given the same network bandwidth overall, OSDs can use a bigger part of the bandwidth for backend traffic.

3. KISS principle.

So if my server has 4 x 10GB/s network should I really split them in 2 x 20GB/s (cluster/public) or am I
better off using 1 x 40GB/s (shared)?


Micha Krause


More information about the ceph-users mailing list