[ceph-users] Ceph cluster network bandwidth?

Anthony Verevkin anthony at verevkin.ca
Mon Nov 20 09:24:24 PST 2017

> From: "John Spray" <jspray at redhat.com>
> Sent: Thursday, November 16, 2017 11:01:35 AM
> On Thu, Nov 16, 2017 at 3:32 PM, David Turner <drakonstein at gmail.com>
> wrote:
> > That depends on another question.  Does the client write all 3
> > copies or
> > does the client send the copy to the primary OSD and then the
> > primary OSD
> > sends the write to the secondaries?  Someone asked this recently,
> > but I
> > don't recall if an answer was given.  I'm not actually certain
> > which is the
> > case.  If it's the latter then the 10Gb pipe from the client is all
> > you
> > need.
> The client sends the write to the primary OSD (via the public
> network)
> and the primary OSD sends it on to the two replicas (via the cluster
> network).
> John

Thank you John! Would you also know if the same is true for Erasure coding?
Is it the client or an OSD that is splitting the request into k+m chunks?
What about reads? Is it the client assembling the erasures or is the primary
OSD proxying each read request?

Also, for replicated sets people often forget that it's not just writes. When
the client is reading data, it comes from the primary OSD only and does not
generate extra traffic on the cluster network. So in the 50/50 read-write use 
case the public and cluster traffic would actually be balanced:
1x read + 1x write on public / 2x write replication on cluster.


More information about the ceph-users mailing list