[ceph-users] Ceph block storage cluster limitations

Christian Balzer chibi at gol.com
Sun Mar 31 01:57:44 PDT 2019


essentially Anthony has given all the answers here.

Seeing as I'm sketching out a double digit PetaByte cluster ATM, I'm piping
up anyway.

On Sat, 30 Mar 2019 15:53:14 -0700 Anthony D'Atri wrote:

> > Hello,
> > 
> > I wanted to know if there are any max limitations on
> > 
> > - Max number of Ceph data nodes
> > - Max number of OSDs per data node
> > - Global max on number of OSDs
> > - Any limitations on the size of each drive managed by OSD?
> > - Any limitation on number of client nodes?
> > - Any limitation on maximum number of RBD volumes that can be created?  
For something you want to implement it probably wouldn't hurt to to shovel
some gold towards RedHat and get replies in writing from them. 
_IF_ they're willing to make such statments. ^o^

> I don’t think there any *architectural* limits, but there can be *practical* limits.  There are a lot of variables and everyone has a unique situation, but some thoughts:
> > Max number of Ceph data nodes  
> May be limited at some extreme by networking.  Don’t cheap out on your switches.
Not every OSD needs to talk to every other OSDs and the MON communication
is light, but yeah, it adds up. 
Note that I said OSDs, not actual hosts, they don't count in any traffic

> > - Max number of OSDs per data node  
> People have run at least 72.  Consider RAM required for a given set of drives, and that a single host/chassis isn’t a big percentage of your cluster.  Ie., don’t have a huge fault domain that will bite you later.  For a production cluster at scale I would suggest at least 12 OSD nodes, but this depends on lots of variables.  Conventional wisdom is 1GB RAM per 1TB of OSD; in practice for a large cluster I would favor somewhat more.  A cluster with, say, 3 nodes of 72 OSDs each is going to be in bad way when one fails.

Yes, proper sizing of failure domains is a must.
Looking at the recent bluestore discussions it feels like 4GB RAM per OSD
is a good starting point. That's something affected by your use case and
requirements (more RAM, more caching can be configured).

> > - Global max on number of OSDs  
> A cluster with at lest 10800 has existed.
> https://indico.cern.ch/event/542464/contributions/2202295/attachments/1289543/1921810/cephday-dan.pdf
> https://indico.cern.ch/event/649159/contributions/2761965/attachments/1544385/2423339/hroussea-storage-at-CERN.pdf
> The larger a cluster becomes, the more careful attention must be paid to topology and tuning.
Amen to that. 

For the same 10PB cluster I've come up with designs ranging from nearly 800
nodes and 13000 OSDs (3x replica, 16x 2.4TB HDDs per 2U node) to slightly
more than half of these numbers with the same HW but 10+4 erasure encoding.

These numbers made me more than slightly anxious and had other people (who
wanted to use that existing HE) outright faint or run away screaming. 

A different design with 60 effective OSDs (10TB HDDs) per 4U  requires
just 28 nodes and 1800 OSDs with 10+4 EC, much more manageable with
regards to numbers and rack space.

The same 4U hardware with 6 RAID6 (10 HDDs each) and thus 6 OSDs per node
and 3x replica on the Ceph level requires 64 nodes and results in only 384
If this is a feasible design will no doubt be debated here by purists. 
It has however the distinct advantage of extreme resilience and is very
unlikely to ever require a rebuild due to a failed OSD (HDD).
And given 1800 disks, failures are statistically going to be a common
It also allows for significantly less CPU resources due to no EC and far
fewer OSDs. 

For the record and to forestall some comments, this is for an object
storage (RGW) cluster dealing with largish (3MB average) objects, so IOPS
aren't the prime objective here.

See also my "Erasure Coding failure domain" mail just now.

> > Also, any advise on using NVMes for OSD drives?  
> They rock.  Evaluate your servers carefully:
> * Some may route PCI through a multi-mode SAS/SATA HBA
> * Watch for PCI bridges or multiplexing
> * Pinning, minimize data over QPI links
> * Faster vs more cores can squeeze out more performance 
> AMD Epyc single-socket systems may be very interesting for NVMe OSD nodes.
I'm happy that somebody else spotted this. ^o^



> > What is the known maximum cluster size that Ceph RBD has been deployed to?  
> See above.
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Rakuten Communications

More information about the ceph-users mailing list