[ceph-users] bucket indices: ssd-only or is a large fast block.db sufficient?
mnelson at redhat.com
Tue Nov 20 08:11:30 PST 2018
One consideration is that you may not be able to fit higher DB levels on
the db partition and end up with a lot of waste (Nick Fisk recently saw
this on his test cluster). We've talked about potentially trying to
pre-compute the hierarchy sizing so that we can align a level boundary
to fit within the db partition size. I'm concerned there could be some
unintended consequences (IE having a media transition and a write-amp
jump hit at the same time) I tend to wonder if we should focus on
either DB or column family sharding and just get some fraction of the
high level SSTs from different shards on the db partition.
I think you could probably make either configuration work, and there are
advantages and disadvantages to each approach. (sizing, complexity,
write-amp, etc). If you go for the 2nd option, you probably still want
some portion of the SSDs carved out for DB/WAL for the data pool which
would shrink how much you'd have available for the flash-only OSDs. One
point I do want to bring up is that we're considering experimenting with
layering bucket index pools on top of objects rather than using OMAP.
No idea if that will pan out (or even how far we'll get), but if that
ends up being a win, you might prefer the second approach as the objects
would end up on flash. The 2nd approach is also the only option as far
as filestore goes, though I'm not sure if that really matters to you guys.
On 11/20/18 8:48 AM, Gregory Farnum wrote:
> Looks like you’ve considered the essential points for bluestore OSDs,
> yep. :)
> My concern would just be the surprisingly-large block.db requirements
> for rgw workloads that have been brought up. (300+GB per OSD, I think
> someone saw/worked out?).
> On Tue, Nov 20, 2018 at 1:35 AM Dan van der Ster <dan at vanderster.com
> <mailto:dan at vanderster.com>> wrote:
> Hi ceph-users,
> Most of our servers have 24 hdds plus 4 ssds.
> Any experience how these should be configured to get the best rgw
> We have two options:
> 1) All osds the same, with data on the hdd and block.db on a 40GB
> ssd partition
> 2) Two osd device types: hdd-only for the rgw data pool and
> ssd-only for bucket index pool
> But all of the bucket index data is in omap, right?
> And all of the omap is stored in the rocks db, right?
> After reading the recent threads about bluefs slow_used_bytes, I had
> the thought that as long as we have a large enough block.db, then
> slow_used_bytes will be 0 and all of the bucket indexes will be on
> ssd-only, regardless of option (1) or (2) above.
> Any thoughts?
> ceph-users mailing list
> ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com>
> ceph-users mailing list
> ceph-users at lists.ceph.com
More information about the ceph-users