[ceph-users] S3 object-size based storage placement policy

David Watzke watzke at avast.com
Wed Nov 1 09:05:49 PDT 2017


Hello everyone,

I would like to implement an object-size based pool-placement policy to 
be used with the S3 API and I'd like to ask for some insights.

In particular, I would like to automatically store objects (via S3 API) 
into different pools based on their size. I.e. <64K objects to a SSD 
pool with replication factor 3 and any larger objects to HDD-based pool 
with erasure coding 4+2.

I would like to do this seamlessly for (S3 API) users - small objects 
automatically go to the quick storage, large objects go elsewhere. Then 
when you fetch the objects, you also shouldn't need to worry about this. 
I don't mind using some external tools to achieve this (nginx reverse 
proxy in front of the object gateway) if needed, as long as the approach 
is reliable.

I found this howto describing how to put objects using a custom storage 
policy via the Swift API:

https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/object_gateway_guide_for_red_hat_enterprise_linux/administration_cli#storage-policies

I was looking for a similiar S3 example and only stumbled upon this 
(couldn't find any other relevant info):

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-September/004998.html

It says that an S3 bucket has to be created with a location constraint 
so I guess it's not possible to specify the storage policy per-object by 
sending a header just like in Swift API? (which would be easily 
achievable with a reverse proxy in front of the object gateway)

Is there some other way to achieve my goal?

Thanks in advance,

-- 
David Watzke



More information about the ceph-users mailing list