[ceph-users] bucket cloning/writable snapshots

Fred Gansevles fred at betterbe.com
Sat Nov 18 00:49:49 PST 2017

Currently our company has +/- 50 apps where every app has its own
data-area on NFS.
We need to switch S3, using Ceph, as our new data layer with
every app using its own s3-bucket, equivalent to the NFS data-area.
The sizes of the data-areas, depending on the app, varies from 1.3 GB to
358 GB.

In order to test multiple versions of the app, we currently make a
writable snapshot of
the data-area to avoid copying or 'polluting' the original (i.e.
'production' data).
Since a snapshot is fast we can make multiple snapshots easy and discard

With Ceph, we would like to do something alike, i.e. 'fast' copying and
easily discardable.
The path we are trying to take is: make a 'clone' of the bucket (i.e.
snapshots) into a test-bucket.

Our design for this is the following:
- We need to have every app bucket in its own unique pool-set.
- (on the ceph-nodes) determine the pool-set that is used by the given
- (on the ceph-nodes) make snapshots of the pools and assign these
snapshots to
  an newly created (writable!) test-bucket.
- after the test is finished, the test-bucket can be removed (either
from the ceph-node
  or the test system).

The test-procedure that is testing the app is aware of its environment,
i.e.: it
'knows' that it has to do specials things to get a test-bucket. The app
itself isn not
aware of this, and just uses whatever bucket is passed to it.

This way the test procedure can use the test-bucket and does not
interfere with the
original data and the app can be run 'as-is' without changing the code.

I have the following question:
    Is this scenario at all possible?
    - if yes: how can I accomplish this?
    - if no: is this a design-flaw (it can be done, just not this way)
             or its simply not possible.


Best regards,

Fred Gansevles
/Devops Engineer/

Auke Vleerstraat 140 E 	*T* +31 (0) 53 48 00 680
7547 AN Enschede 	*E* fgansevles at betterbe.com
<mailto:fgansevles at betterbe.com> 	
CC no. 08097527
*M* +31 (0)6 30 262 174 <tel:+31%20%280%296%2030%20262%20174>
www.betterbe.com <https://www.betterbe.com>
BetterBe accepts no liability for the content of this email, or for the
consequences of any actions taken on the basis of the information
provided, unless that information is subsequently confirmed in writing.
If you are not the intended recipient you are notified that disclosing,
copying, distributing or taking any action in reliance on the contents
of this information is strictly prohibited.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171118/6a9fd265/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: acaljfbihadinhoa.png
Type: image/png
Size: 10832 bytes
Desc: not available
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171118/6a9fd265/attachment.png>

More information about the ceph-users mailing list