[ceph-users] S3 object notifications

Yehuda Sadeh-Weinraub yehuda at redhat.com
Tue Nov 28 23:08:26 PST 2017


On Wed, Nov 29, 2017 at 12:43 AM,  <ceph.novice at habmalnefrage.de> wrote:
> Hi Yehuda.
>
> Are there any examples (doc's, blog posts, ...):
> - how to use that "framework" and especially for the "callbacks"

There's a minimal sync module implementation that does nothing other
than write a debug log for each sync event:

https://github.com/ceph/ceph/blob/master/src/rgw/rgw_sync_module_log.cc

Docs-wise, the elasticsearch luminous blog has a concrete config
example that can be applied to other sync modules:
http://ceph.com/rgw/new-luminous-rgw-metadata-search/


> - for the latest "Metasearch" feature / usage with a S3 client/tools like CyberDuck, s3cmd, AWSCLI or at least boto3?
>   - i.e. is an external ELK still needed or is this somehow included in RGW now?
>

You still need an external elasticsearch server, it's not part of
ceph. However, search can be done by sending requests through rgw's
RESTful api. We have a test that uses boto to generate such requests,
but might not be exactly what you're looking for:
https://github.com/ceph/ceph/blob/master/src/test/rgw/rgw_multi/zone_es.py

Yehuda

> Thanks & regards
>
>
> Gesendet: Dienstag, 28. November 2017 um 13:52 Uhr
> Von: "Yehuda Sadeh-Weinraub" <yehuda at redhat.com>
> An: "Sean Purdy" <s.purdy at cv-library.co.uk>
> Cc: "ceph-users at lists.ceph.com" <ceph-users at lists.ceph.com>
> Betreff: Re: [ceph-users] S3 object notifications
> rgw has a sync modules framework that allows you to write your own
> sync plugins. The system identifies objects changes and triggers
> callbacks that can then act on those changes. For example, the
> metadata search feature that was added recently is using this to send
> objects metadata into elasticsearch for indexing.
>
> Yehuda
>
> On Tue, Nov 28, 2017 at 2:22 PM, Sean Purdy <s.purdy at cv-library.co.uk> wrote:
>> Hi,
>>
>>
>> http://docs.ceph.com/docs/master/radosgw/s3/ says that S3 object notifications are not supported. I'd like something like object notifications so that we can backup new objects in realtime, instead of trawling the whole object list for what's changed.
>>
>> Is there anything similar I can use? I've found Spreadshirt's haproxy fork which traps requests and updates redis - https://github.com/spreadshirt/s3gw-haproxy[https://github.com/spreadshirt/s3gw-haproxy][https://github.com/spreadshirt/s3gw-haproxy[https://github.com/spreadshirt/s3gw-haproxy]] Anybody used that?
>>
>>
>> Thanks,
>>
>> Sean Purdy
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users at lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]]
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]]
>


More information about the ceph-users mailing list