[ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

Alexander Kushnirenko kushnirenko at gmail.com
Wed Oct 11 05:15:33 PDT 2017


Hi, Gregory!

You are absolutely right! Thanks!

The following sequence solves the problem:
rados_striper_set_object_layout_stripe_unit(m_striper, stripe_unit);
rados_striper_set_object_layout_stripe_count(m_striper, stripe_count);
int stripe_size = stripe_unit * stripe_count;
rados_striper_set_object_layout_object_size(m_striper, stripe_size);

Now there is very little in the documentation about meaning of above
parameters.  The only document I found is CERN IT group presentation (page
8).  Perhaps it is obvious.  Also it seems that optimizing these parameters
is meaningful in large scale Ceph installations only.

https://indico.cern.ch/event/531810/contributions/2298934/
attachments/1358128/2053937/Ceph-Experience-at-RAL-final.pdf

Now if I have a 6TB disks, then in default installation there would be
6TB/4MB = 1.5M objects per OSD.  Does it create any performance hit?

Thank you,
Alexander

On Tue, Oct 10, 2017 at 12:38 AM, Gregory Farnum <gfarnum at redhat.com> wrote:

> Well, just from a quick skim, libradosstriper.h has a function
>     rados_striper_set_object_layout_object_size(rados_striper_t striper,
> unsigned int object_size)
> and libradosstriper.hpp has one in RadosStriper
>     set_object_layout_object_size(unsigned int object_size);
>
> So I imagine you specify it with those the same way you've set the stripe
> unit and counts.
>
> On Sat, Oct 7, 2017 at 12:38 PM Alexander Kushnirenko <
> kushnirenko at gmail.com> wrote:
>
>> Hi, Gregory!
>>
>> It turns out that this error is internal CEPH feature. I wrote standalone
>> program to create 132M object in striper mode. It works only for 4M
>> stripe.  If you set stripe_unit = 2M it still creates 4M stripe_unit.
>> Anything bigger than 4M causes crash here
>> <https://github.com/ceph/ceph/blob/master/src/osdc/Striper.cc#L64%5C>:
>>
>>
>> __u32 object_size = layout->object_size;
>>   __u32 su = layout->stripe_unit;
>>   __u32 stripe_count = layout->stripe_count;
>>   assert(object_size >= su);   <------------
>>
>> I'm curious where it gets layout->object_size for object that is just
>> been created.
>>
>> As I understod striper mode was created by CERN guys.  In there document
>> <https://indico.cern.ch/event/542464/contributions/2202295/attachments/1289543/1919853/HEPUsageOfCeph.pdf>
>> they recommend 8M stripe_unit.  But it does not work in luminous.
>>
>> Created I/O context.
>> Connected to pool backup with rados_striper_create
>> Stripe unit OK 8388608
>> Stripe count OK 1
>> /build/ceph-12.2.0/src/osdc/Striper.cc: In function 'static void
>> Striper::file_to_extents(CephContext*, const char*, const
>> file_layout_t*, uint64_t, uint64_t, uint64_t, std::map<object_t,
>> std::vector<ObjectExtent> >&, uint64_t)' thread 7f13bd5c1e00 time
>> 2017-10-07 21:44:58.654778
>> /build/ceph-12.2.0/src/osdc/Striper.cc: 64: FAILED assert(object_size >=
>> su)
>>  ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous
>> (rc)
>>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x102) [0x7f13b3f3b332]
>>  2: (Striper::file_to_extents(CephContext*, char const*, file_layout_t
>> const*, unsigned long, unsigned long, unsigned long, std::map<object_t,
>> std::vector<ObjectExtent, std::allocator<ObjectExtent> >,
>> std::less<object_t>, std::allocator<std::pair<object_t const,
>> std::vector<ObjectExtent, std::allocator<ObjectExtent> > > > >&, unsigned
>> long)+0x1e1e) [0x7f13bce235ee]
>>  3: (Striper::file_to_extents(CephContext*, char const*, file_layout_t
>> const*, unsigned long, unsigned long, unsigned long,
>> std::vector<ObjectExtent, std::allocator<ObjectExtent> >&, unsigned
>> long)+0x51) [0x7f13bce23691]
>>  4: (libradosstriper::RadosStriperImpl::internal_
>> aio_write(std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > const&, boost::intrusive_ptr<libradosstriper::MultiAioCompletionImpl>,
>> ceph::buffer::list const&, unsigned long, unsigned long, ceph_file_layout
>> const&)+0x224) [0x7f13bcda4184]
>>  5: (libradosstriper::RadosStriperImpl::write_in_
>> open_object(std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > const&, ceph_file_layout const&,
>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > const&, ceph::buffer::list const&, unsigned long,
>> unsigned long)+0x13c) [0x7f13bcda476c]
>>  6: (libradosstriper::RadosStriperImpl::write(std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::list
>> const&, unsigned long, unsigned long)+0xd5) [0x7f13bcda4bd5]
>>  7: (rados_striper_write()+0xdb) [0x7f13bcd9ba0b]
>>  8: (()+0x10fb) [0x55dd87b410fb]
>>  9: (__libc_start_main()+0xf1) [0x7f13bc9d72b1]
>>  10: (()+0xbca) [0x55dd87b40bca]
>>
>>
>> On Fri, Sep 29, 2017 at 11:46 PM, Gregory Farnum <gfarnum at redhat.com>
>> wrote:
>>
>>> I haven't used the striper, but it appears to make you specify sizes,
>>> stripe units, and stripe counts. I would expect you need to make sure that
>>> the size is an integer multiple of the stripe unit. And it probably
>>> defaults to a 4MB object if you don't specify one?
>>>
>>> On Fri, Sep 29, 2017 at 2:09 AM Alexander Kushnirenko <
>>> kushnirenko at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm trying to use CEPH-12.2.0 as storage for with Bareos-16.2.4 backup
>>>> with libradosstriper1 support.
>>>>
>>>> Libradosstriber was suggested on this list to solve the problem, that
>>>> current CEPH-12 discourages users from using object with very big size
>>>> (>128MB).  Bareos treat Rados Object as Volume and in CEPH-10 it created
>>>> objects with very big size (10G and more).  CEPH-10 allowed such behaviour,
>>>> put recovery indeed take very long time. So stripping objects seems to be
>>>> the right thing to do.
>>>>
>>>> Bareos supports libradosstriper and the code seems to work. But for
>>>> some reason it run only with stripe_unit=4194304, which seems to be typical
>>>> value for RadosGW for example.  I tried several other values for
>>>> stripe_unit, but the code exit with error.
>>>>
>>>> Is there a particular reason why only 4M size works?  Can one use some
>>>> CLI to test different stripe sizes?
>>>>
>>>> Basic flow of creating object in Bareos is the following:
>>>> rados_ioctx_create(m_cluster, m_rados_poolname, &m_ctx);
>>>> rados_striper_create(m_ctx, &m_striper);
>>>> rados_striper_set_object_layout_stripe_unit(m_striper, m_stripe_unit);
>>>> rados_striper_set_object_layout_stripe_count(m_striper,
>>>> m_stripe_count);
>>>> .....
>>>> status = rados_striper_write(m_striper, m_virtual_filename, buffer,
>>>> count, offset);
>>>>
>>>> Alexander
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users at lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171011/febf4856/attachment.html>


More information about the ceph-users mailing list