[ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size
kushnirenko at gmail.com
Wed Oct 11 05:15:33 PDT 2017
You are absolutely right! Thanks!
The following sequence solves the problem:
int stripe_size = stripe_unit * stripe_count;
Now there is very little in the documentation about meaning of above
parameters. The only document I found is CERN IT group presentation (page
8). Perhaps it is obvious. Also it seems that optimizing these parameters
is meaningful in large scale Ceph installations only.
Now if I have a 6TB disks, then in default installation there would be
6TB/4MB = 1.5M objects per OSD. Does it create any performance hit?
On Tue, Oct 10, 2017 at 12:38 AM, Gregory Farnum <gfarnum at redhat.com> wrote:
> Well, just from a quick skim, libradosstriper.h has a function
> rados_striper_set_object_layout_object_size(rados_striper_t striper,
> unsigned int object_size)
> and libradosstriper.hpp has one in RadosStriper
> set_object_layout_object_size(unsigned int object_size);
> So I imagine you specify it with those the same way you've set the stripe
> unit and counts.
> On Sat, Oct 7, 2017 at 12:38 PM Alexander Kushnirenko <
> kushnirenko at gmail.com> wrote:
>> Hi, Gregory!
>> It turns out that this error is internal CEPH feature. I wrote standalone
>> program to create 132M object in striper mode. It works only for 4M
>> stripe. If you set stripe_unit = 2M it still creates 4M stripe_unit.
>> Anything bigger than 4M causes crash here
>> __u32 object_size = layout->object_size;
>> __u32 su = layout->stripe_unit;
>> __u32 stripe_count = layout->stripe_count;
>> assert(object_size >= su); <------------
>> I'm curious where it gets layout->object_size for object that is just
>> been created.
>> As I understod striper mode was created by CERN guys. In there document
>> they recommend 8M stripe_unit. But it does not work in luminous.
>> Created I/O context.
>> Connected to pool backup with rados_striper_create
>> Stripe unit OK 8388608
>> Stripe count OK 1
>> /build/ceph-12.2.0/src/osdc/Striper.cc: In function 'static void
>> Striper::file_to_extents(CephContext*, const char*, const
>> file_layout_t*, uint64_t, uint64_t, uint64_t, std::map<object_t,
>> std::vector<ObjectExtent> >&, uint64_t)' thread 7f13bd5c1e00 time
>> 2017-10-07 21:44:58.654778
>> /build/ceph-12.2.0/src/osdc/Striper.cc: 64: FAILED assert(object_size >=
>> ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x102) [0x7f13b3f3b332]
>> 2: (Striper::file_to_extents(CephContext*, char const*, file_layout_t
>> const*, unsigned long, unsigned long, unsigned long, std::map<object_t,
>> std::vector<ObjectExtent, std::allocator<ObjectExtent> >,
>> std::less<object_t>, std::allocator<std::pair<object_t const,
>> std::vector<ObjectExtent, std::allocator<ObjectExtent> > > > >&, unsigned
>> long)+0x1e1e) [0x7f13bce235ee]
>> 3: (Striper::file_to_extents(CephContext*, char const*, file_layout_t
>> const*, unsigned long, unsigned long, unsigned long,
>> std::vector<ObjectExtent, std::allocator<ObjectExtent> >&, unsigned
>> long)+0x51) [0x7f13bce23691]
>> 4: (libradosstriper::RadosStriperImpl::internal_
>> aio_write(std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > const&, boost::intrusive_ptr<libradosstriper::MultiAioCompletionImpl>,
>> ceph::buffer::list const&, unsigned long, unsigned long, ceph_file_layout
>> const&)+0x224) [0x7f13bcda4184]
>> 5: (libradosstriper::RadosStriperImpl::write_in_
>> open_object(std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > const&, ceph_file_layout const&,
>> std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > const&, ceph::buffer::list const&, unsigned long,
>> unsigned long)+0x13c) [0x7f13bcda476c]
>> 6: (libradosstriper::RadosStriperImpl::write(std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::list
>> const&, unsigned long, unsigned long)+0xd5) [0x7f13bcda4bd5]
>> 7: (rados_striper_write()+0xdb) [0x7f13bcd9ba0b]
>> 8: (()+0x10fb) [0x55dd87b410fb]
>> 9: (__libc_start_main()+0xf1) [0x7f13bc9d72b1]
>> 10: (()+0xbca) [0x55dd87b40bca]
>> On Fri, Sep 29, 2017 at 11:46 PM, Gregory Farnum <gfarnum at redhat.com>
>>> I haven't used the striper, but it appears to make you specify sizes,
>>> stripe units, and stripe counts. I would expect you need to make sure that
>>> the size is an integer multiple of the stripe unit. And it probably
>>> defaults to a 4MB object if you don't specify one?
>>> On Fri, Sep 29, 2017 at 2:09 AM Alexander Kushnirenko <
>>> kushnirenko at gmail.com> wrote:
>>>> I'm trying to use CEPH-12.2.0 as storage for with Bareos-16.2.4 backup
>>>> with libradosstriper1 support.
>>>> Libradosstriber was suggested on this list to solve the problem, that
>>>> current CEPH-12 discourages users from using object with very big size
>>>> (>128MB). Bareos treat Rados Object as Volume and in CEPH-10 it created
>>>> objects with very big size (10G and more). CEPH-10 allowed such behaviour,
>>>> put recovery indeed take very long time. So stripping objects seems to be
>>>> the right thing to do.
>>>> Bareos supports libradosstriper and the code seems to work. But for
>>>> some reason it run only with stripe_unit=4194304, which seems to be typical
>>>> value for RadosGW for example. I tried several other values for
>>>> stripe_unit, but the code exit with error.
>>>> Is there a particular reason why only 4M size works? Can one use some
>>>> CLI to test different stripe sizes?
>>>> Basic flow of creating object in Bareos is the following:
>>>> rados_ioctx_create(m_cluster, m_rados_poolname, &m_ctx);
>>>> rados_striper_create(m_ctx, &m_striper);
>>>> rados_striper_set_object_layout_stripe_unit(m_striper, m_stripe_unit);
>>>> status = rados_striper_write(m_striper, m_virtual_filename, buffer,
>>>> count, offset);
>>>> ceph-users mailing list
>>>> ceph-users at lists.ceph.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the ceph-users