[ceph-users] optimize bluestore for random write i/o

Stefan Priebe - Profihost AG s.priebe at profihost.ag
Wed Mar 6 05:59:56 PST 2019


Am 06.03.19 um 14:08 schrieb Mark Nelson:
> 
> On 3/6/19 5:12 AM, Stefan Priebe - Profihost AG wrote:
>> Hi Mark,
>> Am 05.03.19 um 23:12 schrieb Mark Nelson:
>>> Hi Stefan,
>>>
>>>
>>> Could you try running your random write workload against bluestore and
>>> then take a wallclock profile of an OSD using gdbpmp? It's available
>>> here:
>>>
>>>
>>> https://github.com/markhpc/gdbpmp
>> sure but it does not work:
>>
>>
>> # ./gdbpmp.py -p 3760442 -n 100 -o gdbpmp.data
>> Attaching to process 3760442...0x00007f917b6a615f in
>> pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>>
>> Thread 1 "ceph-osd" received signal SIGCONT, Continued.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> Done.
>> Gathering Samples
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 2 "log" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 2 "log" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 2 "log" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 2 "log" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 2 "log" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 2 "log" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
>> .
>> Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
>> 0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
>> target:/lib/x86_64-linux-gnu/libpthread.so.0
> 
> 
> Did you get the data file?  I think those are just warnings that pop up
> on some versions of gdb.

Yes there was a file but the osd was marked down while those "warnings"
happened. So it won't be useful.

Greets,
Stefan

> 
> 
>>
>> Greets,
>> Stefan
>>
>>
>>> Thanks,
>>>
>>> Mark
>>>
>>>
>>> On 3/5/19 2:29 AM, Stefan Priebe - Profihost AG wrote:
>>>> Hello list,
>>>>
>>>> while the performance of sequential writes 4k on bluestore is very high
>>>> and even higher than filestore i was wondering what i can do to
>>>> optimize
>>>> random pattern as well.
>>>>
>>>> While using:
>>>> fio --rw=write --iodepth=32 --ioengine=libaio --bs=4k --numjobs=4
>>>> --filename=/tmp/test --size=10G --runtime=60 --group_reporting
>>>> --name=test --direct=1
>>>>
>>>> I get 36000 iop/s on bluestore while having 11500 on filestore.
>>>>
>>>> Using randwrite gives me 17000 on filestore and only 9500 on bluestore.
>>>>
>>>> This is on all flash / ssd running luminous 12.2.10.
>>>>
>>>> Greets,
>>>> Stefan
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users at lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users at lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


More information about the ceph-users mailing list