[ceph-users] optimize bluestore for random write i/o

Stefan Priebe - Profihost AG s.priebe at profihost.ag
Wed Mar 6 03:12:13 PST 2019


Hi Mark,
Am 05.03.19 um 23:12 schrieb Mark Nelson:
> Hi Stefan,
> 
> 
> Could you try running your random write workload against bluestore and
> then take a wallclock profile of an OSD using gdbpmp? It's available here:
> 
> 
> https://github.com/markhpc/gdbpmp

sure but it does not work:


# ./gdbpmp.py -p 3760442 -n 100 -o gdbpmp.data
Attaching to process 3760442...0x00007f917b6a615f in
pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0

Thread 1 "ceph-osd" received signal SIGCONT, Continued.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
Done.
Gathering Samples
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 2 "log" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 2 "log" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 2 "log" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 2 "log" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 2 "log" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 2 "log" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
.
Thread 1 "ceph-osd" received signal SIGINT, Interrupt.
0x00007f917b6a615f in pthread_cond_wait@@GLIBC_2.3.2 () from
target:/lib/x86_64-linux-gnu/libpthread.so.0

Greets,
Stefan


> Thanks,
> 
> Mark
> 
> 
> On 3/5/19 2:29 AM, Stefan Priebe - Profihost AG wrote:
>> Hello list,
>>
>> while the performance of sequential writes 4k on bluestore is very high
>> and even higher than filestore i was wondering what i can do to optimize
>> random pattern as well.
>>
>> While using:
>> fio --rw=write --iodepth=32 --ioengine=libaio --bs=4k --numjobs=4
>> --filename=/tmp/test --size=10G --runtime=60 --group_reporting
>> --name=test --direct=1
>>
>> I get 36000 iop/s on bluestore while having 11500 on filestore.
>>
>> Using randwrite gives me 17000 on filestore and only 9500 on bluestore.
>>
>> This is on all flash / ssd running luminous 12.2.10.
>>
>> Greets,
>> Stefan
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users at lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


More information about the ceph-users mailing list