[ceph-users] Bad performance when two fio write to the same image

Mark Nelson mnelson at redhat.com
Thu Aug 4 14:18:48 PDT 2016


If you search through the archives, there's been a couple of other 
people that have run into this as well with Jewel.  With the librbd 
engine, you are much better using iodepth and/or multiple fio processes 
vs numjobs.  Even pre-jewel, there were gotchas that might not be 
immediately apparent.  If you for instance increase numjobs and do 
sequential reads, after the first job reads some data, it gets cached on 
the OSD, and then all subsequent jobs will re-read the same cached data 
unless you explicitly change the offsets.

IE it was probably never a good idea to use numjobs, but now it's really 
apparent that it's not a good idea. :)

Mark

On 08/04/2016 03:48 PM, Warren Wang - ISD wrote:
> Wow, thanks. I think that¹s the tidbit of info I needed to explain why
> increasing numjobs doesn¹t (anymore) scale performance as expected.
>
> Warren Wang
>
>
>
> On 8/4/16, 7:49 AM, "ceph-users on behalf of Jason Dillaman"
> <ceph-users-bounces at lists.ceph.com on behalf of jdillama at redhat.com> wrote:
>
>> With exclusive-lock, only a single client can have write access to the
>> image at a time. Therefore, if you are using multiple fio processes
>> against the same image, they will be passing the lock back and forth
>> between each other and you can expect bad performance.
>>
>> If you have a use-case where you really need to share the same image
>> between multiple concurrent clients, you will need to disable the
>> exclusive-lock feature (this can be done with the RBD cli on existing
>> images or by passing "--image-shared" when creating new images).
>>
>> On Thu, Aug 4, 2016 at 5:52 AM, Alexandre DERUMIER <aderumier at odiso.com>
>> wrote:
>>> Hi,
>>>
>>> I think this is because of exclusive-lock feature enabled by default
>>> since jessie on rbd image
>>>
>>>
>>> ----- Mail original -----
>>> De: "Zhiyuan Wang" <zhiyuan.wang at Istuary.com>
>>> À: "ceph-users" <ceph-users at lists.ceph.com>
>>> Envoyé: Jeudi 4 Août 2016 11:37:04
>>> Objet: [ceph-users] Bad performance when two fio write to the same image
>>>
>>>
>>>
>>> Hi Guys
>>>
>>> I am testing the performance of Jewel (10.2.2) with FIO, but found the
>>> performance would drop dramatically when two process write to the same
>>> image.
>>>
>>> My environment:
>>>
>>> 1. Server:
>>>
>>> One mon and four OSDs running on the same server.
>>>
>>> Intel P3700 400GB SSD which have 4 partitions, and each for one osd
>>> journal (journal size is 10GB)
>>>
>>> Inter P3700 400GB SSD which have 4 partitions, and each format to XFS
>>> for one osd data (each data is 90GB)
>>>
>>> 10GB network
>>>
>>> CPU: Intel(R) Xeon(R) CPU E5-2660 (it is not the bottleneck)
>>>
>>> Memory: 256GB (it is not the bottleneck)
>>>
>>> 2. Client
>>>
>>> 10GB network
>>>
>>> CPU: Intel(R) Xeon(R) CPU E5-2660 (it is not the bottleneck)
>>>
>>> Memory: 256GB (it is not the bottleneck)
>>>
>>> 3. Ceph
>>>
>>> Default configuration expect use async messager (have tried simple
>>> messager, got nearly the same result)
>>>
>>> 10GB image with 256 pg num
>>>
>>> Test Case
>>>
>>> 1. One Fio process: bs 4KB; iodepth 256; direct 1; ioengine rbd;
>>> randwrite
>>>
>>> The performance is nearly 60MB/s and IOPS is nearly 15K
>>>
>>> Four osd are nearly the same busy
>>>
>>> 2. Two Fio process: bs 4KB; iodepth 256; direct 1; ioengine rbd;
>>> randwrite (write to the same image)
>>>
>>> The performance is nearly 4MB/s each, and IOPS is nearly 1.5K each
>>> Terrible
>>>
>>> And I found that only one osd is busy, the other three are much more
>>> idle on CPU
>>>
>>> And I also run FIO on two clients, the same result
>>>
>>> 3. Two Fio process: bs 4KB; iodepth 256; direct 1; ioengine rbd
>>> randwrite (one to image1, one to image2)
>>>
>>> The performance is nearly 35MB/s each and IOPS is nearly 8.5K each
>>> Reasonable
>>>
>>> Four osd are nearly the same busy
>>>
>>>
>>>
>>>
>>>
>>> Could someone help to explain the reason of TEST 2
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>> Email Disclaimer & Confidentiality Notice
>>>
>>> This message is confidential and intended solely for the use of the
>>> recipient to whom they are addressed. If you are not the intended
>>> recipient you should not deliver, distribute or copy this e-mail. Please
>>> notify the sender immediately by e-mail and delete this e-mail from your
>>> system. Copyright © 2016 by Istuary Innovation Labs, Inc. All rights
>>> reserved.
>>>
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users at lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users at lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> --
>> Jason
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users at lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> This email and any files transmitted with it are confidential and intended solely for the individual or entity to whom they are addressed. If you have received this email in error destroy it immediately. *** Walmart Confidential ***
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>


More information about the ceph-users mailing list