[ceph-users] Huge latency spikes

Ashley Merrick singapore at amerrick.co.uk
Tue Nov 20 06:11:49 PST 2018


Me and quite a few others have had high random latency issues with disk
cache enabled.

,Ash

On Tue, 20 Nov 2018 at 9:09 PM, Alex Litvak <alexander.v.litvak at gmail.com>
wrote:

> John,
>
> If I go with write through, shouldn't disk cache be enabled?
>
> On 11/20/2018 6:12 AM, John Petrini wrote:
> > I would disable cache on the controller for your journals. Use write
> through and no read ahead. Did you make sure the disk cache is disabled?
> >
> > On Tuesday, November 20, 2018, Alex Litvak <alexander.v.litvak at gmail.com
> <mailto:alexander.v.litvak at gmail.com>> wrote:
> >  > I went through raid controller firmware update.  I replaced a pair
> of SSDs with new ones.  Nothing have changed.  Per controller card utility
> it shows that no patrol reading happens and battery
> > backup is in a good shape.  Cache policy is WriteBack.  I am aware on
> the bad battery effect but it doesn't seem to be the case unless controller
> is lying to me.
> >  >
> >  >
> >  > On 11/19/2018 2:39 PM, Brendan Moloney wrote:
> >  >>
> >  >> Hi,
> >  >>
> >  >>> Raid card for journal disks is Perc H730 (Megaraid), RAID 1,
> battery back cache is on
> >  >>>
> >  >>> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write
> Cache if Bad BBU
> >  >>> Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write
> Cache if Bad BBU
> >  >>>
> >  >>> I have  2 other nodes with older Perc H710 and similar SSDs with
> slightly higher wear (6.3% vs 5.18%) but from observation they hardly hit
> 1.5 ms on rear occasion
> >  >>> Cache, RAID, and battery situation is the same.
> >  >>
> >  >> I would take a closer look at the RAID card.  Are you sure the BBU
> is ok?  In the past I noticed the Megaraid cards would do periodic battery
> tests that would completely drain the battery and thus
> > disable the write cache until they reached some threshold of charge
> again.  They also can do periodic "patrol reads" and "consistency checks"
> that can hurt performance. Or the card could just be
> > failing, I have almost gone through more RAID cards than HDDs. The
> unreliability and black box nature of hardware RAID cards is one of the
> things that first got me looking into Ceph (although even
> > mdadm is a big improvement in my opinion).
> >  >>
> >  >> For journals you are better off putting half your OSDs on one SSD
> and half on the other instead of RAID1.
> >  >>
> >  >> -Brendan
> >  >>
> >  >
> >  >
> >  > _______________________________________________
> >  > ceph-users mailing list
> >  > ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com>
> >  > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >  >
> >
> > --
> >
> > John Petrini
> > Platforms Engineer
> >
> >
> > Call CoreDial         215.297.4400 x 232 <tel:215-297-4400>
> >
> > Call CoreDial         www.coredial.com <https://coredial.com/>
> >
> > CoreDial      751 Arbor Way, Hillcrest I, Suite 150 Blue Bell, PA 19422
> > <
> https://www.google.com/maps/place/CoreDial,+LLC/@40.140902,-75.2878857,17z/data=%213m1%214b1%214m5%213m4%211s0x89c6bc587f1cfd47:0x4c79d505f2ee580b%218m2%213d40.140902%214d-75.285697
> >
> >
> > The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or
> > other use of, or taking of any action in reliance upon, this information
> by persons or entities other than the intended recipient is prohibited. If
> you received this in error, please contact the
> > sender and delete the material from any computer.
> >
> >
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users at lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181120/8937f93f/attachment.html>


More information about the ceph-users mailing list