[ceph-users] Huge latency spikes

Ashley Merrick singapore at amerrick.co.uk
Sun Nov 18 08:30:36 PST 2018


hdparm -W /dev/xxx should show you

On Mon, 19 Nov 2018 at 12:28 AM, Alex Litvak <alexander.v.litvak at gmail.com>
wrote:

> All machines state the same.
>
> /opt/MegaRAID/MegaCli/MegaCli64 -LDGetProp -DskCache -Lall -a0
>
> Adapter 0-VD 0(target id: 0): Disk Write Cache : Disk's Default
> Adapter 0-VD 1(target id: 1): Disk Write Cache : Disk's Default
>
> I assume they are all on which is actually bad based on common sense.
>
>
> https://notesbytom.wordpress.com/2016/10/21/dell-perc-megaraid-disk-cache-policy/
>
> An I couldn't find how to confirm it if it is true but vendor wouldn't
> ship drives with cache disabled.
>
> I am getting logs in the controller log which are not shown on other
> servers
>
> 11/18/18  8:21:55: C0:SysDma: localAddrPlb 50e03407a0, localAddr e03407a0
> 11/18/18  8:21:55: C0:SysDma: localAddrPlb 50e03409e0, localAddr e03409e0
> 11/18/18  8:21:55: C0:SysDma: localAddrPlb 50e0340c20, localAddr e0340c20
> 11/18/18  8:21:55: C0:SysDma: localAddrPlb 50e0340e60, localAddr e0340e60
> 11/18/18  8:21:55: C0:SysDma: localAddrPlb 50e03410a0, localAddr e03410a0
> 11/18/18  8:21:55: C0:SysDma: localAddrPlb 50e03401a0, localAddr e03401a0
>
> Not sure if it has any relation to the issue of latency but search
> returned nothing substantial.
>
> On 11/18/2018 8:52 AM, Serkan Çoban wrote:
> > I am not saying controller cache, you should check ssd disk caches.
> > On Sun, Nov 18, 2018 at 11:40 AM Alex Litvak
> > <alexander.v.litvak at gmail.com> wrote:
> >>
> >> All 3 nodes have this status for SSD mirror.  Controller cache is on
> for all 3.
> >>
> >> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache
> if Bad BBU
> >> Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache
> if Bad BBU
> >>
> >>
> >> On 11/18/2018 12:45 AM, Serkan Çoban wrote:
> >>> Does write cache on SSDs enabled on three servers? Can you check them?
> >>> On Sun, Nov 18, 2018 at 9:05 AM Alex Litvak
> >>> <alexander.v.litvak at gmail.com> wrote:
> >>>>
> >>>> Raid card for journal disks is Perc H730 (Megaraid), RAID 1, battery
> back cache is on
> >>>>
> >>>> Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache
> if Bad BBU
> >>>> Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache
> if Bad BBU
> >>>>
> >>>> I have  2 other nodes with older Perc H710 and similar SSDs with
> slightly higher wear (6.3% vs 5.18%) but from observation they hardly hit
> 1.5 ms on rear occasion
> >>>> Cache, RAID, and battery situation is the same.
> >>>>
> >>>> On 11/17/2018 11:38 PM, Serkan Çoban wrote:
> >>>>>> 10ms w_await for SSD is too much. How that SSD is connected to the
> system? Any raid card installed on this system? What is the raid mode?
> >>>>> On Sun, Nov 18, 2018 at 8:25 AM Alex Litvak
> >>>>> <alexander.v.litvak at gmail.com> wrote:
> >>>>>>
> >>>>>> Here is another snapshot.  I wonder if this write io wait is too big
> >>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>> dm-14             0.00     0.00    0.00   23.00     0.00   336.00
>   29.22     0.34   14.74    0.00   14.74   2.87   6.60
> >>>>>> dm-15             0.00     0.00    0.00   16.00     0.00   200.00
>   25.00     0.01    0.75    0.00    0.75   0.75   1.20
> >>>>>> dm-16             0.00     0.00    0.00   17.00     0.00   276.00
>   32.47     0.25   14.94    0.00   14.94   3.35   5.70
> >>>>>> dm-17             0.00     0.00    0.00   17.00     0.00   252.00
>   29.65     0.32   18.65    0.00   18.65   4.00   6.80
> >>>>>> dm-18             0.00     0.00    0.00   15.00     0.00   152.00
>   20.27     0.25   16.80    0.00   16.80   4.07   6.10
> >>>>>> dm-19             0.00     0.00    0.00   13.00     0.00   152.00
>   23.38     0.21   15.92    0.00   15.92   4.85   6.30
> >>>>>> dm-20             0.00     0.00    0.00   20.00     0.00   248.00
>   24.80     0.27   13.60    0.00   13.60   3.25   6.50
> >>>>>> dm-21             0.00     0.00    0.00   17.00     0.00   188.00
>   22.12     0.27   16.00    0.00   16.00   3.59   6.10
> >>>>>> dm-22             0.00     0.00    0.00   20.00     0.00   156.00
>   15.60     0.11    5.55    0.00    5.55   2.95   5.90
> >>>>>> dm-24             0.00     0.00    0.00    8.00     0.00    56.00
>   14.00     0.12   14.62    0.00   14.62   4.75   3.80
> >>>>>> dm-25             0.00     0.00    0.00   19.00     0.00   200.00
>   21.05     0.21   10.89    0.00   10.89   2.74   5.20
> >>>>>>
> >>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>> dm-14             0.00     0.00    0.00   11.00     0.00   136.00
>   24.73     0.11    9.73    0.00    9.73   1.82   2.00
> >>>>>> dm-15             0.00     0.00    0.00   12.00     0.00   136.00
>   22.67     0.04    3.75    0.00    3.75   1.08   1.30
> >>>>>> dm-16             0.00     0.00    0.00    9.00     0.00   104.00
>   23.11     0.09   10.44    0.00   10.44   2.44   2.20
> >>>>>> dm-17             0.00     0.00    0.00    5.00     0.00   160.00
>   64.00     0.02    4.00    0.00    4.00   4.00   2.00
> >>>>>> dm-18             0.00     0.00    0.00    5.00     0.00    52.00
>   20.80     0.03    5.80    0.00    5.80   3.60   1.80
> >>>>>> dm-19             0.00     0.00    0.00   10.00     0.00   104.00
>   20.80     0.08    7.90    0.00    7.90   2.10   2.10
> >>>>>> dm-20             0.00     0.00    0.00    9.00     0.00   132.00
>   29.33     0.10   11.22    0.00   11.22   2.56   2.30
> >>>>>> dm-21             0.00     0.00    0.00    6.00     0.00    68.00
>   22.67     0.07   12.33    0.00   12.33   3.83   2.30
> >>>>>> dm-22             0.00     0.00    0.00    3.00     0.00    20.00
>   13.33     0.01    3.67    0.00    3.67   3.67   1.10
> >>>>>> dm-24             0.00     0.00    0.00    4.00     0.00    24.00
>   12.00     0.07   18.00    0.00   18.00   5.25   2.10
> >>>>>> dm-25             0.00     0.00    0.00    6.00     0.00    64.00
>   21.33     0.06   10.33    0.00   10.33   3.67   2.20
> >>>>>>
> >>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>> dm-14             0.00     0.00    0.00    5.00     0.00   140.00
>   56.00     0.08   15.20    0.00   15.20   5.40   2.70
> >>>>>> dm-15             0.00     0.00    0.00    6.00     0.00   236.00
>   78.67     0.18   30.67    0.00   30.67   6.83   4.10
> >>>>>> dm-16             0.00     0.00    0.00    8.00     0.00    84.00
>   21.00     0.06    7.25    0.00    7.25   1.62   1.30
> >>>>>> dm-17             0.00     0.00    0.00    3.00     0.00    84.00
>   56.00     0.00    0.33    0.00    0.33   0.33   0.10
> >>>>>> dm-18             0.00     0.00    0.00    2.00     0.00    20.00
>   20.00     0.02   12.00    0.00   12.00  12.00   2.40
> >>>>>> dm-19             0.00     0.00    0.00   12.00     0.00    80.00
>   13.33     0.05    4.00    0.00    4.00   2.33   2.80
> >>>>>> dm-20             0.00     0.00    0.00   16.00     0.00   256.00
>   32.00     0.00    0.06    0.00    0.06   0.06   0.10
> >>>>>> dm-21             0.00     0.00    0.00    8.00     0.00   500.00
>  125.00     0.00    0.12    0.00    0.12   0.12   0.10
> >>>>>> dm-22             0.00     0.00    0.00    1.00     0.00     8.00
>   16.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>> dm-24             0.00     0.00    0.00    0.00     0.00     0.00
>    0.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>> dm-25             0.00     0.00    0.00    2.00     0.00    32.00
>   32.00     0.08   40.00    0.00   40.00  20.50   4.10
> >>>>>>
> >>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>> dm-14             0.00     0.00    0.00   10.00     0.00   108.00
>   21.60     0.11   10.80    0.00   10.80   1.90   1.90
> >>>>>> dm-15             0.00     0.00    0.00    5.00     0.00    60.00
>   24.00     0.03    6.20    0.00    6.20   3.40   1.70
> >>>>>> dm-16             0.00     0.00    0.00    6.00     0.00    68.00
>   22.67     0.00    0.17    0.00    0.17   0.17   0.10
> >>>>>> dm-17             0.00     0.00    0.00    7.00     0.00   100.00
>   28.57     0.09   12.29    0.00   12.29   2.86   2.00
> >>>>>> dm-18             0.00     0.00    0.00    5.00     0.00    44.00
>   17.60     0.07   14.00    0.00   14.00   4.00   2.00
> >>>>>> dm-19             0.00     0.00    0.00    2.00     0.00    24.00
>   24.00     0.03   14.00    0.00   14.00   8.50   1.70
> >>>>>> dm-20             0.00     0.00    0.00    7.00     0.00    96.00
>   27.43     0.07    9.57    0.00    9.57   2.00   1.40
> >>>>>> dm-21             0.00     0.00    0.00    7.00     0.00    84.00
>   24.00     0.12   16.57    0.00   16.57   4.86   3.40
> >>>>>> dm-22             0.00     0.00    0.00    5.00     0.00    60.00
>   24.00     0.05   10.20    0.00   10.20   3.40   1.70
> >>>>>> dm-24             0.00     0.00    0.00    2.00     0.00    16.00
>   16.00     0.03   17.00    0.00   17.00   9.00   1.80
> >>>>>> dm-25             0.00     0.00    0.00    4.00     0.00    76.00
>   38.00     0.05   11.75    0.00   11.75   4.75   1.90
> >>>>>>
> >>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>> dm-14             0.00     0.00    0.00    9.00     0.00   104.00
>   23.11     0.14   15.33    0.00   15.33   4.33   3.90
> >>>>>> dm-15             0.00     0.00    0.00   11.00     0.00   128.00
>   23.27     0.01    0.82    0.00    0.82   0.82   0.90
> >>>>>> dm-16             0.00     0.00    0.00   10.00     0.00   120.00
>   24.00     0.09    9.30    0.00    9.30   3.30   3.30
> >>>>>> dm-17             0.00     0.00    0.00    5.00     0.00    72.00
>   28.80     0.04    8.20    0.00    8.20   8.20   4.10
> >>>>>> dm-18             0.00     0.00    0.00    7.00     0.00    68.00
>   19.43     0.08   11.57    0.00   11.57   6.29   4.40
> >>>>>> dm-19             0.00     0.00    0.00    7.00     0.00    76.00
>   21.71     0.10   13.86    0.00   13.86   6.29   4.40
> >>>>>> dm-20             0.00     0.00    0.00   15.00     0.00   176.00
>   23.47     0.12    7.87    0.00    7.87   1.60   2.40
> >>>>>> dm-21             0.00     0.00    0.00    9.00     0.00   112.00
>   24.89     0.17   18.56    0.00   18.56   6.56   5.90
> >>>>>> dm-22             0.00     0.00    0.00    4.00     0.00    56.00
>   28.00     0.04   10.75    0.00   10.75   7.50   3.00
> >>>>>> dm-24             0.00     0.00    0.00    5.00     0.00   532.00
>  212.80     0.02    4.00    0.00    4.00   3.60   1.80
> >>>>>> dm-25             0.00     0.00    0.00    6.00     0.00    60.00
>   20.00     0.11   19.00    0.00   19.00   7.17   4.30
> >>>>>>
> >>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>> dm-14             0.00     0.00    0.00    9.00     0.00   112.00
>   24.89     0.30   33.22    0.00   33.22   4.33   3.90
> >>>>>> dm-15             0.00     0.00    0.00    8.00     0.00    96.00
>   24.00     0.08    9.88    0.00    9.88   2.62   2.10
> >>>>>> dm-16             0.00     0.00    0.00    5.00     0.00    92.00
>   36.80     0.01    1.80    0.00    1.80   1.80   0.90
> >>>>>> dm-17             0.00     0.00    0.00    4.00     0.00   104.00
>   52.00     0.04   10.00    0.00   10.00  10.00   4.00
> >>>>>> dm-18             0.00     0.00    0.00    9.00     0.00    92.00
>   20.44     0.11   12.00    0.00   12.00   4.44   4.00
> >>>>>> dm-19             0.00     0.00    0.00    7.00     0.00    64.00
>   18.29     0.10   14.43    0.00   14.43   4.00   2.80
> >>>>>> dm-20             0.00     0.00    0.00    7.00     0.00    84.00
>   24.00     0.10   14.00    0.00   14.00   3.71   2.60
> >>>>>> dm-21             0.00     0.00    0.00    2.00     0.00    16.00
>   16.00     0.08   39.50    0.00   39.50  20.50   4.10
> >>>>>> dm-22             0.00     0.00    0.00    5.00     0.00    36.00
>   14.40     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>> dm-24             0.00     0.00    0.00    5.00     0.00    52.00
>   20.80     0.09   17.20    0.00   17.20   5.40   2.70
> >>>>>> dm-25             0.00     0.00    0.00    6.00     0.00    48.00
>   16.00     0.10   16.00    0.00   16.00   5.50   3.30
> >>>>>>
> >>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>> dm-14             0.00     0.00    0.00    5.00     0.00    48.00
>   19.20     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>> dm-15             0.00     0.00    0.00    5.00     0.00    44.00
>   17.60     0.01    3.00    0.00    3.00   3.00   1.50
> >>>>>> dm-16             0.00     0.00    0.00    3.00     0.00    32.00
>   21.33     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>> dm-17             0.00     0.00    0.00    4.00     0.00    40.00
>   20.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>> dm-18             0.00     0.00    0.00    3.00     0.00    32.00
>   21.33     0.04   13.00    0.00   13.00   6.67   2.00
> >>>>>> dm-19             0.00     0.00    0.00    3.00     0.00    28.00
>   18.67     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>> dm-20             0.00     0.00    0.00    6.00     0.00    64.00
>   21.33     0.11   17.67    0.00   17.67   3.17   1.90
> >>>>>> dm-21             0.00     0.00    0.00    1.00     0.00    12.00
>   24.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>> dm-22             0.00     0.00    0.00    1.00     0.00    12.00
>   24.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>> dm-24             0.00     0.00    0.00    3.00     0.00    20.00
>   13.33     0.00    0.33    0.00    0.33   0.33   0.10
> >>>>>> dm-25             0.00     0.00    0.00    6.00     0.00    52.00
>   17.33     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>
> >>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>> dm-14             0.00     0.00    0.00   18.00     0.00   236.00
>   26.22     0.42   23.28    0.00   23.28   3.72   6.70
> >>>>>> dm-15             0.00     0.00    0.00   15.00     0.00   240.00
>   32.00     0.12    8.27    0.00    8.27   2.00   3.00
> >>>>>> dm-16             0.00     0.00    0.00   10.00     0.00   120.00
>   24.00     0.23   22.80    0.00   22.80   6.50   6.50
> >>>>>> dm-17             0.00     0.00    0.00    6.00     0.00    68.00
>   22.67     0.06    9.83    0.00    9.83   5.00   3.00
> >>>>>> dm-18             0.00     0.00    0.00   12.00     0.00   152.00
>   25.33     0.10    8.17    0.00    8.17   3.17   3.80
> >>>>>> dm-19             0.00     0.00    0.00   12.00     0.00   136.00
>   22.67     0.14   11.58    0.00   11.58   3.42   4.10
> >>>>>> dm-20             0.00     0.00    0.00   19.00     0.00   296.00
>   31.16     0.20   10.63    0.00   10.63   1.84   3.50
> >>>>>> dm-21             0.00     0.00    0.00   14.00     0.00   132.00
>   18.86     0.17   12.14    0.00   12.14   3.71   5.20
> >>>>>> dm-22             0.00     0.00    0.00   10.00     0.00    84.00
>   16.80     0.12   12.00    0.00   12.00   6.60   6.60
> >>>>>> dm-24             0.00     0.00    0.00    5.00     0.00    40.00
>   16.00     0.10   19.40    0.00   19.40   9.80   4.90
> >>>>>> dm-25             0.00     0.00    0.00    9.00     0.00   152.00
>   33.78     0.08    9.33    0.00    9.33   3.44   3.10
> >>>>>>
> >>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>> dm-14             0.00     0.00    0.00    8.00     0.00   112.00
>   28.00     0.08    9.75    0.00    9.75   3.88   3.10
> >>>>>> dm-15             0.00     0.00    0.00   14.00     0.00   156.00
>   22.29     0.13    9.43    0.00    9.43   2.71   3.80
> >>>>>> dm-16             0.00     0.00    0.00   12.00     0.00   132.00
>   22.00     0.11    8.83    0.00    8.83   2.17   2.60
> >>>>>> dm-17             0.00     0.00    0.00    8.00     0.00   116.00
>   29.00     0.08   10.12    0.00   10.12   4.25   3.40
> >>>>>> dm-18             0.00     0.00    0.00    8.00     0.00   100.00
>   25.00     0.08    9.62    0.00    9.62   5.00   4.00
> >>>>>> dm-19             0.00     0.00    0.00   20.00     0.00   212.00
>   21.20     0.25   12.75    0.00   12.75   2.60   5.20
> >>>>>> dm-20             0.00     0.00    0.00   16.00     0.00   168.00
>   21.00     0.04    2.69    0.00    2.69   0.81   1.30
> >>>>>> dm-21             0.00     0.00    0.00    5.00     0.00    48.00
>   19.20     0.05   10.80    0.00   10.80   7.80   3.90
> >>>>>> dm-22             0.00     0.00    0.00    5.00     0.00    52.00
>   20.80     0.03    6.80    0.00    6.80   3.40   1.70
> >>>>>> dm-24             0.00     0.00    0.00    8.00     0.00    60.00
>   15.00     0.09   11.75    0.00   11.75   2.25   1.80
> >>>>>> dm-25             0.00     0.00    0.00   12.00     0.00   160.00
>   26.67     0.06    5.08    0.00    5.08   1.25   1.50
> >>>>>>
> >>>>>> On 11/17/2018 10:19 PM, Alex Litvak wrote:
> >>>>>>> I stand corrected, I looked at the device iostat, but it was
> partitioned.  Here is a more correct picture of what is going on now.
> >>>>>>>
> >>>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>>> dm-14             0.00     0.00    0.00   19.00     0.00  4116.00
>  433.26     0.01    0.68    0.00    0.68   0.05   0.10
> >>>>>>> dm-15             0.00     0.00    0.00   35.00     0.00  8224.00
>  469.94     0.03    0.86    0.00    0.86   0.06   0.20
> >>>>>>> dm-16             0.00     0.00    0.00   53.00     0.00 12428.00
>  468.98     0.11    2.04    0.00    2.04   0.17   0.90
> >>>>>>> dm-17             0.00     0.00    0.00   43.00     0.00  8344.00
>  388.09     0.09    2.14    0.00    2.14   0.42   1.80
> >>>>>>> dm-18             0.00     0.00    0.00    0.00     0.00     0.00
>    0.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-19             0.00     0.00    0.00   75.00     0.00 16824.00
>  448.64     0.08    1.11    0.00    1.11   0.08   0.60
> >>>>>>> dm-20             0.00     0.00    0.00   70.00     0.00 16452.00
>  470.06     0.06    0.90    0.00    0.90   0.09   0.60
> >>>>>>> dm-21             0.00     0.00    0.00   18.00     0.00  4112.00
>  456.89     0.02    1.00    0.00    1.00   0.11   0.20
> >>>>>>> dm-22             0.00     0.00    0.00   53.00     0.00 12324.00
>  465.06     0.06    0.70    0.00    0.70   0.08   0.40
> >>>>>>> dm-24             0.00     0.00    0.00   18.00     0.00  4272.00
>  474.67     0.02    1.06    0.00    1.06   0.17   0.30
> >>>>>>> dm-25             0.00     0.00    0.00   74.00     0.00 16916.00
>  457.19     0.09    1.26    0.00    1.26   0.18   1.30
> >>>>>>>
> >>>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>>> dm-14             0.00     0.00    0.00    0.00     0.00     0.00
>    0.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-15             0.00     0.00    0.00   17.00     0.00  4108.00
>  483.29     0.02    1.00    0.00    1.00   0.06   0.10
> >>>>>>> dm-16             0.00     0.00    0.00   34.00     0.00  8208.00
>  482.82     0.03    1.00    0.00    1.00   0.06   0.20
> >>>>>>> dm-17             0.00     0.00    0.00    0.00     0.00     0.00
>    0.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-18             0.00     0.00    0.00   36.00     0.00  8220.00
>  456.67     0.05    1.33    0.00    1.33   0.08   0.30
> >>>>>>> dm-19             0.00     0.00    0.00    1.00     0.00     8.00
>   16.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-20             0.00     0.00    0.00   36.00     0.00  8288.00
>  460.44     0.05    1.42    0.00    1.42   0.08   0.30
> >>>>>>> dm-21             0.00     0.00    0.00   34.00     0.00  8208.00
>  482.82     0.03    1.00    0.00    1.00   0.06   0.20
> >>>>>>> dm-22             0.00     0.00    0.00   18.00     0.00  4128.00
>  458.67     0.04    3.22    0.00    3.22   0.17   0.30
> >>>>>>> dm-24             0.00     0.00    0.00    0.00     0.00     0.00
>    0.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-25             0.00     0.00    0.00    0.00     0.00     0.00
>    0.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>>
> >>>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>>> dm-14             0.00     0.00    0.00   20.00     0.00  4032.00
>  403.20     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-15             0.00     0.00    0.00    0.00     0.00     0.00
>    0.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-16             0.00     0.00    0.00    1.00     0.00    20.00
>   40.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-17             0.00     0.00    0.00    4.00     0.00    28.00
>   14.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-18             0.00     0.00    0.00    3.00     0.00    36.00
>   24.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-19             0.00     0.00    0.00    2.00     0.00    20.00
>   20.00     0.01    2.50    0.00    2.50   2.50   0.50
> >>>>>>> dm-20             0.00     0.00    0.00    6.00     0.00    96.00
>   32.00     0.02    3.33    0.00    3.33   2.00   1.20
> >>>>>>> dm-21             0.00     0.00    0.00    0.00     0.00     0.00
>    0.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-22             0.00     0.00    0.00    2.00     0.00    32.00
>   32.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-24             0.00     0.00    0.00   22.00     0.00  4184.00
>  380.36     0.10    4.59    0.00    4.59   0.95   2.10
> >>>>>>> dm-25             0.00     0.00    0.00    0.00     0.00     0.00
>    0.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>>
> >>>>>>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> >>>>>>> dm-14             0.00     0.00    0.00    8.00     0.00  1928.00
>  482.00     0.01    1.00    0.00    1.00   0.12   0.10
> >>>>>>> dm-15             0.00     0.00    0.00    0.00     0.00     0.00
>    0.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-16             0.00     0.00    0.00    3.00     0.00   312.00
>  208.00     0.00    0.33    0.00    0.33   0.33   0.10
> >>>>>>> dm-17             0.00     0.00    0.00   18.00     0.00  4264.00
>  473.78     0.03    1.67    0.00    1.67   0.11   0.20
> >>>>>>> dm-18             0.00     0.00    0.00   17.00     0.00  4104.00
>  482.82     0.03    1.82    0.00    1.82   0.12   0.20
> >>>>>>> dm-19             0.00     0.00    0.00   18.00     0.00  4112.00
>  456.89     0.02    1.06    0.00    1.06   0.11   0.20
> >>>>>>> dm-20             0.00     0.00    0.00   32.00     0.00  4308.00
>  269.25     0.03    0.81    0.00    0.81   0.34   1.10
> >>>>>>> dm-21             0.00     0.00    0.00    0.00     0.00     0.00
>    0.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-22             0.00     0.00    0.00    8.00     0.00   540.00
>  135.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>> dm-24             0.00     0.00    0.00   35.00     0.00  8228.00
>  470.17     0.03    0.97    0.00    0.97   0.06   0.20
> >>>>>>> dm-25             0.00     0.00    0.00    0.00     0.00     0.00
>    0.00     0.00    0.00    0.00    0.00   0.00   0.00
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 11/17/2018 5:27 PM, John Petrini wrote:
> >>>>>>>> The iostat isn't very helpful because there are not many writes.
> I'd recommend disabling cstates entirely, not sure it's your problem but
> it's good practice and if your cluster goes as idle as your
> >>>>>>>> iostat suggests it could be the culprit.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> ceph-users mailing list
> >>>>>>>> ceph-users at lists.ceph.com
> >>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> ceph-users mailing list
> >>>>>> ceph-users at lists.ceph.com
> >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> ceph-users mailing list
> >>>> ceph-users at lists.ceph.com
> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>> _______________________________________________
> >>> ceph-users mailing list
> >>> ceph-users at lists.ceph.com
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>
> >>
> >>
> >> _______________________________________________
> >> ceph-users mailing list
> >> ceph-users at lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users at lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181119/89103de6/attachment.html>


More information about the ceph-users mailing list