[ceph-users] Huge latency spikes

Alex Litvak alexander.v.litvak at gmail.com
Sat Nov 17 20:19:55 PST 2018


I stand corrected, I looked at the device iostat, but it was partitioned.  Here is a more correct picture of what is going on now.

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
dm-14             0.00     0.00    0.00   19.00     0.00  4116.00   433.26     0.01    0.68    0.00    0.68   0.05   0.10
dm-15             0.00     0.00    0.00   35.00     0.00  8224.00   469.94     0.03    0.86    0.00    0.86   0.06   0.20
dm-16             0.00     0.00    0.00   53.00     0.00 12428.00   468.98     0.11    2.04    0.00    2.04   0.17   0.90
dm-17             0.00     0.00    0.00   43.00     0.00  8344.00   388.09     0.09    2.14    0.00    2.14   0.42   1.80
dm-18             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-19             0.00     0.00    0.00   75.00     0.00 16824.00   448.64     0.08    1.11    0.00    1.11   0.08   0.60
dm-20             0.00     0.00    0.00   70.00     0.00 16452.00   470.06     0.06    0.90    0.00    0.90   0.09   0.60
dm-21             0.00     0.00    0.00   18.00     0.00  4112.00   456.89     0.02    1.00    0.00    1.00   0.11   0.20
dm-22             0.00     0.00    0.00   53.00     0.00 12324.00   465.06     0.06    0.70    0.00    0.70   0.08   0.40
dm-24             0.00     0.00    0.00   18.00     0.00  4272.00   474.67     0.02    1.06    0.00    1.06   0.17   0.30
dm-25             0.00     0.00    0.00   74.00     0.00 16916.00   457.19     0.09    1.26    0.00    1.26   0.18   1.30

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
dm-14             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-15             0.00     0.00    0.00   17.00     0.00  4108.00   483.29     0.02    1.00    0.00    1.00   0.06   0.10
dm-16             0.00     0.00    0.00   34.00     0.00  8208.00   482.82     0.03    1.00    0.00    1.00   0.06   0.20
dm-17             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-18             0.00     0.00    0.00   36.00     0.00  8220.00   456.67     0.05    1.33    0.00    1.33   0.08   0.30
dm-19             0.00     0.00    0.00    1.00     0.00     8.00    16.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-20             0.00     0.00    0.00   36.00     0.00  8288.00   460.44     0.05    1.42    0.00    1.42   0.08   0.30
dm-21             0.00     0.00    0.00   34.00     0.00  8208.00   482.82     0.03    1.00    0.00    1.00   0.06   0.20
dm-22             0.00     0.00    0.00   18.00     0.00  4128.00   458.67     0.04    3.22    0.00    3.22   0.17   0.30
dm-24             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-25             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
dm-14             0.00     0.00    0.00   20.00     0.00  4032.00   403.20     0.00    0.00    0.00    0.00   0.00   0.00
dm-15             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-16             0.00     0.00    0.00    1.00     0.00    20.00    40.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-17             0.00     0.00    0.00    4.00     0.00    28.00    14.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-18             0.00     0.00    0.00    3.00     0.00    36.00    24.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-19             0.00     0.00    0.00    2.00     0.00    20.00    20.00     0.01    2.50    0.00    2.50   2.50   0.50
dm-20             0.00     0.00    0.00    6.00     0.00    96.00    32.00     0.02    3.33    0.00    3.33   2.00   1.20
dm-21             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-22             0.00     0.00    0.00    2.00     0.00    32.00    32.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-24             0.00     0.00    0.00   22.00     0.00  4184.00   380.36     0.10    4.59    0.00    4.59   0.95   2.10
dm-25             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
dm-14             0.00     0.00    0.00    8.00     0.00  1928.00   482.00     0.01    1.00    0.00    1.00   0.12   0.10
dm-15             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-16             0.00     0.00    0.00    3.00     0.00   312.00   208.00     0.00    0.33    0.00    0.33   0.33   0.10
dm-17             0.00     0.00    0.00   18.00     0.00  4264.00   473.78     0.03    1.67    0.00    1.67   0.11   0.20
dm-18             0.00     0.00    0.00   17.00     0.00  4104.00   482.82     0.03    1.82    0.00    1.82   0.12   0.20
dm-19             0.00     0.00    0.00   18.00     0.00  4112.00   456.89     0.02    1.06    0.00    1.06   0.11   0.20
dm-20             0.00     0.00    0.00   32.00     0.00  4308.00   269.25     0.03    0.81    0.00    0.81   0.34   1.10
dm-21             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-22             0.00     0.00    0.00    8.00     0.00   540.00   135.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-24             0.00     0.00    0.00   35.00     0.00  8228.00   470.17     0.03    0.97    0.00    0.97   0.06   0.20
dm-25             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00



On 11/17/2018 5:27 PM, John Petrini wrote:
> The iostat isn't very helpful because there are not many writes. I'd recommend disabling cstates entirely, not sure it's your problem but it's good practice and if your cluster goes as idle as your 
> iostat suggests it could be the culprit.
> 
> 
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 




More information about the ceph-users mailing list