[ceph-users] Huge latency spikes

Paul Emmerich paul.emmerich at croit.io
Sat Nov 17 12:14:44 PST 2018


Are you running FileStore? (The config options you are using looks
like a FileStore config)
Try out BlueStore, we've found that it reduces random latency spikes
due to filesystem weirdness a lot.



Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

Am Sa., 17. Nov. 2018 um 21:07 Uhr schrieb Alex Litvak
<alexander.v.litvak at gmail.com>:
>
> I am using libvirt for block device (openstack, proxmox KVM VMs)
> Also I am mounting cephfs inside of VMs and on bare metal hosts.  In
> this case it would be a kernel based client.
>
>  From what I can see based on pool stats cephfs pools have higher
> utilization comparing to block pools during the spikes, how ever it is
> still small.
>
> On 11/17/2018 1:40 PM, Kees Meijs wrote:
> > Hi Alex,
> >
> > What kind of clients do you use? Is it KVM (QEMU) using NBD driver,
> > kernel, or...?
> >
> > Regards,
> > Kees
> >
> > On 17-11-18 20:17, Alex Litvak wrote:
> >> Hello everyone,
> >>
> >> I am trying to troubleshoot cluster exhibiting huge spikes of latency.
> >> I cannot quite catch it because it happens during the light activity
> >> and randomly affects one osd node out of 3 in the pool.
> >>
> >> This is a file store.
> >> I see some osds exhibit applied latency  of 400 ms, 1 minute load
> >> average shuts to 60.  Client commit latency with queue shoots to 300ms
> >> and journal latency (return write ack for client) (journal on Intel
> >> DC-S3710 SSD) shoots on 40 ms
> >>
> >> op_w_process_latency showed 250 ms and client read-modify-write
> >> operation readable/applied latency jumped to 1.25 s on one of the OSDs
> >>
> >> I rescheduled the scrubbing and deep scrubbing and was watching ceph
> >> -w activity so it is definitely not related.
> >>
> >> At the same time node shows 98 % cpu idle no significant changes in
> >> memory utilization, no errors on network with bandwidth utilization
> >> between 20 - 50 Mbit on client and back end networks
> >>
> >> OSD node has 12 OSDs (2TB rust) 2 partitioned SSD journal disks, 32 GB
> >> RAM, dial 6 core / 12 thread CPUs
> >>
> >> This is perhaps the most relevant part of ceph config
> >>
> >> debug lockdep = 0/0
> >> debug context = 0/0
> >> debug crush = 0/0
> >> debug buffer = 0/0
> >> debug timer = 0/0
> >> debug journaler = 0/0
> >> debug osd = 0/0
> >> debug optracker = 0/0
> >> debug objclass = 0/0
> >> debug filestore = 0/0
> >> debug journal = 0/0
> >> debug ms = 0/0
> >> debug monc = 0/0
> >> debug tp = 0/0
> >> debug auth = 0/0
> >> debug finisher = 0/0
> >> debug heartbeatmap = 0/0
> >> debug perfcounter = 0/0
> >> debug asok = 0/0
> >> debug throttle = 0/0
> >>
> >> [osd]
> >>          journal_dio = true
> >>          journal_aio = true
> >>          osd_journal = /var/lib/ceph/osd/$cluster-$id-journal/journal
> >>          osd_journal_size = 2048     ; journal size, in megabytes
> >>      osd crush update on start = false
> >>          osd mount options xfs =
> >> "rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M"
> >>          osd_op_threads = 5
> >>          osd_disk_threads = 4
> >>          osd_pool_default_size = 2
> >>          osd_pool_default_min_size = 1
> >>          osd_pool_default_pg_num = 512
> >>          osd_pool_default_pgp_num = 512
> >>          osd_crush_chooseleaf_type = 1
> >>          ; osd pool_default_crush_rule = 1
> >>      ; new options 04.12.2015
> >>      filestore_op_threads = 4
> >>          osd_op_num_threads_per_shard = 1
> >>          osd_op_num_shards = 25
> >>          filestore_fd_cache_size = 64
> >>          filestore_fd_cache_shards = 32
> >>      filestore_fiemap = false
> >>      ; Reduce impact of scrub (needs cfq on osds)
> >>      osd_disk_thread_ioprio_class = "idle"
> >>      osd_disk_thread_ioprio_priority = 7
> >>      osd_deep_scrub_interval = 1211600
> >>          osd_scrub_begin_hour = 19
> >>          osd_scrub_end_hour = 4
> >>          osd_scrub_sleep = 0.1
> >> [client]
> >>      rbd_cache = true
> >>      rbd_cache_size = 67108864
> >>      rbd_cache_max_dirty = 50331648
> >>      rbd_cache_target_dirty = 33554432
> >>      rbd_cache_max_dirty_age = 2
> >>      rbd_cache_writethrough_until_flush = true
> >>
> >> OSD logs and system log at that time show nothing interesting.
> >>
> >> Any clue of what to look for in order to diagnose the load / latency
> >> spikes would be really appreciated.
> >>
> >> Thank you
> >>
> >> _______________________________________________
> >> ceph-users mailing list
> >> ceph-users at lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users at lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


More information about the ceph-users mailing list