[ceph-users] ceph.conf tuning ... please comment

Piotr Dałek piotr.dalek at corp.ovh.com
Wed Dec 6 04:17:17 PST 2017


On 17-12-06 07:01 AM, Stefan Kooman wrote:
> [osd]
> # http://docs.ceph.com/docs/master/rados/configuration/mon-osd-interaction/
> osd crush update on start = false
> osd heartbeat interval = 1             # default 6
> osd mon heartbeat interval = 10        # default 30
> osd mon report interval min = 1        # default 5
> osd mon report interval max = 15       # default 120
> 
> The osd would almost immediately see a "cut off" to their partner OSD's
> in the placement group. By default they wait 6 seconds before sending
> their report to the monitors. During our analysis this is exactly the
> time the monitors were keeping an election. By tuning all of the above
> we could get them to send their reports faster, and by the time the
> election process was finished the monitors would handle the reports from
> the OSDs and come to the conclusion that a DC is down, flag it down
> and allow for normal client IO again.
> 
> Of course, stability and data safety is most important to us. So if any
> of these settings make you worry please let us know.

Heartbeats, especially in Luminous, are quite heavy bandwidth-wise if you 
have a lot of OSDs in clusters. You may want to keep osd heartbeat interval 
at 3 lowest, or if that's not acceptable then at least set "osd heartbeat 
min size" to 0.

-- 
Piotr Dałek
piotr.dalek at corp.ovh.com
https://www.ovh.com/us/


More information about the ceph-users mailing list