[ceph-users] ceph.conf tuning ... please comment

Van Leeuwen, Robert rovanleeuwen at ebay.com
Wed Dec 6 04:23:27 PST 2017


Lets start with a disclaimer: Not an expert on any of these ceph tuning settings :)

However, in general with cluster intervals/timings:
You are trading quick failovers detection for:
1) Processing power: 
You might starve yourself of resources when expanding the cluster.
If you multiply all the changes you might actually create a lot of network chatter and need a lot of extra processing power.
With this in mind I would be very careful changing any setting by an order of magnitude unless you exactly know the impact.
2) You might create a very “nervous” cluster. E.g. a short network hiccup: OSDs marked out.
This creates a large amount of data shuffling which could flood a network link which could create more short network hiccups which would create flapping OSDs etc.

IMHO would change the mindset on tuning from: 
- what is the fastest possible failure detection time of a broken datacenter
- In case of catastrophic DC failure (which happens maybe once every few years) are the defaults that bad that you must change from what is the widely-tested deployment?

Off course it might be that the defaults are just a random number a dev put in and this is exactly what should be done in each deployment ;)
I am sure some other people have better insights in these specific settings.

Robert van Leeuwen

On 12/6/17, 7:01 AM, "ceph-users on behalf of Stefan Kooman" <ceph-users-bounces at lists.ceph.com on behalf of stefan at bit.nl> wrote:

    Dear list,
    In a ceph blog post about the new Luminous release there is a paragraph
    on the need for ceph tuning [1]:
    "If you are a Ceph power user and believe there is some setting that you
    need to change for your environment to get the best performance, please
    tell uswed like to either adjust our defaults so that your change isnt
    necessary or have a go at convincing you that you shouldnt be tuning
    that option."
    We have been tuning several ceph.conf parameters in order to allow for
    "fast failure" when an entire datacenter goes offline. We now have
    continued operation (no pending IO) after ~ 7 seconds. We have changed
    the following parameters:
    # https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs.ceph.com%2Fdocs%2Fmaster%2Frados%2Fconfiguration%2Fmon-osd-interaction%2F&data=02%7C01%7Crovanleeuwen%40ebay.com%7C0fba2ce3537b4bf4de9308d53c6f066d%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636481369899935585&sdata=arg0XlbGTI6w3Kr2Rf0KRPa5U3VJS5pyDGpgA8NzX%2FA%3D&reserved=0
    osd heartbeat grace = 4                  # default 6
    # Do _NOT_  scale based on laggy estimations
    mon osd adjust heartbeat grace = false
    ^^ without this setting it could take up to two minutes before ceph
    flagged a whole datacenter down (after we cut connectivity to the DC).
    Not sure how the estimation is done, but not good enough for us.
    # https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs.ceph.com%2Fdocs%2Fmaster%2Frados%2Fconfiguration%2Fmon-config-ref%2F&data=02%7C01%7Crovanleeuwen%40ebay.com%7C0fba2ce3537b4bf4de9308d53c6f066d%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636481369899935585&sdata=tYkO4M5iiK38u%2By0ZMBdXEROH2OtfoQM2iPSnAHc12k%3D&reserved=0
    # TUNING #
    mon lease = 1.0                        # default 5
    mon election timeout = 2               # default 5 
    mon lease renew interval factor = 0.4  # default 0.6
    mon lease ack timeout factor = 1.5     # default 2.0
    mon timecheck interval = 60            # default 300
    Above checks are there to make the whole process faster. After a DC
    failure the monitors will need a re-election (depending on what DC and
    who was a leader and who were peon). While going through mon
    debug logging we have observed that this whole process is really fast
    (things happen to be done in milliseconds). We have a quite low latency
    network, so I guess we can cut some slack here. Ceph won't make any
    decisions while there is no consensus, so better get that consensus as
    soon as possible.
    # https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs.ceph.com%2Fdocs%2Fmaster%2Frados%2Fconfiguration%2Fmon-osd-interaction%2F%23monitor-settings&data=02%7C01%7Crovanleeuwen%40ebay.com%7C0fba2ce3537b4bf4de9308d53c6f066d%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636481369899935585&sdata=VRr2wpmbugh9YodMppLP24odLyHKJASh%2BQ%2FcGjog568%3D&reserved=0 
    mon osd reporter subtree level = datacenter
    ^^ We do want to make sure at least two datacenters are seeing a
    datacenter go down, not individual hosts.
    # https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs.ceph.com%2Fdocs%2Fmaster%2Frados%2Fconfiguration%2Fmon-osd-interaction%2F&data=02%7C01%7Crovanleeuwen%40ebay.com%7C0fba2ce3537b4bf4de9308d53c6f066d%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636481369899935585&sdata=arg0XlbGTI6w3Kr2Rf0KRPa5U3VJS5pyDGpgA8NzX%2FA%3D&reserved=0
    osd crush update on start = false
    osd heartbeat interval = 1             # default 6
    osd mon heartbeat interval = 10        # default 30
    osd mon report interval min = 1        # default 5
    osd mon report interval max = 15       # default 120
    The osd would almost immediately see a "cut off" to their partner OSD's
    in the placement group. By default they wait 6 seconds before sending
    their report to the monitors. During our analysis this is exactly the
    time the monitors were keeping an election. By tuning all of the above
    we could get them to send their reports faster, and by the time the
    election process was finished the monitors would handle the reports from
    the OSDs and come to the conclusion that a DC is down, flag it down
    and allow for normal client IO again.
    Of course, stability and data safety is most important to us. So if any
    of these settings make you worry please let us know.
    Gr. Stefan
    [1]: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fceph.com%2Fcommunity%2Fnew-luminous-rados-improvements%2F&data=02%7C01%7Crovanleeuwen%40ebay.com%7C0fba2ce3537b4bf4de9308d53c6f066d%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636481369899935585&sdata=z888vKUYfYaZahwEbGnave4%2F9FcBBZ%2FfzcJQeFp7YAQ%3D&reserved=0
    | BIT BV  https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.bit.nl%2F&data=02%7C01%7Crovanleeuwen%40ebay.com%7C0fba2ce3537b4bf4de9308d53c6f066d%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636481369899935585&sdata=nqPladpP6VbLYSeRHEbseEL4JSmFRS0s%2BhRCTxzAQng%3D&reserved=0        Kamer van Koophandel 09090351
    | GPG: 0xD14839C6                   +31 318 648 688 / info at bit.nl
    ceph-users mailing list
    ceph-users at lists.ceph.com

More information about the ceph-users mailing list