[ceph-users] Effects of restoring a cluster's mon from an older backup

Gregory Farnum gfarnum at redhat.com
Fri Nov 9 13:35:51 PST 2018


On Thu, Nov 8, 2018 at 3:41 AM Hector Martin <hector at marcansoft.com> wrote:

> I'm experimenting with single-host Ceph use cases, where HA is not
> important but data durability is.
>
> How does a Ceph cluster react to its (sole) mon being rolled back to an
> earlier state? The idea here is that the mon storage may not be
> redundant but would be (atomically, e.g. lvm snapshot and dump) backed
> up, say, daily. If the cluster goes down and then is brought back up
> with a mon backup that is several days to hours old, while the OSDs are
> up to date, what are the potential consequences?
>
> Of course I expect maintenance operations to be affected (obviously any
> OSDs added/removed would likely get confused). But what about regular
> operation? Things like snapshots and snapshot ranges. Is this likely to
> cause data loss, or would the OSDs and clients largely not be affected
> as long as the cluster config has not changed?
>
> There's a way of rebuilding the monmap from OSD data:
>
>
> http://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/#recovery-using-osds
>
> Would this be preferable to just restoring the mon from a backup?


Yes, do that, don't try and back up your monitor. If you restore a monitor
from backup then the monitor — your authoritative data source — will warp
back in time on what the OSD peering intervals look like, which snapshots
have been deleted and created, etc. It would be a huge disaster and
probably every running daemon or client would have to pause IO until the
monitor generated enough map epochs to "catch up" — and then the rest of
the cluster would start applying those changes and nothing would work right.



> What
> about the MDS map?
>

Unlike the OSDMap, the MDSMap doesn't really keep track of any persistent
data so it's much safer to rebuild or reset from scratch.
-Greg


>
> --
> Hector Martin (hector at marcansoft.com)
> Public Key: https://mrcn.st/pub
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181109/eb85e4cc/attachment.html>


More information about the ceph-users mailing list