[ceph-users] Mimic upgrade 13.2.1 > 13.2.2 monmap changed

Paul Emmerich paul.emmerich at croit.io
Thu Oct 4 08:30:30 PDT 2018


You can manually extract, edit, and inject the mon map to manually fix it.
In this case you probably need to:

1. check what exactly is going on, inspect the mon map of all mons
2. maybe the IP addresses changed or something? see if you can fix it
somehow without editing the monmap
3. adjust the mon map accordingly and inject it back into 2 mons

You can use these instructions to get/set/edit the mon map:
http://docs.ceph.com/docs/mimic/rados/operations/add-or-rm-mons/#removing-monitors-from-an-unhealthy-cluster

Paul
Am Do., 4. Okt. 2018 um 14:40 Uhr schrieb Nino Bosteels
<n.bosteels at proximedia.be>:
>
> Hello list,
>
>
>
> I’m having a serious issue, since my ceph cluster has become unresponsive. I was upgrading my cluster (3 servers, 3 monitors) from 13.2.1 to 13.2.2, which shouldn’t be a problem.
>
>
>
> Though on reboot my first host reported:
>
>
>
> starting mon.ceph01 rank -1 at 192.168.200.197:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph01 fsid 27dd45f1-28b5-4ac6-81ab-c62bc581130c
>
> mon.cephxx at -1(probing) e5 preinit fsid 27dd45f1-28b5-4ac6-81ab-c62bc581130c
>
> mon.cephxx at -1(probing) e5 not in monmap and have been in a quorum before; must have been removed
>
> -1 mon.cephxx at -1(probing) e5 commit suicide!
>
> -1 failed to initialize
>
>
>
> I thought, perhaps the monitor doesn’t want to accept the monmap of the other 2, because of the version-difference. Sadly, I upgraded and rebooted the second server.
>
>
>
> Since the cluster is unresponsive (because more than half of the monitors is offline / out of quorum). The logs of my second host, it keeps spamming:
>
>
>
> 2018-10-04 14:39:06.802 7fed0058f700 -1 mon.ceph02 at 1(probing) e6 get_health_metrics reporting 14 slow ops, oldest is auth(proto 0 27 bytes epoch 6)
>
>
>
> Any help VERY MUCH appreciated, this sucks.
>
>
>
> Thanks
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90


More information about the ceph-users mailing list