[ceph-users] Luminous cluster stuck when adding monitor
nico.schottelius at ungleich.ch
Wed Oct 4 12:16:34 PDT 2017
the logfile I produced has already debug mon = 20 set:
[21:03:51] server1:~# grep "debug mon" /etc/ceph/ceph.conf
debug mon = 20
It is clear that server1 is out of quorum, however how do we make it
being part of the quorum again?
I expected that the quorum finding process is triggered automatically
after restarting the monitor, or is that incorrect?
Gregory Farnum <gfarnum at redhat.com> writes:
> You'll need to change the config so that it's running "debug mon = 20" for
> the log to be very useful here. It does say that it's dropping client
> connections because it's been out of quorum for too long, which is the
> correct behavior in general. I'd imagine that you've got clients trying to
> connect to the new monitor instead of the ones already in the quorum and
> not passing around correctly; this is all configurable.
> On Wed, Oct 4, 2017 at 4:09 AM Nico Schottelius <
> nico.schottelius at ungleich.ch> wrote:
>> Good morning,
>> we have recently upgraded our kraken cluster to luminous and since then
>> noticed an odd behaviour: we cannot add a monitor anymore.
>> As soon as we start a new monitor (server2), ceph -s and ceph -w start to
>> The situation became worse, since one of our staff stopped an existing
>> monitor (server1), as restarting that monitor results in the same
>> situation, ceph -s hangs until we stop the monitor again.
>> We kept the monitor running for some minutes, but the situation never
>> cleares up.
>> The network does not have any firewall in between the nodes and there
>> are no host firewalls.
>> I have attached the output of the monitor on server1, running in
>> foreground using
>> root at server1:~# ceph-mon -i server1 --pid-file
>> /var/lib/ceph/run/mon.server1.pid -c /etc/ceph/ceph.conf --cluster ceph
>> --setuser ceph --setgroup ceph -d 2>&1 | tee cephmonlog
>> Does anyone see any obvious problem in the attached log?
>> Any input or hint would be appreciated!
>> Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
>> ceph-users mailing list
>> ceph-users at lists.ceph.com
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
More information about the ceph-users