[ceph-users] osdmap several thousand epochs behind latest

Chris Apsey bitskrieg at bitskrieg.net
Sun Jul 9 17:06:17 PDT 2017


Had a fairly substantial network interruption that knocked out about 
~270 osds:

      health HEALTH_ERR
             273/384 in osds are down
             noup,nodown,noout flag(s) set
      monmap e2: 3 mons at 
             election epoch 138, quorum 0,1,2 
         mgr no daemons active
      osdmap e37718: 384 osds: 111 up, 384 in; 16764 remapped pgs

We've had network interruptions before, and normally OSDs come back on 
their own, or do so with a service restart.  This time, no such luck 
(I'm guessing the scale was just too much).  After a few hours of trying 
to figure out why OSD services were running on the hosts (according to 
systemd) but marked 'down' in ceph osd tree, I found this thread: 
which appears to perfectly describe the scenario (high CPU usage, osdmap 
way out of sync, etc.)

I've taken the steps outlined and set the appropriate flags and am 
monitoring the 'catch up' progress of the OSDs.  The OSD farthest behind 
is about 5000 epochs out of sync, so I assume it will be a few hours 
before I see CPU usage level out.

Once the OSDs are caught up, are there any other steps I should take 
before 'ceph osd unset noup' (or anything to do after)?

Thanks in advance,


Chris Apsey
bitskrieg at bitskrieg.net

More information about the ceph-users mailing list