[ceph-users] RBD image has no active watchers while OpenStack KVM VM is running

Wido den Hollander wido at 42on.com
Thu Nov 30 01:00:15 PST 2017


> Op 29 november 2017 om 14:56 schreef Jason Dillaman <jdillama at redhat.com>:
> 
> 
> We experienced this problem in the past on older (pre-Jewel) releases
> where a PG split that affected the RBD header object would result in
> the watch getting lost by librados. Any chance you know if the
> affected RBD header objects were involved in a PG split? Can you
> generate a gcore dump of one of the affected VMs and ceph-post-file it
> for analysis?
> 

There was no PG splitting in the recent months on this cluster, so that's not something that might have happened here.

I've asked the OpenStack team for a gcore dump, but they have to get that cleared before they can send it to me.

This might take a bit of time!

Wido

> As for the VM going R/O, that is the expected behavior when a client
> breaks the exclusive lock held by a (dead) client.
> 
> On Wed, Nov 29, 2017 at 8:48 AM, Wido den Hollander <wido at 42on.com> wrote:
> > Hi,
> >
> > On a OpenStack environment I encountered a VM which went into R/O mode after a RBD snapshot was created.
> >
> > Digging into this I found 10s (out of thousands) RBD images which DO have a running VM, but do NOT have a watcher on the RBD image.
> >
> > For example:
> >
> > $ rbd status volumes/volume-79773f2e-1f40-4eca-b9f0-953fa8d83086
> >
> > 'Watchers: none'
> >
> > The VM is however running since September 5th 2017 with Jewel 10.2.7 on the client.
> >
> > In the meantime the cluster was already upgraded to 10.2.10
> >
> > Looking further I also found a Compute node with 10.2.10 installed which also has RBD images without watchers.
> >
> > Restarting or live migrating the VM to a different host resolves this issue.
> >
> > The internet is full of posts where RBD images still have Watchers when people don't expect them, but in this case I'm expecting a watcher which isn't there.
> >
> > The main problem right now is that creating a snapshot potentially puts a VM in Read-Only state because of the lack of notification.
> >
> > Has anybody seen this as well?
> >
> > Thanks,
> >
> > Wido
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users at lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> -- 
> Jason


More information about the ceph-users mailing list