[ceph-users] Live migration of VM using librbd and OpenStack

[Travis Rhoden]

Just for posterity, my ultimate solution was to patch nova on each
compute host to always return True in _check_shared_storage_test_file
(nova/virt/libvirt/driver.py)

This did make migration work with "nova live-migration", with one
caveat.  Since Nova is assuming that /var/lib/nova/instances is on
shared storage (and since I hard coded the check to say "yes, it
really is"), it thinks the /var/lib/nova/instances/<domain> folder
will exist at both source and destination, and makes no attempt to
create it on the destination.  So before I run live-migration, I pop
over to the source host, and rsync that folder to the destination.

A little dirty, but it allows me to move running VMs around just fine
in cases of maintenance on a host, which is exactly what I need.

Thanks for everyone's feedback.

 - Travis

On Tue, Mar 12, 2013 at 6:33 PM, Travis Rhoden <trhoden at gmail.com> wrote:
>
> On Tue, Mar 12, 2013 at 5:06 PM, Josh Durgin <josh.durgin at inktank.com>
> wrote:
>>
>> On 03/12/2013 01:48 PM, Travis Rhoden wrote:
>>>
>>> Hi Josh,
>>>
>>> Thanks for the info.  So if I want to do live migration with VMs that
>>> were
>>> launched with boot-from-volume, I'll need to use virsh to do the
>>> migration,
>>> rather than Nova.  Okay, that should be doable.  As an aside, I will
>>> probably want to look at the OpenStack DB and figure out how to tell it
>>> that the VM has moved to a different host.  I'd rather there not be a
>>> disconnect between Nova and libvirt about where the VM lives.  =)
>>
>>
>> It's probably not too hard to edit nova to skip the checks when the
>> instance is volume-backed, but if you don't want to do that, libvirt
>> should be fine, and a bit more flexible.
>
>
> After messing with it for a few hours, I'm thinking about doing just that.
> The nova edits should be easy.  Looks like it tests for shared storage by
> writing a file on the  migration destination, and trying to read it at the
> source.  I should be able to just comment out the check entirely, or make
> the check always pass.
>
> The virsh migrate strategy has been surprisingly difficult.  Since I'm
> migrating a Nova VM, I had to the following pre-requisites (so far).
>
> Define the /var/lib/nova/instance/<domain> dir on the destination
> Define/migrate the nova libvirt-nwfilter for the specific VM
>
> Then, when I try to do the actual migration, I always get (at the source):
>
>  error: internal error Process exited while reading console log output:
> chardev: opening backend "file" failed: Permission denied
>
> So QEMU is bailing, saying it can't read the console.log file.  When I go
> look at that file, it is created, but with owner root:root and perms 0600.
> However, libvirtd makes it libvirt-qemu:kvm, 0600 before KVM tries to
> actually start the VM.  I've always found this dynamic file ownership bit in
> KVM/libvirt/qemu very confusing.  Anyways, I tried a few different things,
> debug logging, etc.  Even tried disabling apparmor.  Still get permission
> denied each time.
>
> The commands I"m running manually should be identical to what OpenStack is
> doing, so I can't figure out why their migrate is working and mine wouldn't.
> Oh well, will edit Nova and give that shot.
>
>  - Travis