[ceph-users] Filestore to Bluestore migration question

Hector Martin "marcan" hector at marcansoft.com
Tue Nov 6 13:31:26 PST 2018


If /dev/hdd67/data67 does not exist, try `vgchange -a y` and that should make it exist, then try again. Not sure why this would ever happen, though, since I expect lower level stuff to take care of activating LVM LVs.

If it does exist, I get the feeling that your original ceph-volume prepare command created the OSD filesystems in your root filesystem as files (probably because the OSD directories already existed for some reason). In that case, yes, you should re-create them, since the first time it wasn't done correctly. Before you do that, make sure you unmount the tmpfs that is now mounted, that no osd directories remain for your BlueStore OSDs, that you remove them from the mons, etc. You want to make sure your environment is clean so everything works as it should. Might be worth removing and re-creating the LVM LVs to make sure the tags are gone too.

On November 7, 2018 6:12:43 AM GMT+09:00, "Hayashida, Mami" <mami.hayashida at uky.edu> wrote:
>This is becoming even more confusing. I got rid of those
>ceph-disk at 6[0-9].service
>(which had been symlinked to /dev/null).  Moved
>/var/lib/ceph/osd/ceph-6[0-9] to  /var/...../osd_old/.  Then, I ran
>`ceph-volume lvm activate --all`.  I got once again
>
>root at osd1:~# ceph-volume lvm activate --all
>--> Activating OSD ID 67 FSID 17cd6755-76f9-4160-906c-1bf13d09fb3d
>Running command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-67
>--> Absolute path not found for executable: restorecon
>--> Ensure $PATH environment variable contains common executable
>locations
>Running command: ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
>/dev/hdd67/data67 --path /var/lib/ceph/osd/ceph-67
>stderr: failed to read label for /dev/hdd67/data67: (2) No such file or
>directory
>-->  RuntimeError: command returned non-zero exit status: 1
>
>But when I ran `df` and `mount` ceph-67 is the only one that exists.
>(and
>in  /var/lib/ceph/osd/)
>
>root at osd1:~# df -h | grep ceph-6
>tmpfs           126G     0  126G   0% /var/lib/ceph/osd/ceph-67
>
>root at osd1:~# mount | grep ceph-6
>tmpfs on /var/lib/ceph/osd/ceph-67 type tmpfs (rw,relatime)
>
>root at osd1:~# ls /var/lib/ceph/osd/ | grep ceph-6
>ceph-67
>
>But in I cannot restart any of these 10 daemons (`systemctl start
>ceph-osd at 6
>[0-9]`).
>
>I am wondering if I should zap these 10 osds and start over although at
>this point I am afraid even zapping may not be a simple task....
>
>
>
>On Tue, Nov 6, 2018 at 3:44 PM, Hector Martin <hector at marcansoft.com>
>wrote:
>
>> On 11/7/18 5:27 AM, Hayashida, Mami wrote:
>> > 1. Stopped osd.60-69:  no problem
>> > 2. Skipped this and went to #3 to check first
>> > 3. Here, `find /etc/systemd/system | grep ceph-volume` returned
>> > nothing.  I see in that directory
>> >
>> > /etc/systemd/system/ceph-disk at 60.service    # and 61 - 69.
>> >
>> > No ceph-volume entries.
>>
>> Get rid of those, they also shouldn't be there. Then `systemctl
>> daemon-reload` and continue, see if you get into a good state.
>basically
>> feel free to nuke anything in there related to OSD 60-69, since
>whatever
>> is needed should be taken care of by the ceph-volume activation.
>>
>>
>> --
>> Hector Martin (hector at marcansoft.com)
>> Public Key: https://mrcn.st/pub
>>
>
>
>
>-- 
>*Mami Hayashida*
>
>*Research Computing Associate*
>Research Computing Infrastructure
>University of Kentucky Information Technology Services
>301 Rose Street | 102 James F. Hardymon Building
>Lexington, KY 40506-0495
>mami.hayashida at uky.edu
>(859)323-7521

-- 
Hector Martin "marcan" (hector at marcansoft.com)
Public key: https://mrcn.st/pub
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181107/6b05eddf/attachment.html>


More information about the ceph-users mailing list