[ceph-users] Filestore to Bluestore migration question

Ashley Merrick singapore at amerrick.co.uk
Wed Nov 7 07:37:07 PST 2018


Sorry my mixup.

Therefore you shouldn’t be running ZAP against /dev/sda as this will wipe
the whole SSD.

I Guess currently in its setup it’s using a partition on /dev/sda? Like
/dev/sda2 for example.

,Ashley

On Wed, 7 Nov 2018 at 11:30 PM, Hayashida, Mami <mami.hayashida at uky.edu>
wrote:

> Yes, that was indeed a copy-and-paste mistake.  I am trying to use
> /dev/sdh (hdd) for data and a part of /dev/sda (ssd)  for the journal.
> That's how the Filestore is set-up.  So, for the Bluestore, data on
> /dev/sdh,  wal and db on /dev/sda.
>
>
> On Wed, Nov 7, 2018 at 10:26 AM, Ashley Merrick <singapore at amerrick.co.uk>
> wrote:
>
>> ceph osd destroy 70  --yes-i-really-mean-it
>>
>> I am guessing that’s a copy and paste mistake and should say 120.
>>
>> Is the SSD @ /dev/sdh fully for the OSD120 is a partition on this SSD the
>> journal and other partitions are for other SSD’s?
>>
>> On Wed, 7 Nov 2018 at 11:21 PM, Hayashida, Mami <mami.hayashida at uky.edu>
>> wrote:
>>
>>> I would agree with that.  So, here is what I am planning on doing
>>> today.  I will try this from scratch on a different OSD node from the very
>>> first step and log input and output for every step.  Here is the outline of
>>> what I think (based on all the email exchanges so far) should happen.
>>>
>>> *******
>>> Trying to convert osd.120 to Bluestore.  Data is on /sda/sdh.
>>>  Filestore Journal is on a partition drive (40GB) on /dev/sda.
>>>
>>> #Mark those OSDs out
>>> ceph osd out 120
>>>
>>> # Stop the OSDs
>>> systemctl kill ceph-osd at 120
>>>
>>> # Unmount the filesystem
>>> sudo umount /var/lib/ceph/osd/ceph-120
>>>
>>> # Destroy the data
>>> ceph-volume lvm zap /dev/sdh --destroy   # data disk
>>> ceph-volume lvm zap /dev/sda --destroy   # ssd for wal and db
>>>
>>> # Inform the cluster
>>> ceph osd destroy 70  --yes-i-really-mean-it
>>>
>>> # Check all the /etc/fstab and /etc/systemd/system to make sure that all
>>> the references to the filesystem is gone. Run
>>> ln -sf /dev/null /etc/systemd/system/ceph-disk at 70.service
>>>
>>> # Create PVs, VGs, LVs
>>> pvcreate /dev/sda # for wal and db
>>> pvcreate /dev/sdh # for data
>>>
>>> vgcreate ssd0 /dev/sda
>>> vgcreate hdd120  /dev/sdh
>>>
>>> lvcreate -L 40G -n db120 ssd0
>>> lvcreate -l 100%VG data120 hdd120
>>>
>>> # Run ceph-volume
>>> ceph-volume lvm prepare --bluestore --data hdd120/data120 --block.db
>>> ssd0/db120  --osd-id 120
>>>
>>> # Activate
>>> ceph-volume lvm activate 120 <osd fsid>
>>>
>>> ******
>>> Does this sound right?
>>>
>>> On Tue, Nov 6, 2018 at 4:32 PM, Alfredo Deza <adeza at redhat.com> wrote:
>>>
>>>> It is pretty difficult to know what step you are missing if we are
>>>> getting the `activate --all` command.
>>>>
>>>> Maybe if you try one by one, capturing each command, throughout the
>>>> process, with output. In the filestore-to-bluestore guides we never
>>>> advertise `activate --all` for example.
>>>>
>>>> Something is missing here, and I can't tell what it is.
>>>> On Tue, Nov 6, 2018 at 4:13 PM Hayashida, Mami <mami.hayashida at uky.edu>
>>>> wrote:
>>>> >
>>>> > This is becoming even more confusing. I got rid of those ceph-disk at 6[0-9].service
>>>> (which had been symlinked to /dev/null).  Moved
>>>> /var/lib/ceph/osd/ceph-6[0-9] to  /var/...../osd_old/.  Then, I ran
>>>> `ceph-volume lvm activate --all`.  I got once again
>>>> >
>>>> > root at osd1:~# ceph-volume lvm activate --all
>>>> > --> Activating OSD ID 67 FSID 17cd6755-76f9-4160-906c-1bf13d09fb3d
>>>> > Running command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-67
>>>> > --> Absolute path not found for executable: restorecon
>>>> > --> Ensure $PATH environment variable contains common executable
>>>> locations
>>>> > Running command: ceph-bluestore-tool --cluster=ceph prime-osd-dir
>>>> --dev /dev/hdd67/data67 --path /var/lib/ceph/osd/ceph-67
>>>> >  stderr: failed to read label for /dev/hdd67/data67: (2) No such file
>>>> or directory
>>>> > -->  RuntimeError: command returned non-zero exit status: 1
>>>> >
>>>> > But when I ran `df` and `mount` ceph-67 is the only one that exists.
>>>> (and in  /var/lib/ceph/osd/)
>>>> >
>>>> > root at osd1:~# df -h | grep ceph-6
>>>> > tmpfs           126G     0  126G   0% /var/lib/ceph/osd/ceph-67
>>>> >
>>>> > root at osd1:~# mount | grep ceph-6
>>>> > tmpfs on /var/lib/ceph/osd/ceph-67 type tmpfs (rw,relatime)
>>>> >
>>>> > root at osd1:~# ls /var/lib/ceph/osd/ | grep ceph-6
>>>> > ceph-67
>>>> >
>>>> > But in I cannot restart any of these 10 daemons (`systemctl start
>>>> ceph-osd at 6[0-9]`).
>>>> >
>>>> > I am wondering if I should zap these 10 osds and start over although
>>>> at this point I am afraid even zapping may not be a simple task....
>>>> >
>>>> >
>>>> >
>>>> > On Tue, Nov 6, 2018 at 3:44 PM, Hector Martin <hector at marcansoft.com>
>>>> wrote:
>>>> >>
>>>> >> On 11/7/18 5:27 AM, Hayashida, Mami wrote:
>>>> >> > 1. Stopped osd.60-69:  no problem
>>>> >> > 2. Skipped this and went to #3 to check first
>>>> >> > 3. Here, `find /etc/systemd/system | grep ceph-volume` returned
>>>> >> > nothing.  I see in that directory
>>>> >> >
>>>> >> > /etc/systemd/system/ceph-disk at 60.service    # and 61 - 69.
>>>> >> >
>>>> >> > No ceph-volume entries.
>>>> >>
>>>> >> Get rid of those, they also shouldn't be there. Then `systemctl
>>>> >> daemon-reload` and continue, see if you get into a good state.
>>>> basically
>>>> >> feel free to nuke anything in there related to OSD 60-69, since
>>>> whatever
>>>> >> is needed should be taken care of by the ceph-volume activation.
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Hector Martin (hector at marcansoft.com)
>>>> >> Public Key: https://mrcn.st/pub
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Mami Hayashida
>>>> > Research Computing Associate
>>>> >
>>>> > Research Computing Infrastructure
>>>> > University of Kentucky Information Technology Services
>>>> > 301 Rose Street
>>>> <https://maps.google.com/?q=301+Rose+Street&entry=gmail&source=g> |
>>>> 102 James F. Hardymon Building
>>>> > Lexington, KY 40506-0495
>>>> > mami.hayashida at uky.edu
>>>> > (859)323-7521
>>>>
>>>
>>>
>>>
>>> --
>>> *Mami Hayashida*
>>>
>>> *Research Computing Associate*
>>> Research Computing Infrastructure
>>> University of Kentucky Information Technology Services
>>> 301 Rose Street
>>> <https://maps.google.com/?q=301+Rose+Street&entry=gmail&source=g> | 102
>>> James F. Hardymon Building
>>> Lexington, KY 40506-0495
>>> mami.hayashida at uky.edu
>>> (859)323-7521
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users at lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>
>
> --
> *Mami Hayashida*
>
> *Research Computing Associate*
> Research Computing Infrastructure
> University of Kentucky Information Technology Services
> 301 Rose Street
> <https://maps.google.com/?q=301+Rose+Street&entry=gmail&source=g> | 102
> James F. Hardymon Building
> Lexington, KY 40506-0495
> mami.hayashida at uky.edu
> (859)323-7521
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181107/bbfd4847/attachment.html>


More information about the ceph-users mailing list