[ceph-users] Filestore to Bluestore migration question

Hayashida, Mami mami.hayashida at uky.edu
Mon Nov 5 13:20:37 PST 2018


Yes, I still have the volume log showing the activation process for
ssd0/db60 (and 61-69 as well).   I will email it to you directly as an
attachment.


On Mon, Nov 5, 2018 at 4:14 PM, Alfredo Deza <adeza at redhat.com> wrote:

> On Mon, Nov 5, 2018 at 4:04 PM Hayashida, Mami <mami.hayashida at uky.edu>
> wrote:
> >
> > WOW.  With you two guiding me through every step, the 10 OSDs in
> question are now added back to the cluster as Bluestore disks!!!  Here are
> my responses to the last email from Hector:
> >
> > 1. I first checked the permissions and they looked like this
> >
> > root at osd1:/var/lib/ceph/osd/ceph-60# ls -l
> > total 56
> > -rw-r--r-- 1 ceph ceph         384 Nov  2 16:20 activate.monmap
> > -rw-r--r-- 1 ceph ceph 10737418240 Nov  2 16:20 block
> > lrwxrwxrwx 1 ceph ceph          14 Nov  2 16:20 block.db ->
> /dev/ssd0/db60
> >
> > root at osd1:~# ls -l /dev/ssd0/
> > ...
> > lrwxrwxrwx 1 root root 7 Nov  5 12:38 db60 -> ../dm-2
> >
> > root at osd1:~# ls -la /dev/
> > ...
> > brw-rw----  1 root disk    252,   2 Nov  5 12:38 dm-2
>
> This looks like a bug. You mentioned you are running 12.2.9, and we
> haven't seen problems in ceph-volume that fail to update the
> permissions on OSD devices. No one should need a UDEV rule to set the
> permissions for
> devices, this is a ceph-volume task.
>
> When a system starts and the OSD activation happens, it always ensures
> that the permissions are set correctly. Could you find the section of
> the logs in /var/log/ceph/ceph-volume.log that shows the activation
> process for ssd0/db60 ?
>
> Hopefully you still have those around, it would help us determine why
> the permissions aren't being set correctly.
>
> > ...
> >
> > 2. I then ran ceph-volume activate --all again.  Saw the same error for
> osd.67 I described many emails ago..  None of the permissions changed.  I
> tried restarting ceph-osd at 60, but got the same error as before:
> >
> > 2018-11-05 15:34:52.001782 7f5a15744e00  0 set uid:gid to 64045:64045
> (ceph:ceph)
> > 2018-11-05 15:34:52.001808 7f5a15744e00  0 ceph version 12.2.9 (
> 9e300932ef8a8916fb3fda78c58691a6ab0f4217) luminous (stable), process
> ceph-osd, pid 36506
> > 2018-11-05 15:34:52.021717 7f5a15744e00  0 pidfile_write: ignore empty
> --pid-file
> > 2018-11-05 15:34:52.033478 7f5a15744e00  0 load: jerasure load: lrc
> load: isa
> > 2018-11-05 15:34:52.033557 7f5a15744e00  1 bdev create path
> /var/lib/ceph/osd/ceph-60/block type kernel
> > 2018-11-05 15:34:52.033572 7f5a15744e00  1 bdev(0x5651bd1b8d80
> /var/lib/ceph/osd/ceph-60/block) open path /var/lib/ceph/osd/ceph-60/block
> > 2018-11-05 15:34:52.033888 7f5a15744e00  1 bdev(0x5651bd1b8d80
> /var/lib/ceph/osd/ceph-60/block) open size 10737418240 (0x280000000,
> 10GiB) block_size 4096 (4KiB) rotational
> > 2018-11-05 15:34:52.033958 7f5a15744e00  1 bluestore(/var/lib/ceph/osd/ceph-60)
> _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2
> > 2018-11-05 15:34:52.033984 7f5a15744e00  1 bdev(0x5651bd1b8d80
> /var/lib/ceph/osd/ceph-60/block) close
> > 2018-11-05 15:34:52.318993 7f5a15744e00  1 bluestore(/var/lib/ceph/osd/ceph-60)
> _mount path /var/lib/ceph/osd/ceph-60
> > 2018-11-05 15:34:52.319064 7f5a15744e00  1 bdev create path
> /var/lib/ceph/osd/ceph-60/block type kernel
> > 2018-11-05 15:34:52.319073 7f5a15744e00  1 bdev(0x5651bd1b8fc0
> /var/lib/ceph/osd/ceph-60/block) open path /var/lib/ceph/osd/ceph-60/block
> > 2018-11-05 15:34:52.319356 7f5a15744e00  1 bdev(0x5651bd1b8fc0
> /var/lib/ceph/osd/ceph-60/block) open size 10737418240 (0x280000000,
> 10GiB) block_size 4096 (4KiB) rotational
> > 2018-11-05 15:34:52.319415 7f5a15744e00  1 bluestore(/var/lib/ceph/osd/ceph-60)
> _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2
> > 2018-11-05 15:34:52.319491 7f5a15744e00  1 bdev create path
> /var/lib/ceph/osd/ceph-60/block.db type kernel
> > 2018-11-05 15:34:52.319499 7f5a15744e00  1 bdev(0x5651bd1b9200
> /var/lib/ceph/osd/ceph-60/block.db) open path /var/lib/ceph/osd/ceph-60/
> block.db
> > 2018-11-05 15:34:52.319514 7f5a15744e00 -1 bdev(0x5651bd1b9200
> /var/lib/ceph/osd/ceph-60/block.db) open open got: (13) Permission denied
> > 2018-11-05 15:34:52.319648 7f5a15744e00 -1 bluestore(/var/lib/ceph/osd/ceph-60)
> _open_db add block device(/var/lib/ceph/osd/ceph-60/block.db) returned:
> (13) Permission denied
> > 2018-11-05 15:34:52.319666 7f5a15744e00  1 bdev(0x5651bd1b8fc0
> /var/lib/ceph/osd/ceph-60/block) close
> > 2018-11-05 15:34:52.598249 7f5a15744e00 -1 osd.60 0 OSD:init: unable to
> mount object store
> > 2018-11-05 15:34:52.598269 7f5a15744e00 -1  ** ERROR: osd init failed:
> (13) Permission denied
> >
> > 3. Finally, I literally copied and pasted the udev rule Hector wrote out
> for me, then rebooted the server.
> >
> > 4. I tried restarting ceph-osd at 60 -- this time it came right up!!!  I
> was able to start all the rest, including ceph-osd at 67 which I thought did
> not get activated by lvm.
> >
> > 5. I checked from the admin node and verified osd.60-69 are all in the
> cluster as Bluestore OSDs and they indeed are.
> >
> > ********************
> > Thank you SO MUCH, both of you, for putting up with my novice questions
> all the way.  I am planning to convert the rest of the cluster the same way
> by reviewing this entire thread to trace what steps need to be taken.
> >
> > Mami
> >
> > On Mon, Nov 5, 2018 at 3:00 PM, Hector Martin <hector at marcansoft.com>
> wrote:
> >>
> >>
> >>
> >> On 11/6/18 3:31 AM, Hayashida, Mami wrote:
> >> > 2018-11-05 12:47:01.075573 7f1f2775ae00 -1
> bluestore(/var/lib/ceph/osd/ceph-60) _open_db add block
> device(/var/lib/ceph/osd/ceph-60/block.db) returned: (13) Permission
> denied
> >>
> >> Looks like the permissions on the block.db device are wrong. As far as I
> >> know ceph-volume is responsible for setting this at activation time.
> >>
> >> > I already ran the "ceph-volume lvm activate --all "  command right
> after
> >> > I prepared (using "lvm prepare") those OSDs.  Do I need to run the
> >> > "activate" command again?
> >>
> >> The activation is required on every boot to create the
> >> /var/lib/ceph/osd/* directory, but that should be automatically done by
> >> systemd units (since you didn't run it after the reboot and yet the
> >> directories exist, it seems to have worked).
> >>
> >> Can you ls -l the OSD directory (/var/lib/ceph/osd/ceph-60/) and also
> >> any devices symlinked to from there, to see the permissions?
> >>
> >> Then run the activate command again and list the permissions again to
> >> see if they have changed, and if they have, try to start the OSD again.
> >>
> >> I found one Ubuntu bug that suggests there may be a race condition:
> >>
> >> https://bugs.launchpad.net/bugs/1767087
> >>
> >> I get the feeling the ceph-osd activation may be happening before the
> >> block.db device is ready, so when it gets created by LVM it's already
> >> too late and doesn't have the right permissions. You could fix it with a
> >> udev rule (like Ubuntu did) but if this is indeed your issue then it
> >> sounds like something that should be fixed in Ceph. Perhaps all you need
> >> is a systemd unit override to make sure ceph-volume@* services only
> >> start after LVM is ready.
> >>
> >> A usable udev rule could look like this (e.g. put it in
> >> /etc/udev/rules.d/90-lvm-permisions.rules):
> >>
> >> ACTION=="change", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", \
> >> ENV{DM_LV_NAME}=="db*", ENV{DM_VG_NAME}=="ssd0", \
> >> OWNER="ceph", GROUP="ceph", MODE="660"
> >>
> >> Reboot after that and see if the OSDs come up without further action.
> >>
> >> --
> >> Hector Martin (hector at marcansoft.com)
> >> Public Key: https://mrcn.st/pub
> >
> >
> >
> >
> > --
> > Mami Hayashida
> > Research Computing Associate
> >
> > Research Computing Infrastructure
> > University of Kentucky Information Technology Services
> > 301 Rose Street | 102 James F. Hardymon Building
> > Lexington, KY 40506-0495
> > mami.hayashida at uky.edu
> > (859)323-7521
>



-- 
*Mami Hayashida*

*Research Computing Associate*
Research Computing Infrastructure
University of Kentucky Information Technology Services
301 Rose Street | 102 James F. Hardymon Building
Lexington, KY 40506-0495
mami.hayashida at uky.edu
(859)323-7521
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181105/0f45151f/attachment.html>


More information about the ceph-users mailing list