[ceph-users] Filestore to Bluestore migration question

Hayashida, Mami mami.hayashida at uky.edu
Tue Oct 30 13:55:00 PDT 2018

I am relatively new to Ceph and need some advice on Bluestore migration. I
tried migrating a few of our test cluster nodes from Filestore to Bluestore
by following this (
as the cluster is currently running 12.2.9. The cluster, originally set up
by my predecessors, was running Jewel until I upgraded it recently to

OSDs in each OSD host is set up in such a way that for ever 10 data HDD
disks, there is one SSD drive that is holding their journals.  For example,
osd.0 data is on /dev/sdh and its Filestore journal is on a partitioned
part of /dev/sda. So, lsblk shows something like

sda       8:0    0 447.1G  0 disk
├─sda1    8:1    0    40G  0 part      # journal for osd.0

sdh       8:112  0   3.7T  0 disk
└─sdh1    8:113  0   3.7T  0 part /var/lib/ceph/osd/ceph-0

It seems like this was all set up by my predecessor with the following
command :

ceph-deploy osd create osd0:sdh:/dev/sda

Since sda is an SSD drive, even after Bluestore migration, I plan to keep
the DB and WAL for sdh (and 9 other data disks) on the /sda drive.

I followed all the way up to number 6 of
Then, instead of ceph-volume lvm zap $DEVICE, I used from an admin node,
ceph-deploy disk zap to wipe out the contents on those two drives.  Since
osd.0 - 9 share the SSD drive for their journals, I did the same for
osd.{1..9} as well as /dev/sda.  I then destroyed osd.{0..9}  using the osd
destroy command (step 8).

Where something definitely went wrong was the last part. ceph-volume lvm
create command shown assumes that WAL and DB will be on the same device as
the data.  I tried adding --block.wal --block.data to it, but did not work.
I tried various ceph-deploy commands taken from various versions of the
doc, but nothing seemed to work.  I even tried manually creating LVs for
WAL and DB (
but that did not work, either. At some point, the following command seems
to have worked (the only command that did not return an error), but then
all the OSDs on the node shut down and I could not bring any of them back

sudo ceph-disk prepare --bluestore /dev/sdh --block.wal=/dev/sda
--block.db=/dev/sda --osd-id 0

Since this is a test cluster with essentially no data on it, I can always
start over.  But I do need to know how to properly migrate OSDs from
Filestore to Bluestore in this type of setting (with Filestore journal
residing on an SSD) for our production clusters.  Please let me know if
there are any steps missing in the documentation particularly for a case
like this and what commands I need to run to achieve what I am trying to
do.  Also, if it is advisable to upgrade to Mimic first, then perform the
FIlestore to Bluestore migration, that is an option as well.

*Mami Hayashida*

*Research Computing Associate*
Research Computing Infrastructure
University of Kentucky Information Technology Services
301 Rose Street | 102 James F. Hardymon Building
Lexington, KY 40506-0495
mami.hayashida at uky.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181030/27ca2322/attachment.html>

More information about the ceph-users mailing list