[ceph-users] blustore - howto remove object that is crashing osd

Marek Grzybowski marek.grzybowski at gmail.com
Sun Oct 8 08:03:07 PDT 2017


Hi
  I have single node cephfs on top of EC pool on top of bluestore ( my dream setup ;) ) .
I hit bug that is crushing my osd ( first only one , then second before data backfilled  ):
http://tracker.ceph.com/issues/20997

After some investigation i found that there is object
("3#22:addebba8:::10000014f8e.00000145:head#32ce3" ( please note abnormal "#32ce3" at the end )  )
that  is probably causing this.

 - ceph-bluestore-tool deep fsck : ceph-bluestore-tool  --path /var/lib/ceph/osd/ceph-1/ --deep yes --command fsck
   "finish with 0 errors"
 - ceph-objectstore-tool fsck :  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1/ --op fsck
   "fsck found no errors"
 - export of the affected pg (22.5s3) is failing:

ceph-objectstore-tool --force --data-path /var/lib/ceph/osd/ceph-1/  --pgid 22.5s3 --op export --file /tank1/ceph-recovery/recover.22.5s3v2 2>&1 | tee recover.22.5s3v2.log
....
Read 3#22:addea9c4:::1000000d42b.00000073:head#
Read 3#22:addeb3f7:::10000009d3b.00000440:head#
Read 3#22:addeb487:::1000000a84f.00000e50:head#
Read 3#22:addebba8:::10000014f8e.00000145:head#32ce3
getattr failure object_info -61
export_files error -61

  - tools like ceph-objectstore-tool shows warning when they for example list objects for pg  :
Error getting attr on : 22.5s3_head,3#22:addebba8:::10000014f8e.00000145:head#/32ce3, (61) No data available

  - I can clearly see directory for this strange object, when i fuse-mount bluestore.

/mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3
/mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/bitwise_hash
/mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/data
/mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/omap
/mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/attr
/mnt/target/22.5s3_head/all/3#22:addebba8:::10000014f8e.00000145:head#32ce3/omap_header

I read all documentation, and a loot of therads on this list
( this list is really great for info about tools like ceph-objectstore-tool ),
but i can not find any way to remove/modify this broken object from bluestore.

My data are anacessable, i do not know when and if this bug will be fixed in ceph code.
Should i focus on removing this strange/broken object (like hack ceph-objectstore-tool or ceph-bluestore-tool),
or is there something else i can/should do to get my osd.1 up and running again ?


-- 
  Kind Regards
    Marek Grzybowski




More information about the ceph-users mailing list