[ceph-users] Inconsistent PG, repair doesn't work

Brett Chancellor bchancellor at salesforce.com
Wed Oct 10 10:46:26 PDT 2018


Hi all,
  I have an inconsistent PG. I've tried running a repair and manual deep
scrub, but neither operation seems to actually do anything.  I've also
tried stopping the primary OSD, removing the object, and restarting the
OSD. The system copies the object back, but the inconsistent PG ERR remains.

## Ceph Health
HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent
OSD_SCRUB_ERRORS 1 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
    pg 75.302 is active+clean+inconsistent, acting [208,120,235]

## OSD log
2018-10-10 13:43:08.734034 7feb3bf96700  0 log_channel(cluster) log [DBG] :
75.302 deep-scrub starts
2018-10-10 13:43:35.355037 7feb3bf96700 -1 log_channel(cluster) log [ERR] :
75.302 shard 235: soid
75:40d6b566:::rbd_data.81d5654895863d.0000000000001900:head candidate had a
read error
2018-10-10 13:44:06.476651 7feb3bf96700 -1 log_channel(cluster) log [ERR] :
75.302 deep-scrub 0 missing, 1 inconsistent objects
2018-10-10 13:44:06.476659 7feb3bf96700 -1 log_channel(cluster) log [ERR] :
75.302 deep-scrub 1 errors

## list-inconsistent-obj fails to report anything
$ sudo rados list-inconsistent-pg vir400-volumes
["75.302"]
$ sudo rados list-inconsistent-obj 75.302
No scrub information available for pg 75.302
error 2: (2) No such file or directory

## PG Query Information
https://pastebin.com/5wa3mWDC


Thanks,
-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181010/678011c3/attachment.html>


More information about the ceph-users mailing list