[ceph-users] Inconsistent PG won't repair

Richard Bade hitrich at gmail.com
Wed Nov 8 13:20:07 PST 2017

For anyone that encounters this in the future, I was able to resolve
the issue by finding the three osd's that the object is on. One by one
I stop the osd, flushed the journal and used the objectstore tool to
remove the data (sudo ceph-objectstore-tool --data-path
/var/lib/ceph/osd/ceph-19 --journal-path
/dev/disk/by-partlabel/journal19 --pool tier3-rbd-3X
rbd_data.19cdf512ae8944a.000000000001bb56 remove). Then I started the
osd again and let it recover before moving on to the next osd.
After the object was deleted from all three osd's I ran a scrub on the
PG (ceph pg scrub 3.f05). Once the scrub was finished the
inconsistency went away.
Note, the object in question was empty (size of zero bytes) before I
started this process. I emptied the object by moving the rbd image to
another pool.


On 24 October 2017 at 14:34, Richard Bade <hitrich at gmail.com> wrote:
> What I'm thinking about trying is using the ceph-objectstore-tool to
> remove the offending clone metadata. From the help the syntax is this:
> ceph-objectstore-tool ... <object> remove-clone-metadata <cloneid>
> i.e. something like for my object and expected clone from the log message
> ceph-objectstore-tool rbd_data.19cdf512ae8944a.000000000001bb56
> remove-clone-metadata 148d2
> Anyone had experience with this? I'm not 100% sure if this will
> resolve the issue or cause much the same situation (since it's already
> expecting a clone that's not there currently).
> Rich
> On 21 October 2017 at 14:13, Brad Hubbard <bhubbard at redhat.com> wrote:
>> On Sat, Oct 21, 2017 at 1:59 AM, Richard Bade <hitrich at gmail.com> wrote:
>>> Hi Lincoln,
>>> Yes the object is 0-bytes on all OSD's. Has the same filesystem
>>> date/time too. Before I removed the rbd image (migrated disk to
>>> different pool) it was 4MB on all the OSD's and md5 checksum was the
>>> same on all so it seems that only metadata is inconsistent.
>>> Thanks for your suggestion, I just looked into this as I thought maybe
>>> I can delete the object (since it's empty anyway). But I just get file
>>> not found:
>>> ~$ rados stat rbd_data.19cdf512ae8944a.000000000001bb56 --pool=tier3-rbd-3X
>>>  error stat-ing
>>> tier3-rbd-3X/rbd_data.19cdf512ae8944a.000000000001bb56: (2) No such
>>> file or directory
>> Maybe try downing the osds involved?
>>> Regards,
>>> Rich
>>> On 21 October 2017 at 04:32, Lincoln Bryant <lincolnb at uchicago.edu> wrote:
>>>> Hi Rich,
>>>> Is the object inconsistent and 0-bytes on all OSDs?
>>>> We ran into a similar issue on Jewel, where an object was empty across the board but had inconsistent metadata. Ultimately it was resolved by doing a "rados get" and then a "rados put" on the object. *However* that was a last ditch effort after I couldn't get any other repair option to work, and I have no idea if that will cause any issues down the road :)
>>>> --Lincoln
>>>>> On Oct 20, 2017, at 10:16 AM, Richard Bade <hitrich at gmail.com> wrote:
>>>>> Hi Everyone,
>>>>> In our cluster running 0.94.10 we had a pg pop up as inconsistent
>>>>> during scrub. Previously when this has happened running ceph pg repair
>>>>> [pg_num] has resolved the problem. This time the repair runs but it
>>>>> remains inconsistent.
>>>>> ~$ ceph health detail
>>>>> HEALTH_ERR 1 pgs inconsistent; 2 scrub errors; noout flag(s) set
>>>>> pg 3.f05 is active+clean+inconsistent, acting [171,23,131]
>>>>> 1 scrub errors
>>>>> The error in the logs is:
>>>>> cstor01 ceph-mon: osd.171 45 : deep-scrub
>>>>> 3.f05 3/68ab5f05/rbd_data.19cdf512ae8944a.000000000001bb56/snapdir
>>>>> expected clone 3/68ab5f05/rbd_data.19cdf512ae8944a.000000000001bb56/148d2
>>>>> Now, I've tried several things to resolve this. I've tried stopping
>>>>> each of the osd's in turn and running a repair. I've located the rbd
>>>>> image and removed it to empty out the object. The object is now zero
>>>>> bytes but still inconsistent. I've tried stopping each osd, removing
>>>>> the object and starting the osd again. It correctly identifies the
>>>>> object as missing and repair works to fix this but it still remains
>>>>> inconsistent.
>>>>> I've run out of ideas.
>>>>> The object is now zero bytes:
>>>>> ~$ find /var/lib/ceph/osd/ceph-23/current/3.f05_head/ -name
>>>>> "*19cdf512ae8944a.000000000001bb56*" -ls
>>>>> 537598582      0 -rw-r--r--   1 root     root            0 Oct 21
>>>>> 03:54 /var/lib/ceph/osd/ceph-23/current/3.f05_head/DIR_5/DIR_0/DIR_F/DIR_5/DIR_B/rbd\\udata.19cdf512ae8944a.000000000001bb56__snapdir_68AB5F05__3
>>>>> How can I resolve this? Is there some way to remove the empty object
>>>>> completely? I saw reference to ceph-objectstore-tool which has some
>>>>> options to remove-clone-metadata but I don't know how to use this.
>>>>> Will using this to remove the mentioned 148d2 expected clone resolve
>>>>> this? Or would this do the opposite as it would seem that it can't
>>>>> find that clone?
>>>>> Documentation on this tool is sparse.
>>>>> Any help here would be appreciated.
>>>>> Regards,
>>>>> Rich
>>>>> _______________________________________________
>>>>> ceph-users mailing list
>>>>> ceph-users at lists.ceph.com
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users at lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> --
>> Cheers,
>> Brad

More information about the ceph-users mailing list