[ceph-users] inconsistent pg on erasure coded pool

Frédéric Nass frederic.nass at univ-lorraine.fr
Thu Oct 5 00:01:40 PDT 2017


Hi Kenneth,

You should check for drive or XFS related errors in /var/log/message 
files on all nodes. We've had a similar issue in the past with a bad 
block on a hard drive.
We've had to :

1. Stop the OSD associated to the drive that had a bad block, flush its 
journal (ceph-osd -i $osd --flush-journal) and umount the filesystem,
2. Clear the bad blocks in the RAID/PERC Controller,
3. xfs_repair the partition, and partprobe the drive to start the OSD again,
4. ceph pg repair <pg_id>

Regards,

Frédéric.

Le 04/10/2017 à 14:02, Kenneth Waegeman a écrit :
> Hi,
>
> We have some inconsistency / scrub error on a Erasure coded pool, that 
> I can't seem to solve.
>
> [root at osd008 ~]# ceph health detail
> HEALTH_ERR 1 pgs inconsistent; 1 scrub errors
> pg 5.144 is active+clean+inconsistent, acting 
> [81,119,148,115,142,100,25,63,48,11,43]
> 1 scrub errors
>
> In the log files, it seems there is 1 missing shard:
>
> /var/log/ceph/ceph-osd.81.log.2.gz:2017-10-02 23:49:11.940624 
> 7f0a9d7e2700 -1 log_channel(cluster) log [ERR] : 5.144s0 shard 63(7) 
> missing 5:2297a2e1:::10014e2d8d5.00000000:head
> /var/log/ceph/ceph-osd.81.log.2.gz:2017-10-03 00:48:06.681941 
> 7f0a9d7e2700 -1 log_channel(cluster) log [ERR] : 5.144s0 deep-scrub 1 
> missing, 0 inconsistent objects
> /var/log/ceph/ceph-osd.81.log.2.gz:2017-10-03 00:48:06.681947 
> 7f0a9d7e2700 -1 log_channel(cluster) log [ERR] : 5.144 deep-scrub 1 
> errors
>
> I tried running ceph pg repair on the pg, but nothing changed. I also 
> tried starting a new deep-scrub on the  osd 81 (ceph osd deep-scrub 
> 81) but I don't see any deep-scrub starting at the osd.
>
> How can we solve this ?
>
> Thank you!
>
>
> Kenneth
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



More information about the ceph-users mailing list