[ceph-users] Unfound object on erasure when recovering

Jan Pekař - Imatic jan.pekar at imatic.cz
Thu Oct 4 10:21:48 PDT 2018


I thought, that putting disk "in" solves the problem, but not. Problem is there, but cluster don't see object as unfound, so it reports IO 
error.

This is what I got when using rados get command -

error getting erasure_3_1/100001eec49.00000000: (5) Input/output error

So maybe problem appeared before trying to re-balance my cluster and was invisible to me. But it never happened before and scrub and 
depp-scrub is running regularly.

I don't know where to continue with debugging this problem.

JP

On 3.10.2018 08:47, Jan Pekař - Imatic wrote:
> Hi all,
>
> I'm playing with my testing cluster with ceph 12.2.8 installed.
>
> It happened to me for the second time, that I have 1 unfound objects on erasure coded pool.
>
> I have erasure with 3+1 configuration.
>
> First time I was adding additional disk. During cluster rebalance I noticed one unfound object. I hoped, that it will be fixed after 
> cluster rebalance, but it was not.
>
> I coped with marking object as lost, because disk IO on that object stuck.
>
> Yesterday I was trying to remove one disk so I marked it out.
>
> After few hours I noticed also one object unfound. This is dump of pg list_missing.
>
> There is strange pool number, snapid (I'm not using snapshots on that pool, it is just pool for cephfs data), also locations array looks 
> strange.
>
> I decided to put disk I wanted to remove back "in" and unfound object disappeared.
>
> Can you give me additional informations to this problem? Should I debug it more?
>
> Thank you
>
> {
>     "offset": {
>         "oid": "",
>         "key": "",
>         "snapid": 0,
>         "hash": 0,
>         "max": 0,
>         "pool": -9223372036854775808,
>         "namespace": ""
>     },
>     "num_missing": 0,
>     "num_unfound": 1,
>     "objects": [
>         {
>             "oid": {
>                 "oid": "100001eec49.00000000",
>                 "key": "",
>                 "snapid": -2,
>                 "hash": 586898362,
>                 "max": 0,
>                 "pool": 10,
>                 "namespace": ""
>             },
>             "need": "13528'6795",
>             "have": "0'0",
>             "flags": "none",
>             "locations": [
>                 "7(3)"
>             ]
>         }
>     ],
>     "more": false
> }
>
>

-- 
============
Ing. Jan Pekař
jan.pekar at imatic.cz | +420603811737
----
Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz
============
--



More information about the ceph-users mailing list