[ceph-users] rocksdb: Corruption: missing start of fragmented record

Gregory Farnum gfarnum at redhat.com
Sun Nov 19 02:01:40 PST 2017


On Thu, Nov 2, 2017 at 8:55 AM Michael <mehe.schmid at gmx.ch> wrote:

> Christian Balzer wrote:
> > Your exact system configuration (HW, drives, controller, settings, etc)
> > would be interesting as I can think of plenty scenarios on how to corrupt
> > things that normally shouldn't be affected by such actions
> Oh, the hardware in question is consumer grade and not new. Some old i7
> machine. But my current guess is that the specific hardware is
> semi-unrelated.
>
> I think there probably is just a WAL log or entry in that that wasn't
> finished writing to disk or corrupted. But WAL aren't something that
> should be fully assumed to be complete and correct in the case of some
> failure, right? They are WAL.
>
> If this can't be fixed automatically with some command, I would simply
> like to have a look at & tinker trivially the with these DB files if
> possible (which I still haven't figured a way how to do).
>

Your hardware and configuration is very relevant. As you note, the WAL
should be able to handle being incompletely-written, and both Ceph and
RocksDB are designed to handle failures mid-write. That RocksDB *isn't*
doing that here implies either
1) there's a fatal bug in rocksdb, or
2) your hardware configuration is not honoring the storage consistency APIs
correctly.

Given past experience, (2) is far more likely than (1) is. The RocksDB
community may be able to give you more immediate guidance to what exactly
went wrong, but I'd look at whether you have a writeback cache somewhere
that isn't reflecting ordering requirements, or if your disk passes a crash
consistency tester. (No, I don't know one off-hand. But many disks lie
horribly even about stuff like flushes.)
-Greg


>
>
> Christian Balzer wrote:
> > Now that bit is quite disconcerting, though you're one release behind the
> > curve and from what I read .2 has plenty more bug fixes coming.
> Fair point. I just tried with 12.2.1 (on pre-release Ubuntu bionic now).
>
> Doesn't change anything - fsck doesn't fix rocksdb, the bluestore won't
> mount, the OSD won't activate and the error is the same.
>
> Is there any fix in .2 that might address this, or do you just mean that
> in general there will be bug fixes?
>
>
> Thanks for your response!
>
> - Michael
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171119/cbfcca75/attachment.html>


More information about the ceph-users mailing list