[ceph-users] [EXTERNAL] Re: OSD service won't stay running - pg incomplete

Paul Emmerich paul.emmerich at croit.io
Fri Mar 15 16:10:41 PDT 2019


* Reducing min_size on an EC pool is equivalent to running with
min_size = 1 on a replicated pool; search mailing list archive for
details
* Running x+1 EC setups makes no sense because you have to choose
between being one disk failure away from an IO outage or running with
reduced min_size

Not being able to access it with ceph-objectstoretool is usually a
very bad sign :(

Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Thu, Mar 14, 2019 at 4:35 PM Benjamin.Zieglmeier
<Benjamin.Zieglmeier at target.com> wrote:
>
> Would you be willing to elaborate on what configuration specifically is bad? That would be helpful for future reference.
>
> Yes, we have tried to access with ceph-objectstore-tool to export the shard. The command spits out the tcmalloc lines shown in my previous output and then crashes with an 'Abort'.
>
> On 3/14/19, 5:22 AM, "Paul Emmerich" <paul.emmerich at croit.io> wrote:
>
>     You should never run a production cluster with this configuration.
>
>     Have you tried to access the disk with ceph-objectstoretool? The goal
>     would be export the shard of the PG on that disk and import it into
>     any other OSD.
>
>
>     Paul
>
>
>
>
>     Paul
>
>     --
>     Paul Emmerich
>
>     Looking for help with your Ceph cluster? Contact us at https://croit.io
>
>     croit GmbH
>     Freseniusstr. 31h
>     81247 München
>     www.croit.io
>     Tel: +49 89 1896585 90
>
>     On Wed, Mar 13, 2019 at 7:08 PM Benjamin.Zieglmeier
>     <Benjamin.Zieglmeier at target.com> wrote:
>     >
>     > After restarting several OSD daemons in our ceph cluster a couple days ago, a couple of our OSDs won’t come online. The services start and crash with the below error. We have one pg marked as incomplete, and will not peer. The pool is erasure coded, 2+1, currently set to size=3, min_size=2. The incomplete pg states it is not peering due to:
>     >
>     >
>     >
>     > "comment": "not enough complete instances of this PG" and:
>     >
>     >            "down_osds_we_would_probe": [
>     >
>     >                 7,
>     >
>     >                 16
>     >
>     >             ],
>     >
>     > 7 is completely lost, drive dead, 16 will not come online (refer to log output below).
>     >
>     >
>     >
>     > We’ve tried searching user-list and tweaking osd conf settings for several days, to no avail. Reaching out here as a last ditch effort before we have to give up on the pg.
>     >
>     >
>     >
>     > tcmalloc: large alloc 1073741824 bytes == 0x560ada35c000 @  0x7f5c1081e4ef 0x7f5c1083dbd6 0x7f5c0e945ab9 0x7f5c0e9466cb 0x7f5c0e946774 0x7f5c0e9469df 0x560a8fdb7db0 0x560a8fda8d28 0x560a8fdaa6b6 0x560a8fdab973 0x560a8fdacbb6 0x560a8f9f8f88 0x560a8f983d83 0x560a8f9b5d7e 0x560a8f474069 0x7f5c0dfc5445 0x560a8f514373
>     >
>     > tcmalloc: large alloc 2147483648 bytes == 0x560b1a35c000 @  0x7f5c1081e4ef 0x7f5c1083dbd6 0x7f5c0e945ab9 0x7f5c0e9466cb 0x7f5c0e946774 0x7f5c0e9469df 0x560a8fdb7db0 0x560a8fda8d28 0x560a8fdaa6b6 0x560a8fdab973 0x560a8fdacbb6 0x560a8f9f8f88 0x560a8f983d83 0x560a8f9b5d7e 0x560a8f474069 0x7f5c0dfc5445 0x560a8f514373
>     >
>     > tcmalloc: large alloc 4294967296 bytes == 0x560b9a35c000 @  0x7f5c1081e4ef 0x7f5c1083dbd6 0x7f5c0e945ab9 0x7f5c0e9466cb 0x7f5c0e946774 0x7f5c0e9469df 0x560a8fdb7db0 0x560a8fda8d28 0x560a8fdaa6b6 0x560a8fdab973 0x560a8fdacbb6 0x560a8f9f8f88 0x560a8f983d83 0x560a8f9b5d7e 0x560a8f474069 0x7f5c0dfc5445 0x560a8f514373
>     >
>     > tcmalloc: large alloc 3840745472 bytes == 0x560a9a334000 @  0x7f5c1081e4ef 0x7f5c1083dbd6 0x7f5c0e945ab9 0x7f5c0e945c76 0x7f5c0e94623e 0x560a8fdea280 0x560a8fda8f36 0x560a8fdaa6b6 0x560a8fdab973 0x560a8fdacbb6 0x560a8f9f8f88 0x560a8f983d83 0x560a8f9b5d7e 0x560a8f474069 0x7f5c0dfc5445 0x560a8f514373
>     >
>     > tcmalloc: large alloc 2728992768 bytes == 0x560e779ee000 @  0x7f5c1081e4ef 0x7f5c1083f010 0x560a8faa5674 0x560a8faa7125 0x560a8fa835a7 0x560a8fa5aa3c 0x560a8fa5c238 0x560a8fa77dcc 0x560a8fe439ef 0x560a8fe43c03 0x560a8fe5acd4 0x560a8fda75ec 0x560a8fda9260 0x560a8fdaa6b6 0x560a8fdab973 0x560a8fdacbb6 0x560a8f9f8f88 0x560a8f983d83 0x560a8f9b5d7e 0x560a8f474069 0x7f5c0dfc5445 0x560a8f514373
>     >
>     > /builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f5c0a749700 time 2019-03-13 12:46:39.632156
>     >
>     > /builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/KernelDevice.cc: 384: FAILED assert(0 == "unexpected aio error")
>     >
>     > 2019-03-13 12:46:39.632132 7f5c0a749700 -1 bdev(0x560a99c05000 /var/lib/ceph/osd/ceph-16/block) aio to 4817558700032~2728988672 but returned: 2147479552
>     >
>     >  ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
>     >
>     >  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x560a8fadd2a0]
>     >
>     >  2: (KernelDevice::_aio_thread()+0xd34) [0x560a8fa7fe24]
>     >
>     >  3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x560a8fa8517d]
>     >
>     >  4: (()+0x7e25) [0x7f5c0efb0e25]
>     >
>     >  5: (clone()+0x6d) [0x7f5c0e0a1bad]
>     >
>     >  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>     >
>     > 2019-03-13 12:46:39.633822 7f5c0a749700 -1 /builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f5c0a749700 time 2019-03-13 12:46:39.632156
>     >
>     > /builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/KernelDevice.cc: 384: FAILED assert(0 == "unexpected aio error")
>     >
>     >
>     >
>     >  ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
>     >
>     >  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x560a8fadd2a0]
>     >
>     >  2: (KernelDevice::_aio_thread()+0xd34) [0x560a8fa7fe24]
>     >
>     >  3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x560a8fa8517d]
>     >
>     >  4: (()+0x7e25) [0x7f5c0efb0e25]
>     >
>     >  5: (clone()+0x6d) [0x7f5c0e0a1bad]
>     >
>     >  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>     >
>     >
>     >
>     >     -1> 2019-03-13 12:46:39.632132 7f5c0a749700 -1 bdev(0x560a99c05000 /var/lib/ceph/osd/ceph-16/block) aio to 4817558700032~2728988672 but returned: 2147479552
>     >
>     >      0> 2019-03-13 12:46:39.633822 7f5c0a749700 -1 /builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f5c0a749700 time 2019-03-13 12:46:39.632156
>     >
>     > /builddir/build/BUILD/ceph-12.2.5/src/os/bluestore/KernelDevice.cc: 384: FAILED assert(0 == "unexpected aio error")
>     >
>     >
>     >
>     >  ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
>     >
>     >  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x560a8fadd2a0]
>     >
>     >  2: (KernelDevice::_aio_thread()+0xd34) [0x560a8fa7fe24]
>     >
>     >  3: (KernelDevice::AioCompletionThread::entry()+0xd) [0x560a8fa8517d]
>     >
>     >  4: (()+0x7e25) [0x7f5c0efb0e25]
>     >
>     >  5: (clone()+0x6d) [0x7f5c0e0a1bad]
>     >
>     >  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>     >
>     >
>     >
>     > *** Caught signal (Aborted) **
>     >
>     >  in thread 7f5c0a749700 thread_name:bstore_aio
>     >
>     >  ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
>     >
>     >  1: (()+0xa41911) [0x560a8fa9e911]
>     >
>     >  2: (()+0xf6d0) [0x7f5c0efb86d0]
>     >
>     >  3: (gsignal()+0x37) [0x7f5c0dfd9277]
>     >
>     >  4: (abort()+0x148) [0x7f5c0dfda968]
>     >
>     >  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x560a8fadd414]
>     >
>     >  6: (KernelDevice::_aio_thread()+0xd34) [0x560a8fa7fe24]
>     >
>     >  7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x560a8fa8517d]
>     >
>     >  8: (()+0x7e25) [0x7f5c0efb0e25]
>     >
>     >  9: (clone()+0x6d) [0x7f5c0e0a1bad]
>     >
>     > 2019-03-13 12:46:39.635955 7f5c0a749700 -1 *** Caught signal (Aborted) **
>     >
>     >  in thread 7f5c0a749700 thread_name:bstore_aio
>     >
>     >
>     >
>     >  ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
>     >
>     >  1: (()+0xa41911) [0x560a8fa9e911]
>     >
>     >  2: (()+0xf6d0) [0x7f5c0efb86d0]
>     >
>     >  3: (gsignal()+0x37) [0x7f5c0dfd9277]
>     >
>     >  4: (abort()+0x148) [0x7f5c0dfda968]
>     >
>     >  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x560a8fadd414]
>     >
>     >  6: (KernelDevice::_aio_thread()+0xd34) [0x560a8fa7fe24]
>     >
>     >  7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x560a8fa8517d]
>     >
>     >  8: (()+0x7e25) [0x7f5c0efb0e25]
>     >
>     >  9: (clone()+0x6d) [0x7f5c0e0a1bad]
>     >
>     >  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>     >
>     >
>     >
>     >      0> 2019-03-13 12:46:39.635955 7f5c0a749700 -1 *** Caught signal (Aborted) **
>     >
>     >  in thread 7f5c0a749700 thread_name:bstore_aio
>     >
>     >
>     >
>     >  ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
>     >
>     >  1: (()+0xa41911) [0x560a8fa9e911]
>     >
>     >  2: (()+0xf6d0) [0x7f5c0efb86d0]
>     >
>     >  3: (gsignal()+0x37) [0x7f5c0dfd9277]
>     >
>     >  4: (abort()+0x148) [0x7f5c0dfda968]
>     >
>     >  5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x284) [0x560a8fadd414]
>     >
>     >  6: (KernelDevice::_aio_thread()+0xd34) [0x560a8fa7fe24]
>     >
>     >  7: (KernelDevice::AioCompletionThread::entry()+0xd) [0x560a8fa8517d]
>     >
>     >  8: (()+0x7e25) [0x7f5c0efb0e25]
>     >
>     >  9: (clone()+0x6d) [0x7f5c0e0a1bad]
>     >
>     >  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>     >
>     >
>     >
>     > Aborted
>     >
>     >
>     >
>     > _______________________________________________
>     > ceph-users mailing list
>     > ceph-users at lists.ceph.com
>     > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


More information about the ceph-users mailing list