[ceph-users] RGW: ERROR: failed to distribute cache

Yehuda Sadeh-Weinraub yehuda at redhat.com
Mon Nov 6 11:17:54 PST 2017

On Mon, Nov 6, 2017 at 7:29 AM, Wido den Hollander <wido at 42on.com> wrote:
> Hi,
> On a Ceph Luminous (12.2.1) environment I'm seeing RGWs stall and about the same time I see these errors in the RGW logs:
> 2017-11-06 15:50:24.859919 7f8f5fa1a700  0 ERROR: failed to distribute cache for gn1-pf.rgw.data.root:.bucket.meta.XXXXX:eb32b1ca-807a-4867-aea5-ff43ef7647c6.20755572.20
> 2017-11-06 15:50:41.768881 7f8f7824b700  0 ERROR: failed to distribute cache for gn1-pf.rgw.data.root:XXXXX
> 2017-11-06 15:55:15.781739 7f8f7824b700  0 ERROR: failed to distribute cache for gn1-pf.rgw.meta:.meta:bucket.instance:XXXXX:eb32b1ca-807a-4867-aea5-ff43ef7647c6.20755572.32:_XK5LExyXXXXX6EEIXxCD5Cws:1
> 2017-11-06 15:55:25.784404 7f8f7824b700  0 ERROR: failed to distribute cache for gn1-pf.rgw.data.root:.bucket.meta.XXXXX:eb32b1ca-807a-4867-aea5-ff43ef7647c6.20755572.32
> I see one message from a year ago: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-June/010531.html
> The setup has two RGWs running:
> - ceph-rgw1
> - ceph-rgw2
> While trying to figure this out I see that a "radosgw-admin period pull" hangs for ever.
> I don't know if that is related, but it's something I've noticed.
> Mainly I see that at random times the RGW stalls for about 30 seconds and while that happens these messages show up in the RGW's log.

do you happen to know if there's a dynamic resharding happening? The
dynamic resharding should only affect the writes to the specific
bucket, and should not affect cache distribution though. Originally I
thought it could be HUP signal related issue, but that seem to be
fixed in 12.2.1.


> Is anybody else running into this issue?
> Wido
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

More information about the ceph-users mailing list