[ceph-users] RGW: ERROR: failed to distribute cache

Mark Schouten mark at tuxis.nl
Mon Nov 6 07:48:50 PST 2017


I see this once on both my RGW's today:
rgw01:

2017-11-06 10:36:35.070068 7f4a4f300700  0 ERROR: failed to distribute cache for default.rgw.meta:.meta:bucket.instance:XXX/YYY:ZZZ.30636654.1::0
2017-11-06 10:36:45.139068 7f4a4f300700  0 ERROR: failed to distribute cache for default.rgw.data.root:.bucket.meta.XXX:YYY:ZZZ.30636654.1



rgw02:
2017-11-06 10:38:29.606736 7f2463658700  0 ERROR: failed to distribute cache for default.rgw.meta:.meta:bucket.instance:XXX/YYY:ZZZ.30636741.1::0
2017-11-06 10:38:39.647266 7f2463658700  0 ERROR: failed to distribute cache for default.rgw.data.root:.bucket.meta.XXX:YYY:ZZZ.30636741.1


Not sure if it's a coincidence, but it is the bucket that should be dynamically reindexed for resharding, which is broken (Issue #22046)

Met vriendelijke groeten,

-- 
Kerio Operator in de Cloud? https://www.kerioindecloud.nl/
Mark Schouten  | Tuxis Internet Engineering
KvK: 61527076 | http://www.tuxis.nl/
T: 0318 200208 | info at tuxis.nl



 Van:   Wido den Hollander <wido at 42on.com> 
 Aan:   <ceph-users at lists.ceph.com> 
 Verzonden:   6-11-2017 16:29 
 Onderwerp:   [ceph-users] RGW:  ERROR: failed to distribute cache 

Hi, 
 
On a Ceph Luminous (12.2.1) environment I'm seeing RGWs stall and about the same time I see these errors in the RGW logs: 
 
2017-11-06 15:50:24.859919 7f8f5fa1a700  0 ERROR: failed to distribute cache for gn1-pf.rgw.data.root:.bucket.meta.XXXXX:eb32b1ca-807a-4867-aea5-ff43ef7647c6.20755572.20 
2017-11-06 15:50:41.768881 7f8f7824b700  0 ERROR: failed to distribute cache for gn1-pf.rgw.data.root:XXXXX 
2017-11-06 15:55:15.781739 7f8f7824b700  0 ERROR: failed to distribute cache for gn1-pf.rgw.meta:.meta:bucket.instance:XXXXX:eb32b1ca-807a-4867-aea5-ff43ef7647c6.20755572.32:_XK5LExyXXXXX6EEIXxCD5Cws:1 
2017-11-06 15:55:25.784404 7f8f7824b700  0 ERROR: failed to distribute cache for gn1-pf.rgw.data.root:.bucket.meta.XXXXX:eb32b1ca-807a-4867-aea5-ff43ef7647c6.20755572.32 
 
I see one message from a year ago: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-June/010531.html 
 
The setup has two RGWs running: 
 
- ceph-rgw1 
- ceph-rgw2 
 
While trying to figure this out I see that a "radosgw-admin period pull" hangs for ever. 
 
I don't know if that is related, but it's something I've noticed. 
 
Mainly I see that at random times the RGW stalls for about 30 seconds and while that happens these messages show up in the RGW's log. 
 
Is anybody else running into this issue? 
 
Wido 
_______________________________________________ 
ceph-users mailing list 
ceph-users at lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171106/24e01bd3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3378 bytes
Desc: Electronic Signature S/MIME
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171106/24e01bd3/attachment.bin>


More information about the ceph-users mailing list