[ceph-users] Large OMAP Objects in default.rgw.log pool

Samuel Taylor Liston sam.liston at utah.edu
Thu Mar 7 10:46:34 PST 2019


Hello All, 
	I have recently had 32 large map objects appear in my default.rgw.log pool.  Running luminous 12.2.8.  

	Not sure what to think about these.    I’ve done a lot of reading about how when these normally occur it is related to a bucket needing resharding, but it doesn’t look like my default.rgw.log pool  has anything in it, let alone buckets.  Here’s some info on the system:  
        
[root at elm-rgw01 ~]# ceph versions
{
    "mon": {
        "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 5
    },
    "mgr": {
        "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 1
    },
    "osd": {
        "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 192
    },
    "mds": {},
    "rgw": {
        "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 1
    },
    "overall": {
        "ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 199
    }
}
[root at elm-rgw01 ~]# ceph osd pool ls
.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
default.rgw.buckets.index
default.rgw.buckets.non-ec
default.rgw.buckets.data
[root at elm-rgw01 ~]# ceph health detail
HEALTH_WARN 32 large omap objects
LARGE_OMAP_OBJECTS 32 large omap objects
    32 large objects found in pool 'default.rgw.log'
    Search the cluster log for 'Large omap object found' for more details.— 

Looking closer at these object they are all of size 0.  Also that pool shows a capacity usage of 0:

(just a sampling of the 236 objects at size 0)

[root at elm-mon01 ceph]# for i in `rados ls -p default.rgw.log`; do echo ${i}; rados stat -p default.rgw.log ${i};done
obj_delete_at_hint.0000000078
default.rgw.log/obj_delete_at_hint.0000000078 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000070
default.rgw.log/obj_delete_at_hint.0000000070 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000104
default.rgw.log/obj_delete_at_hint.0000000104 mtime 2019-03-07 11:39:20.000000, size 0
obj_delete_at_hint.0000000026
default.rgw.log/obj_delete_at_hint.0000000026 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000028
default.rgw.log/obj_delete_at_hint.0000000028 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000040
default.rgw.log/obj_delete_at_hint.0000000040 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000015
default.rgw.log/obj_delete_at_hint.0000000015 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000069
default.rgw.log/obj_delete_at_hint.0000000069 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000095
default.rgw.log/obj_delete_at_hint.0000000095 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000003
default.rgw.log/obj_delete_at_hint.0000000003 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000047
default.rgw.log/obj_delete_at_hint.0000000047 mtime 2019-03-07 11:39:19.000000, size 0


[root at elm-mon01 ceph]# rados df
POOL_NAME                  USED    OBJECTS   CLONES COPIES     MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS    RD      WR_OPS    WR     
.rgw.root                  1.09KiB         4      0         12                  0       0        0     14853 9.67MiB         0     0B 
default.rgw.buckets.data    444TiB 166829939      0 1000979634                  0       0        0 362357590  859TiB 909188749 703TiB 
default.rgw.buckets.index       0B       358      0       1074                  0       0        0 729694496 1.04TiB 522654976     0B 
default.rgw.buckets.non-ec      0B       182      0        546                  0       0        0 194204616  148GiB  97962607     0B 
default.rgw.control             0B         8      0         24                  0       0        0         0      0B         0     0B 
default.rgw.log                 0B       236      0        708                  0       0        0  33268863 3.01TiB  18415356     0B 
default.rgw.meta           16.2KiB        67      0        201                  0       0        0 466663427  371GiB     27647 146KiB 

total_objects    166830794
total_used       668TiB
total_avail      729TiB
total_space      1.36PiB


Does anyone know the importance of these objects or if they can safely be deleted?
Thank you,

Sam Liston (sam.liston at utah.edu)
========================================
Center for High Performance Computing
155 S. 1452 E. Rm 405
Salt Lake City, Utah 84112 (801)232-6932
========================================






More information about the ceph-users mailing list