[ceph-users] understanding % used in ceph df

Florian Engelmann florian.engelmann at everyware.ch
Fri Oct 19 02:47:37 PDT 2018


Hi,


Our Ceph cluster is a 6 Node cluster each node having 8 disks. The 
cluster is used for object storage only (right now). We do use EC 3+2 on 
the buckets.data pool.

We had a problem with RadosGW segfaulting (12.2.5) till we upgraded to 
12.2.8. We had almost 30.000 radosgw crashes leading to millions of 
unreferenced objects (failed multiuploads?). It filled our cluster so 
fast that we are now in danger to run out of space.

As you can see we are reweighting some OSDs right now. But the real 
question is how "used" is calculated in ceph df.

Global: %RAW USED = 76.49%

while

x-1.rgw.buckets.data Used = 90.32%

Am I right this is because we should still be "able" to loose one OSD node?

If thats true - reweight can help just a little to rebalance the 
capacity used on each node?

The only chance we have right now to survive until new HDDs arrive is to 
delete objects, right?


ceph -s
   cluster:
     id:     a2222146-6561-307e-b032-xxxxxxxxxxxxx
     health: HEALTH_WARN
             3 nearfull osd(s)
             13 pool(s) nearfull
             1 large omap objects
             766760/180478374 objects misplaced (0.425%)

   services:
     mon:     3 daemons, quorum ceph1-mon3,ceph1-mon2,ceph1-mon1
     mgr:     ceph1-mon2(active), standbys: ceph1-mon1, ceph1-mon3
     osd:     36 osds: 36 up, 36 in; 24 remapped pgs
     rgw:     3 daemons active
     rgw-nfs: 2 daemons active

   data:
     pools:   13 pools, 1424 pgs
     objects: 36.10M objects, 115TiB
     usage:   200TiB used, 61.6TiB / 262TiB avail
     pgs:     766760/180478374 objects misplaced (0.425%)
              1400 active+clean
              16   active+remapped+backfill_wait
              8    active+remapped+backfilling

   io:
     client:   3.05MiB/s rd, 0B/s wr, 1.12kop/s rd, 37op/s wr
     recovery: 306MiB/s, 91objects/s

ceph df
GLOBAL:
     SIZE       AVAIL       RAW USED     %RAW USED
     262TiB     61.6TiB       200TiB         76.49
POOLS:
     NAME                        ID     USED        %USED     MAX AVAIL 
    OBJECTS
     iscsi-images                1          35B         0       6.87TiB 
           5
     .rgw.root                   2      3.57KiB         0       6.87TiB 
          18
     x-1.rgw.buckets.data       6       115TiB     90.32       12.4TiB 
   36090523
     x-1.rgw.control            7           0B         0       6.87TiB 
          8
     x-1.rgw.meta               8       943KiB         0       6.87TiB 
       3265
     x-1.rgw.log                9           0B         0       6.87TiB 
        407
     x-1.rgw.buckets.index      12          0B         0       6.87TiB 
       3096
     x-1.rgw.buckets.non-ec     13          0B         0       6.87TiB 
       1623
     default.rgw.meta            14        373B         0       6.87TiB 
           3
     default.rgw.control         15          0B         0       6.87TiB 
           8
     default.rgw.log             16          0B         0       6.87TiB 
           0
     scbench                     17          0B         0       6.87TiB 
           0
     rbdbench                    18     1.00GiB      0.01       6.87TiB 
         260



Regards,
Flo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5210 bytes
Desc: not available
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181019/ad8f7af7/attachment.bin>


More information about the ceph-users mailing list