[ceph-users] ceph df space usage confusion - balancing needed?

Oliver Freyermuth freyermuth at physik.uni-bonn.de
Fri Oct 26 19:43:25 PDT 2018


Am 27.10.18 um 04:12 schrieb Linh Vu:
> Should be fine as long as your "mgr/balancer/max_misplaced" is reasonable. I find the default value of 0.05 decent enough, although from experience that seems like 0.05% rather than 5% as suggested here: http://docs.ceph.com/docs/luminous/mgr/balancer/  

Ok! I did actually choose 0.01. Interestingly, during the initial large rebalancing, it went up to > 2 % of misplaced objects (in small steps) until I decided to stop the balancer for a day to give the cluster
enough time to adapt. 
 
> You can also choose to turn it on only during certain hours when the cluster might be less busy. The config-keys are there somewhere (there's a post by Dan van der Ster on the ML about them) but they don't actually work in 12.2.8 at least, when I tried them. I suggest just use cron to turn the balancer on and off. 

I found that mail in the archives. Indeed, that seems helpful. I'll start with permanently leaving the balancer on for now and observe if it has any impact. Since we rarely change the cluster's layout,
it should effectively just sit there silently most of the time. 

Thanks!
Oliver

> 
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> *From:* Oliver Freyermuth <freyermuth at physik.uni-bonn.de>
> *Sent:* Friday, 26 October 2018 9:32:14 PM
> *To:* Linh Vu; Janne Johansson
> *Cc:* ceph-users at lists.ceph.com; Peter Wienemann
> *Subject:* Re: [ceph-users] ceph df space usage confusion - balancing needed?
>  
> Dear Cephalopodians,
> 
> thanks for all your feedback!
> 
> I finally "pushed the button" and let upmap run for ~36 hours.
> Previously, we had ~63 % usage of our CephFS with only 50 % raw usage, now, we see only 53.77 % usage.
> 
> That's as close as I expect things to ever become, and we gained about 70 TiB of free storage by that, which is almost one file server.
> So the outcome is really close to perfection :-).
> 
> I'm leaving the balancer active now in upmap mode. Any bad experiences with leaving it active "forever"?
> 
> Cheers and many thanks again,
>         Oliver
> 
> Am 23.10.18 um 01:14 schrieb Linh Vu:
>> Upmap is awesome. I ran it on our new cluster before we started ingesting data, so that the PG count is balanced on all OSDs. After ingesting about 315TB, it's still beautifully balanced. Note: we have a few nodes with 8TB OSDs, and the rest on 10TBs. 
>> 
>> 
>> # ceph osd df plain
>> ID  CLASS    WEIGHT  REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS 
>>   0   mf1hdd 7.27739  1.00000 7.28TiB 2.06TiB 5.21TiB 28.34 1.01 144 
>>   1   mf1hdd 7.27739  1.00000 7.28TiB 2.07TiB 5.21TiB 28.38 1.02 144 
>>   2   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 
>>   3   mf1hdd 7.27739  1.00000 7.28TiB 2.06TiB 5.21TiB 28.37 1.02 144 
>>   4   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 
>>   5   mf1hdd 7.27739  1.00000 7.28TiB 2.02TiB 5.26TiB 27.73 0.99 141 
>>   6   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.94 1.00 142 
>>   7   mf1hdd 7.27739  1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144 
>>   8   mf1hdd 7.27739  1.00000 7.28TiB 2.02TiB 5.26TiB 27.76 0.99 141 
>>   9   mf1hdd 7.27739  1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142 
>>  10   mf1hdd 7.27739  1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144 
>>  11   mf1hdd 7.27739  1.00000 7.28TiB 2.04TiB 5.24TiB 27.99 1.00 142 
>>  12   mf1hdd 7.27739  1.00000 7.28TiB 2.02TiB 5.26TiB 27.75 0.99 141 
>>  13   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 
>>  14   mf1hdd 7.27739  1.00000 7.28TiB 2.02TiB 5.26TiB 27.78 0.99 141 
>>  15   mf1hdd 7.27739  1.00000 7.28TiB 2.07TiB 5.21TiB 28.38 1.02 144 
>> 224 nvmemeta 0.02179  1.00000 22.3GiB 1.52GiB 20.8GiB  6.82 0.24 185 
>> 225 nvmemeta 0.02179  1.00000 22.4GiB 1.49GiB 20.9GiB  6.68 0.24 182 
>> 144   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>> 145   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>> 146   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>> 147   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 148   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 
>> 149   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 
>> 150   mf1hdd 8.91019  1.00000 8.91TiB 2.51TiB 6.40TiB 28.12 1.01 175 
>> 151   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 
>> 152   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.87 1.00 173 
>> 153   mf1hdd 8.91019  1.00000 8.91TiB 2.51TiB 6.40TiB 28.16 1.01 175 
>> 154   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.79 1.00 173 
>> 155   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 156   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>> 157   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>> 158   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.00 1.00 174 
>> 159   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>> 242 nvmemeta 0.02179  1.00000 22.3GiB 1.50GiB 20.8GiB  6.70 0.24 182 
>> 243 nvmemeta 0.02179  1.00000 22.4GiB 1.45GiB 20.9GiB  6.48 0.23 183 
>> 160   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173 
>> 161   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>> 162   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 
>> 163   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>> 164   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>> 165   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>> 166   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>> 167   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 
>> 168   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 
>> 169   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.86 1.00 173 
>> 170   mf1hdd 8.91019  1.00000 8.91TiB 2.51TiB 6.40TiB 28.12 1.01 175 
>> 171   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 
>> 172   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 
>> 173   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 
>> 174   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>> 175   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173 
>> 244 nvmemeta 0.02179  1.00000 22.3GiB 1.49GiB 20.9GiB  6.65 0.24 182 
>> 245 nvmemeta 0.02179  1.00000 22.4GiB 1.53GiB 20.8GiB  6.86 0.25 185 
>> 176   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>> 177   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 178   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 
>> 179   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>> 180   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>> 181   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>> 182   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 183   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.94 1.00 174 
>> 184   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 185   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 
>> 186   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 
>> 187   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 
>> 188   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 
>> 189   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>> 190   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 191   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174 
>> 246 nvmemeta 0.02179  1.00000 22.3GiB 1.48GiB 20.9GiB  6.64 0.24 184 
>> 247 nvmemeta 0.02179  1.00000 22.4GiB 1.47GiB 20.9GiB  6.57 0.24 183 
>> 192   mf1hdd 7.27739  1.00000 7.28TiB 2.06TiB 5.21TiB 28.34 1.01 144 
>> 193   mf1hdd 7.27739  1.00000 7.28TiB 2.04TiB 5.24TiB 27.96 1.00 142 
>> 194   mf1hdd 7.27739  1.00000 7.28TiB 2.06TiB 5.22TiB 28.31 1.01 144 
>> 195   mf1hdd 7.27739  1.00000 7.28TiB 2.05TiB 5.23TiB 28.14 1.01 143 
>> 196   mf1hdd 7.27739  1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142 
>> 197   mf1hdd 7.27739  1.00000 7.28TiB 2.05TiB 5.23TiB 28.16 1.01 143 
>> 198   mf1hdd 7.27739  1.00000 7.28TiB 2.06TiB 5.22TiB 28.34 1.01 144 
>> 199   mf1hdd 7.27739  1.00000 7.28TiB 2.04TiB 5.24TiB 27.98 1.00 142 
>> 200   mf1hdd 7.27739  1.00000 7.28TiB 2.05TiB 5.23TiB 28.15 1.01 143 
>> 201   mf1hdd 7.27739  1.00000 7.28TiB 2.06TiB 5.21TiB 28.36 1.02 144 
>> 202   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142 
>> 203   mf1hdd 7.27739  1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142 
>> 204   mf1hdd 7.27739  1.00000 7.28TiB 2.06TiB 5.21TiB 28.36 1.02 144 
>> 205   mf1hdd 7.27739  1.00000 7.28TiB 2.02TiB 5.26TiB 27.75 0.99 141 
>> 206   mf1hdd 7.27739  1.00000 7.28TiB 2.02TiB 5.25TiB 27.79 1.00 141 
>> 207   mf1hdd 7.27739  1.00000 7.28TiB 2.04TiB 5.24TiB 27.98 1.00 142 
>> 248 nvmemeta 0.02179  1.00000 22.3GiB 1.48GiB 20.9GiB  6.62 0.24 182 
>> 249 nvmemeta 0.02179  1.00000 22.4GiB 1.49GiB 20.9GiB  6.66 0.24 182 
>> 208   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 
>> 209   mf1hdd 7.27739  1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144 
>> 210   mf1hdd 7.27739  1.00000 7.28TiB 2.02TiB 5.26TiB 27.74 0.99 141 
>> 211   mf1hdd 7.27739  1.00000 7.28TiB 2.05TiB 5.23TiB 28.13 1.01 143 
>> 212   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.93 1.00 142 
>> 213   mf1hdd 7.27739  1.00000 7.28TiB 2.02TiB 5.26TiB 27.73 0.99 141 
>> 214   mf1hdd 7.27739  1.00000 7.28TiB 2.05TiB 5.23TiB 28.19 1.01 143 
>> 215   mf1hdd 7.27739  1.00000 7.28TiB 2.02TiB 5.26TiB 27.76 0.99 141 
>> 216   mf1hdd 7.27739  1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142 
>> 217   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 
>> 218   mf1hdd 7.27739  1.00000 7.28TiB 2.05TiB 5.23TiB 28.19 1.01 143 
>> 219   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 
>> 220   mf1hdd 7.27739  1.00000 7.28TiB 2.05TiB 5.22TiB 28.22 1.01 143 
>> 221   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142 
>> 222   mf1hdd 7.27739  1.00000 7.28TiB 2.05TiB 5.23TiB 28.15 1.01 143 
>> 223   mf1hdd 7.27739  1.00000 7.28TiB 2.05TiB 5.23TiB 28.15 1.01 143 
>> 250 nvmemeta 0.02179  1.00000 22.3GiB 1.46GiB 20.9GiB  6.54 0.23 183 
>> 251 nvmemeta 0.02179  1.00000 22.4GiB 1.49GiB 20.9GiB  6.66 0.24 183 
>>  16   mf1hdd 7.27739  1.00000 7.28TiB 2.06TiB 5.21TiB 28.34 1.01 144 
>>  17   mf1hdd 7.27739  1.00000 7.28TiB 2.05TiB 5.23TiB 28.16 1.01 143 
>>  18   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142 
>>  19   mf1hdd 7.27739  1.00000 7.28TiB 2.02TiB 5.26TiB 27.75 0.99 141 
>>  20   mf1hdd 7.27739  1.00000 7.28TiB 2.05TiB 5.23TiB 28.14 1.01 143 
>>  21   mf1hdd 7.27739  1.00000 7.28TiB 2.05TiB 5.23TiB 28.17 1.01 143 
>>  22   mf1hdd 7.27739  1.00000 7.28TiB 2.02TiB 5.26TiB 27.78 0.99 141 
>>  23   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 
>>  24   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142 
>>  25   mf1hdd 7.27739  1.00000 7.28TiB 2.02TiB 5.26TiB 27.78 0.99 141 
>>  26   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.94 1.00 142 
>>  27   mf1hdd 7.27739  1.00000 7.28TiB 2.04TiB 5.24TiB 27.97 1.00 142 
>>  28   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.95 1.00 142 
>>  29   mf1hdd 7.27739  1.00000 7.28TiB 2.03TiB 5.24TiB 27.96 1.00 142 
>>  30   mf1hdd 7.27739  1.00000 7.28TiB 2.04TiB 5.24TiB 27.98 1.00 142 
>>  31   mf1hdd 7.27739  1.00000 7.28TiB 2.06TiB 5.21TiB 28.35 1.02 144 
>> 226 nvmemeta 0.02179  1.00000 22.3GiB 1.48GiB 20.9GiB  6.62 0.24 182 
>> 227 nvmemeta 0.02179  1.00000 22.4GiB 1.46GiB 20.9GiB  6.51 0.23 182 
>>  32   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>>  33   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.42TiB 28.00 1.00 174 
>>  34   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.86 1.00 173 
>>  35   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 
>>  36   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>>  37   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174 
>>  38   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173 
>>  39   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 
>>  40   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>>  41   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 
>>  42   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>>  43   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.88 1.00 173 
>>  44   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173 
>>  45   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  46   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.86 1.00 173 
>>  47   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 
>> 228 nvmemeta 0.02179  1.00000 22.3GiB 1.49GiB 20.9GiB  6.65 0.24 182 
>> 229 nvmemeta 0.02179  1.00000 22.4GiB 1.45GiB 20.9GiB  6.47 0.23 183 
>>  48   mf1hdd 8.91019  1.00000 8.91TiB 2.51TiB 6.40TiB 28.17 1.01 175 
>>  49   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  50   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>>  51   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 
>>  52   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  53   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  54   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>>  55   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>>  56   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 
>>  57   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.88 1.00 173 
>>  58   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  59   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>>  60   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>>  61   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174 
>>  62   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 
>>  63   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.80 1.00 173 
>> 230 nvmemeta 0.02179  1.00000 22.3GiB 1.45GiB 20.9GiB  6.48 0.23 183 
>> 231 nvmemeta 0.02179  1.00000 22.4GiB 1.48GiB 20.9GiB  6.63 0.24 182 
>>  64   mf1hdd 8.91019  1.00000 8.91TiB 2.51TiB 6.40TiB 28.15 1.01 175 
>>  65   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>>  66   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 
>>  67   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>>  68   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>>  69   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 
>>  70   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174 
>>  71   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 
>>  72   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>>  73   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 
>>  74   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 
>>  75   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.02 1.00 174 
>>  76   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 
>>  77   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  78   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 
>>  79   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 
>> 232 nvmemeta 0.02179  1.00000 22.3GiB 1.47GiB 20.9GiB  6.57 0.24 182 
>> 233 nvmemeta 0.02179  1.00000 22.4GiB 1.50GiB 20.9GiB  6.70 0.24 184 
>>  80   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  81   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 
>>  82   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 
>>  83   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 
>>  84   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.79 1.00 173 
>>  85   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  86   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  87   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 
>>  88   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>>  89   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>>  90   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  91   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  92   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>>  93   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 
>>  94   mf1hdd 8.91019  1.00000 8.91TiB 2.51TiB 6.40TiB 28.15 1.01 175 
>>  95   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.78 0.99 173 
>> 234 nvmemeta 0.02179  1.00000 22.3GiB 1.51GiB 20.8GiB  6.77 0.24 185 
>> 235 nvmemeta 0.02179  1.00000 22.4GiB 1.49GiB 20.9GiB  6.65 0.24 182 
>>  96   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>>  97   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  98   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>>  99   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174 
>> 100   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 
>> 101   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>> 102   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 
>> 103   mf1hdd 8.91019  1.00000 8.91TiB 2.51TiB 6.40TiB 28.21 1.01 175 
>> 104   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 105   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 106   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 107   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 
>> 108   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 109   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>> 110   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>> 111   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 
>> 236 nvmemeta 0.02179  1.00000 22.3GiB 1.46GiB 20.9GiB  6.53 0.23 185 
>> 237 nvmemeta 0.02179  1.00000 22.4GiB 1.46GiB 20.9GiB  6.54 0.23 182 
>> 112   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 113   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 114   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.97 1.00 174 
>> 115   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>> 116   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 117   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 118   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 
>> 119   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.96 1.00 174 
>> 120   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 121   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 
>> 122   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.85 1.00 173 
>> 123   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.02 1.00 174 
>> 124   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 
>> 125   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.01 1.00 174 
>> 126   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.98 1.00 174 
>> 127   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 
>> 238 nvmemeta 0.02179  1.00000 22.3GiB 1.48GiB 20.9GiB  6.62 0.24 182 
>> 239 nvmemeta 0.02179  1.00000 22.4GiB 1.51GiB 20.8GiB  6.75 0.24 182 
>> 128   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.83 1.00 173 
>> 129   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>> 130   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 28.00 1.00 174 
>> 131   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 
>> 132   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>> 133   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 134   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>> 135   mf1hdd 8.91019  1.00000 8.91TiB 2.51TiB 6.40TiB 28.13 1.01 175 
>> 136   mf1hdd 8.91019  1.00000 8.91TiB 2.49TiB 6.42TiB 27.99 1.00 174 
>> 137   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 138   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>> 139   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.84 1.00 173 
>> 140   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.81 1.00 173 
>> 141   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 142   mf1hdd 8.91019  1.00000 8.91TiB 2.50TiB 6.41TiB 28.00 1.00 174 
>> 143   mf1hdd 8.91019  1.00000 8.91TiB 2.48TiB 6.43TiB 27.82 1.00 173 
>> 240 nvmemeta 0.02179  1.00000 22.3GiB 1.61GiB 20.7GiB  7.22 0.26 184 
>> 241 nvmemeta 0.02179  1.00000 22.4GiB 1.43GiB 20.9GiB  6.41 0.23 182 
>>                         TOTAL 1.85PiB  528TiB 1.33PiB 27.93          
>> MIN/MAX VAR: 0.23/1.02  STDDEV: 7.10
>> 
>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>> *From:* ceph-users <ceph-users-bounces at lists.ceph.com> on behalf of Oliver Freyermuth <freyermuth at physik.uni-bonn.de>
>> *Sent:* Sunday, 21 October 2018 6:57:49 AM
>> *To:* Janne Johansson
>> *Cc:* ceph-users at lists.ceph.com; Peter Wienemann
>> *Subject:* Re: [ceph-users] ceph df space usage confusion - balancing needed?
>>  
>> Ok, I'll try out the balancer end of the upcoming week then (after we've fixed a HW-issue with one of our mons
>> and the cooling system).
>> 
>> Until then, any further advice and whether upmap is recommended over crush-compat (all clients are Luminous) are welcome ;-).
>> 
>> Cheers,
>> Oliver
>> 
>> Am 20.10.18 um 21:26 schrieb Janne Johansson:
>>> Ok, can't say "why" then, I'd reweigh them somewhat to even it out,
>>> 1.22 -vs- 0.74 in variance is a lot, so either a balancer plugin for
>>> the MGRs, a script or just a few manual tweaks might be in order.
>>> 
>>> Den lör 20 okt. 2018 kl 21:02 skrev Oliver Freyermuth
>>> <freyermuth at physik.uni-bonn.de>:
>>>>
>>>> All OSDs are of the very same size. One OSD host has slightly more disks (33 instead of 31), though.
>>>> So also that that can't explain the hefty difference.
>>>>
>>>> I attach the output of "ceph osd tree" and "ceph osd df".
>>>>
>>>> The crush rule for the ceph_data pool is:
>>>> rule cephfs_data {
>>>>         id 2
>>>>         type erasure
>>>>         min_size 3
>>>>         max_size 6
>>>>         step set_chooseleaf_tries 5
>>>>         step set_choose_tries 100
>>>>         step take default class hdd
>>>>         step chooseleaf indep 0 type host
>>>>         step emit
>>>> }
>>>> So that only considers the hdd device class. EC is done with k=4 m=2.
>>>>
>>>> So I don't see any imbalance on the hardware level, but only a somewhat uneven distribution of PGs.
>>>> Am I missing something, or is this really just a case for the ceph balancer plugin?
>>>> I'm just a bit astonished this effect is so huge.
>>>> Maybe our 4096 PGs for the ceph_data pool are not enough to get an even distribution without balancing?
>>>> But it yields about 100 PGs per OSD, as you can see...
>>>>
>>>> --------------------------------------------------------------
>>>> # ceph osd tree
>>>> ID  CLASS WEIGHT    TYPE NAME       STATUS REWEIGHT PRI-AFF
>>>>  -1       826.26428 root default
>>>>  -3         0.43700     host mon001
>>>>   0   ssd   0.21799         osd.0       up  1.00000 1.00000
>>>>   1   ssd   0.21799         osd.1       up  1.00000 1.00000
>>>>  -5         0.43700     host mon002
>>>>   2   ssd   0.21799         osd.2       up  1.00000 1.00000
>>>>   3   ssd   0.21799         osd.3       up  1.00000 1.00000
>>>> -31         1.81898     host mon003
>>>> 230   ssd   0.90999         osd.230     up  1.00000 1.00000
>>>> 231   ssd   0.90999         osd.231     up  1.00000 1.00000
>>>> -10       116.64600     host osd001
>>>>   4   hdd   3.64499         osd.4       up  1.00000 1.00000
>>>>   5   hdd   3.64499         osd.5       up  1.00000 1.00000
>>>>   6   hdd   3.64499         osd.6       up  1.00000 1.00000
>>>>   7   hdd   3.64499         osd.7       up  1.00000 1.00000
>>>>   8   hdd   3.64499         osd.8       up  1.00000 1.00000
>>>>   9   hdd   3.64499         osd.9       up  1.00000 1.00000
>>>>  10   hdd   3.64499         osd.10      up  1.00000 1.00000
>>>>  11   hdd   3.64499         osd.11      up  1.00000 1.00000
>>>>  12   hdd   3.64499         osd.12      up  1.00000 1.00000
>>>>  13   hdd   3.64499         osd.13      up  1.00000 1.00000
>>>>  14   hdd   3.64499         osd.14      up  1.00000 1.00000
>>>>  15   hdd   3.64499         osd.15      up  1.00000 1.00000
>>>>  16   hdd   3.64499         osd.16      up  1.00000 1.00000
>>>>  17   hdd   3.64499         osd.17      up  1.00000 1.00000
>>>>  18   hdd   3.64499         osd.18      up  1.00000 1.00000
>>>>  19   hdd   3.64499         osd.19      up  1.00000 1.00000
>>>>  20   hdd   3.64499         osd.20      up  1.00000 1.00000
>>>>  21   hdd   3.64499         osd.21      up  1.00000 1.00000
>>>>  22   hdd   3.64499         osd.22      up  1.00000 1.00000
>>>>  23   hdd   3.64499         osd.23      up  1.00000 1.00000
>>>>  24   hdd   3.64499         osd.24      up  1.00000 1.00000
>>>>  25   hdd   3.64499         osd.25      up  1.00000 1.00000
>>>>  26   hdd   3.64499         osd.26      up  1.00000 1.00000
>>>>  27   hdd   3.64499         osd.27      up  1.00000 1.00000
>>>>  28   hdd   3.64499         osd.28      up  1.00000 1.00000
>>>>  29   hdd   3.64499         osd.29      up  1.00000 1.00000
>>>>  30   hdd   3.64499         osd.30      up  1.00000 1.00000
>>>>  31   hdd   3.64499         osd.31      up  1.00000 1.00000
>>>>  32   hdd   3.64499         osd.32      up  1.00000 1.00000
>>>>  33   hdd   3.64499         osd.33      up  1.00000 1.00000
>>>>  34   hdd   3.64499         osd.34      up  1.00000 1.00000
>>>>  35   hdd   3.64499         osd.35      up  1.00000 1.00000
>>>> -13       116.64600     host osd002
>>>>  36   hdd   3.64499         osd.36      up  1.00000 1.00000
>>>>  37   hdd   3.64499         osd.37      up  1.00000 1.00000
>>>>  38   hdd   3.64499         osd.38      up  1.00000 1.00000
>>>>  39   hdd   3.64499         osd.39      up  1.00000 1.00000
>>>>  40   hdd   3.64499         osd.40      up  1.00000 1.00000
>>>>  41   hdd   3.64499         osd.41      up  1.00000 1.00000
>>>>  42   hdd   3.64499         osd.42      up  1.00000 1.00000
>>>>  43   hdd   3.64499         osd.43      up  1.00000 1.00000
>>>>  44   hdd   3.64499         osd.44      up  1.00000 1.00000
>>>>  45   hdd   3.64499         osd.45      up  1.00000 1.00000
>>>>  46   hdd   3.64499         osd.46      up  1.00000 1.00000
>>>>  47   hdd   3.64499         osd.47      up  1.00000 1.00000
>>>>  48   hdd   3.64499         osd.48      up  1.00000 1.00000
>>>>  49   hdd   3.64499         osd.49      up  1.00000 1.00000
>>>>  50   hdd   3.64499         osd.50      up  1.00000 1.00000
>>>>  51   hdd   3.64499         osd.51      up  1.00000 1.00000
>>>>  52   hdd   3.64499         osd.52      up  1.00000 1.00000
>>>>  53   hdd   3.64499         osd.53      up  1.00000 1.00000
>>>>  54   hdd   3.64499         osd.54      up  1.00000 1.00000
>>>>  55   hdd   3.64499         osd.55      up  1.00000 1.00000
>>>>  56   hdd   3.64499         osd.56      up  1.00000 1.00000
>>>>  57   hdd   3.64499         osd.57      up  1.00000 1.00000
>>>>  58   hdd   3.64499         osd.58      up  1.00000 1.00000
>>>>  59   hdd   3.64499         osd.59      up  1.00000 1.00000
>>>>  60   hdd   3.64499         osd.60      up  1.00000 1.00000
>>>>  61   hdd   3.64499         osd.61      up  1.00000 1.00000
>>>>  62   hdd   3.64499         osd.62      up  1.00000 1.00000
>>>>  63   hdd   3.64499         osd.63      up  1.00000 1.00000
>>>>  64   hdd   3.64499         osd.64      up  1.00000 1.00000
>>>>  65   hdd   3.64499         osd.65      up  1.00000 1.00000
>>>>  66   hdd   3.64499         osd.66      up  1.00000 1.00000
>>>>  67   hdd   3.64499         osd.67      up  1.00000 1.00000
>>>> -16       116.63899     host osd003
>>>>  68   hdd   3.64499         osd.68      up  1.00000 1.00000
>>>>  69   hdd   3.64499         osd.69      up  1.00000 1.00000
>>>>  70   hdd   3.64499         osd.70      up  1.00000 1.00000
>>>>  71   hdd   3.64499         osd.71      up  1.00000 1.00000
>>>>  72   hdd   3.64499         osd.72      up  1.00000 1.00000
>>>>  73   hdd   3.64499         osd.73      up  1.00000 1.00000
>>>>  74   hdd   3.64499         osd.74      up  1.00000 1.00000
>>>>  75   hdd   3.64499         osd.75      up  1.00000 1.00000
>>>>  76   hdd   3.64499         osd.76      up  1.00000 1.00000
>>>>  77   hdd   3.64499         osd.77      up  1.00000 1.00000
>>>>  78   hdd   3.64499         osd.78      up  1.00000 1.00000
>>>>  79   hdd   3.64499         osd.79      up  1.00000 1.00000
>>>>  80   hdd   3.63899         osd.80      up  1.00000 1.00000
>>>>  81   hdd   3.64499         osd.81      up  1.00000 1.00000
>>>>  82   hdd   3.64499         osd.82      up  1.00000 1.00000
>>>>  83   hdd   3.64499         osd.83      up  1.00000 1.00000
>>>>  84   hdd   3.64499         osd.84      up  1.00000 1.00000
>>>>  85   hdd   3.64499         osd.85      up  1.00000 1.00000
>>>>  86   hdd   3.64499         osd.86      up  1.00000 1.00000
>>>>  87   hdd   3.64499         osd.87      up  1.00000 1.00000
>>>>  88   hdd   3.64499         osd.88      up  1.00000 1.00000
>>>>  89   hdd   3.64499         osd.89      up  1.00000 1.00000
>>>>  90   hdd   3.64499         osd.90      up  1.00000 1.00000
>>>>  91   hdd   3.64499         osd.91      up  1.00000 1.00000
>>>>  92   hdd   3.64499         osd.92      up  1.00000 1.00000
>>>>  93   hdd   3.64499         osd.93      up  1.00000 1.00000
>>>>  94   hdd   3.64499         osd.94      up  1.00000 1.00000
>>>>  95   hdd   3.64499         osd.95      up  1.00000 1.00000
>>>>  96   hdd   3.64499         osd.96      up  1.00000 1.00000
>>>>  97   hdd   3.64499         osd.97      up  1.00000 1.00000
>>>>  98   hdd   3.64499         osd.98      up  1.00000 1.00000
>>>>  99   hdd   3.64499         osd.99      up  1.00000 1.00000
>>>> -19       116.64600     host osd004
>>>> 100   hdd   3.64499         osd.100     up  1.00000 1.00000
>>>> 101   hdd   3.64499         osd.101     up  1.00000 1.00000
>>>> 102   hdd   3.64499         osd.102     up  1.00000 1.00000
>>>> 103   hdd   3.64499         osd.103     up  1.00000 1.00000
>>>> 104   hdd   3.64499         osd.104     up  1.00000 1.00000
>>>> 105   hdd   3.64499         osd.105     up  1.00000 1.00000
>>>> 106   hdd   3.64499         osd.106     up  1.00000 1.00000
>>>> 107   hdd   3.64499         osd.107     up  1.00000 1.00000
>>>> 108   hdd   3.64499         osd.108     up  1.00000 1.00000
>>>> 109   hdd   3.64499         osd.109     up  1.00000 1.00000
>>>> 110   hdd   3.64499         osd.110     up  1.00000 1.00000
>>>> 111   hdd   3.64499         osd.111     up  1.00000 1.00000
>>>> 112   hdd   3.64499         osd.112     up  1.00000 1.00000
>>>> 113   hdd   3.64499         osd.113     up  1.00000 1.00000
>>>> 114   hdd   3.64499         osd.114     up  1.00000 1.00000
>>>> 115   hdd   3.64499         osd.115     up  1.00000 1.00000
>>>> 116   hdd   3.64499         osd.116     up  1.00000 1.00000
>>>> 117   hdd   3.64499         osd.117     up  1.00000 1.00000
>>>> 118   hdd   3.64499         osd.118     up  1.00000 1.00000
>>>> 119   hdd   3.64499         osd.119     up  1.00000 1.00000
>>>> 120   hdd   3.64499         osd.120     up  1.00000 1.00000
>>>> 121   hdd   3.64499         osd.121     up  1.00000 1.00000
>>>> 122   hdd   3.64499         osd.122     up  1.00000 1.00000
>>>> 123   hdd   3.64499         osd.123     up  1.00000 1.00000
>>>> 124   hdd   3.64499         osd.124     up  1.00000 1.00000
>>>> 125   hdd   3.64499         osd.125     up  1.00000 1.00000
>>>> 126   hdd   3.64499         osd.126     up  1.00000 1.00000
>>>> 127   hdd   3.64499         osd.127     up  1.00000 1.00000
>>>> 128   hdd   3.64499         osd.128     up  1.00000 1.00000
>>>> 129   hdd   3.64499         osd.129     up  1.00000 1.00000
>>>> 130   hdd   3.64499         osd.130     up  1.00000 1.00000
>>>> 131   hdd   3.64499         osd.131     up  1.00000 1.00000
>>>> -22       116.64600     host osd005
>>>> 132   hdd   3.64499         osd.132     up  1.00000 1.00000
>>>> 133   hdd   3.64499         osd.133     up  1.00000 1.00000
>>>> 134   hdd   3.64499         osd.134     up  1.00000 1.00000
>>>> 135   hdd   3.64499         osd.135     up  1.00000 1.00000
>>>> 136   hdd   3.64499         osd.136     up  1.00000 1.00000
>>>> 137   hdd   3.64499         osd.137     up  1.00000 1.00000
>>>> 138   hdd   3.64499         osd.138     up  1.00000 1.00000
>>>> 139   hdd   3.64499         osd.139     up  1.00000 1.00000
>>>> 140   hdd   3.64499         osd.140     up  1.00000 1.00000
>>>> 141   hdd   3.64499         osd.141     up  1.00000 1.00000
>>>> 142   hdd   3.64499         osd.142     up  1.00000 1.00000
>>>> 143   hdd   3.64499         osd.143     up  1.00000 1.00000
>>>> 144   hdd   3.64499         osd.144     up  1.00000 1.00000
>>>> 145   hdd   3.64499         osd.145     up  1.00000 1.00000
>>>> 146   hdd   3.64499         osd.146     up  1.00000 1.00000
>>>> 147   hdd   3.64499         osd.147     up  1.00000 1.00000
>>>> 148   hdd   3.64499         osd.148     up  1.00000 1.00000
>>>> 149   hdd   3.64499         osd.149     up  1.00000 1.00000
>>>> 150   hdd   3.64499         osd.150     up  1.00000 1.00000
>>>> 151   hdd   3.64499         osd.151     up  1.00000 1.00000
>>>> 152   hdd   3.64499         osd.152     up  1.00000 1.00000
>>>> 153   hdd   3.64499         osd.153     up  1.00000 1.00000
>>>> 154   hdd   3.64499         osd.154     up  1.00000 1.00000
>>>> 155   hdd   3.64499         osd.155     up  1.00000 1.00000
>>>> 156   hdd   3.64499         osd.156     up  1.00000 1.00000
>>>> 157   hdd   3.64499         osd.157     up  1.00000 1.00000
>>>> 158   hdd   3.64499         osd.158     up  1.00000 1.00000
>>>> 159   hdd   3.64499         osd.159     up  1.00000 1.00000
>>>> 160   hdd   3.64499         osd.160     up  1.00000 1.00000
>>>> 161   hdd   3.64499         osd.161     up  1.00000 1.00000
>>>> 162   hdd   3.64499         osd.162     up  1.00000 1.00000
>>>> 163   hdd   3.64499         osd.163     up  1.00000 1.00000
>>>> -25       116.63335     host osd006
>>>> 164   hdd   3.64499         osd.164     up  1.00000 1.00000
>>>> 165   hdd   3.64499         osd.165     up  1.00000 1.00000
>>>> 166   hdd   3.64499         osd.166     up  1.00000 1.00000
>>>> 167   hdd   3.64499         osd.167     up  1.00000 1.00000
>>>> 168   hdd   3.64499         osd.168     up  1.00000 1.00000
>>>> 169   hdd   3.64499         osd.169     up  1.00000 1.00000
>>>> 170   hdd   3.64499         osd.170     up  1.00000 1.00000
>>>> 171   hdd   3.64499         osd.171     up  1.00000 1.00000
>>>> 172   hdd   3.64499         osd.172     up  1.00000 1.00000
>>>> 173   hdd   3.64499         osd.173     up  1.00000 1.00000
>>>> 174   hdd   3.64499         osd.174     up  1.00000 1.00000
>>>> 175   hdd   3.64499         osd.175     up  1.00000 1.00000
>>>> 176   hdd   3.64499         osd.176     up  1.00000 1.00000
>>>> 177   hdd   3.64499         osd.177     up  1.00000 1.00000
>>>> 178   hdd   3.64499         osd.178     up  1.00000 1.00000
>>>> 179   hdd   3.64499         osd.179     up  1.00000 1.00000
>>>> 180   hdd   3.64499         osd.180     up  1.00000 1.00000
>>>> 181   hdd   3.64499         osd.181     up  1.00000 1.00000
>>>> 182   hdd   3.64499         osd.182     up  1.00000 1.00000
>>>> 183   hdd   3.64499         osd.183     up  1.00000 1.00000
>>>> 184   hdd   3.64499         osd.184     up  1.00000 1.00000
>>>> 185   hdd   3.64499         osd.185     up  1.00000 1.00000
>>>> 186   hdd   3.64499         osd.186     up  1.00000 1.00000
>>>> 187   hdd   3.64499         osd.187     up  1.00000 1.00000
>>>> 188   hdd   3.64499         osd.188     up  1.00000 1.00000
>>>> 189   hdd   3.64499         osd.189     up  1.00000 1.00000
>>>> 190   hdd   3.64499         osd.190     up  1.00000 1.00000
>>>> 191   hdd   3.64499         osd.191     up  1.00000 1.00000
>>>> 192   hdd   3.63869         osd.192     up  1.00000 1.00000
>>>> 193   hdd   3.64499         osd.193     up  1.00000 1.00000
>>>> 194   hdd   3.64499         osd.194     up  1.00000 1.00000
>>>> 195   hdd   3.64499         osd.195     up  1.00000 1.00000
>>>> -28       123.71500     host osd007
>>>> 196   hdd   3.63899         osd.196     up  1.00000 1.00000
>>>> 197   hdd   3.63899         osd.197     up  1.00000 1.00000
>>>> 198   hdd   3.63899         osd.198     up  1.00000 1.00000
>>>> 199   hdd   3.63899         osd.199     up  1.00000 1.00000
>>>> 200   hdd   3.63899         osd.200     up  1.00000 1.00000
>>>> 201   hdd   3.63899         osd.201     up  1.00000 1.00000
>>>> 202   hdd   3.63899         osd.202     up  1.00000 1.00000
>>>> 203   hdd   3.63899         osd.203     up  1.00000 1.00000
>>>> 204   hdd   3.63899         osd.204     up  1.00000 1.00000
>>>> 205   hdd   3.63899         osd.205     up  1.00000 1.00000
>>>> 206   hdd   3.63899         osd.206     up  1.00000 1.00000
>>>> 207   hdd   3.63899         osd.207     up  1.00000 1.00000
>>>> 208   hdd   3.63899         osd.208     up  1.00000 1.00000
>>>> 209   hdd   3.63899         osd.209     up  1.00000 1.00000
>>>> 210   hdd   3.63899         osd.210     up  1.00000 1.00000
>>>> 211   hdd   3.63899         osd.211     up  1.00000 1.00000
>>>> 212   hdd   3.63899         osd.212     up  1.00000 1.00000
>>>> 213   hdd   3.63899         osd.213     up  1.00000 1.00000
>>>> 214   hdd   3.63899         osd.214     up  1.00000 1.00000
>>>> 215   hdd   3.63899         osd.215     up  1.00000 1.00000
>>>> 216   hdd   3.63899         osd.216     up  1.00000 1.00000
>>>> 217   hdd   3.63899         osd.217     up  1.00000 1.00000
>>>> 218   hdd   3.63899         osd.218     up  1.00000 1.00000
>>>> 219   hdd   3.63899         osd.219     up  1.00000 1.00000
>>>> 220   hdd   3.63899         osd.220     up  1.00000 1.00000
>>>> 221   hdd   3.63899         osd.221     up  1.00000 1.00000
>>>> 222   hdd   3.63899         osd.222     up  1.00000 1.00000
>>>> 223   hdd   3.63899         osd.223     up  1.00000 1.00000
>>>> 224   hdd   3.63899         osd.224     up  1.00000 1.00000
>>>> 225   hdd   3.63899         osd.225     up  1.00000 1.00000
>>>> 226   hdd   3.63899         osd.226     up  1.00000 1.00000
>>>> 227   hdd   3.63899         osd.227     up  1.00000 1.00000
>>>> 228   hdd   3.63899         osd.228     up  1.00000 1.00000
>>>> 229   hdd   3.63899         osd.229     up  1.00000 1.00000
>>>>
>>>> # ceph osd df
>>>> ID  CLASS WEIGHT  REWEIGHT SIZE  USE   AVAIL %USE  VAR  PGS
>>>>   0   ssd 0.21799  1.00000  223G 4305M  219G  1.88 0.04  64
>>>>   1   ssd 0.21799  1.00000  223G 3353M  220G  1.46 0.03  64
>>>>   2   ssd 0.21799  1.00000  223G 1836M  221G  0.80 0.02  56
>>>>   3   ssd 0.21799  1.00000  223G 2276M  221G  0.99 0.02  72
>>>> 230   ssd 0.90999  1.00000  931G 2775M  928G  0.29 0.01  60
>>>> 231   ssd 0.90999  1.00000  931G 3279M  928G  0.34 0.01  68
>>>>   4   hdd 3.64499  1.00000 3726G 1916G 1809G 51.45 1.02 111
>>>>   5   hdd 3.64499  1.00000 3726G 2144G 1581G 57.54 1.15 124
>>>>   6   hdd 3.64499  1.00000 3726G 1887G 1838G 50.66 1.01 109
>>>>   7   hdd 3.64499  1.00000 3726G 1642G 2083G 44.09 0.88  95
>>>>   8   hdd 3.64499  1.00000 3726G 1969G 1756G 52.86 1.05 114
>>>>   9   hdd 3.64499  1.00000 3726G 1577G 2148G 42.34 0.84  91
>>>>  10   hdd 3.64499  1.00000 3726G 2043G 1682G 54.85 1.09 118
>>>>  11   hdd 3.64499  1.00000 3726G 1957G 1768G 52.54 1.05 113
>>>>  12   hdd 3.64499  1.00000 3726G 1970G 1755G 52.89 1.05 114
>>>>  13   hdd 3.64499  1.00000 3726G 1676G 2049G 45.00 0.90  97
>>>>  14   hdd 3.64499  1.00000 3726G 1764G 1961G 47.35 0.94 102
>>>>  15   hdd 3.64499  1.00000 3726G 2161G 1564G 58.02 1.15 125
>>>>  16   hdd 3.64499  1.00000 3726G 1749G 1976G 46.96 0.93 101
>>>>  17   hdd 3.64499  1.00000 3726G 2022G 1703G 54.29 1.08 117
>>>>  18   hdd 3.64499  1.00000 3726G 1784G 1941G 47.89 0.95 103
>>>>  19   hdd 3.64499  1.00000 3726G 1783G 1942G 47.85 0.95 103
>>>>  20   hdd 3.64499  1.00000 3726G 1731G 1994G 46.47 0.92 100
>>>>  21   hdd 3.64499  1.00000 3726G 1939G 1786G 52.06 1.04 112
>>>>  22   hdd 3.64499  1.00000 3726G 2061G 1664G 55.34 1.10 119
>>>>  23   hdd 3.64499  1.00000 3726G 1538G 2187G 41.29 0.82  89
>>>>  24   hdd 3.64499  1.00000 3726G 2168G 1557G 58.19 1.16 125
>>>>  25   hdd 3.64499  1.00000 3726G 2023G 1702G 54.32 1.08 117
>>>>  26   hdd 3.64499  1.00000 3726G 1900G 1825G 51.00 1.02 110
>>>>  27   hdd 3.64499  1.00000 3726G 2126G 1599G 57.08 1.14 123
>>>>  28   hdd 3.64499  1.00000 3726G 1938G 1787G 52.02 1.04 112
>>>>  29   hdd 3.64499  1.00000 3726G 1663G 2062G 44.65 0.89  96
>>>>  30   hdd 3.64499  1.00000 3726G 2163G 1562G 58.07 1.16 125
>>>>  31   hdd 3.64499  1.00000 3726G 1919G 1806G 51.52 1.03 111
>>>>  32   hdd 3.64499  1.00000 3726G 1696G 2029G 45.52 0.91  98
>>>>  33   hdd 3.64499  1.00000 3726G 1438G 2287G 38.61 0.77  83
>>>>  34   hdd 3.64499  1.00000 3726G 2073G 1652G 55.64 1.11 120
>>>>  35   hdd 3.64499  1.00000 3726G 2079G 1646G 55.81 1.11 120
>>>>  36   hdd 3.64499  1.00000 3726G 1955G 1770G 52.50 1.04 113
>>>>  37   hdd 3.64499  1.00000 3726G 2180G 1545G 58.51 1.16 126
>>>>  38   hdd 3.64499  1.00000 3726G 1664G 2061G 44.66 0.89  96
>>>>  39   hdd 3.64499  1.00000 3726G 2003G 1722G 53.77 1.07 116
>>>>  40   hdd 3.64499  1.00000 3726G 1782G 1943G 47.84 0.95 103
>>>>  41   hdd 3.64499  1.00000 3726G 1903G 1822G 51.08 1.02 110
>>>>  42   hdd 3.64499  1.00000 3726G 1747G 1978G 46.89 0.93 101
>>>>  43   hdd 3.64499  1.00000 3726G 2021G 1704G 54.26 1.08 117
>>>>  44   hdd 3.64499  1.00000 3726G 1852G 1873G 49.71 0.99 107
>>>>  45   hdd 3.64499  1.00000 3726G 1522G 2203G 40.87 0.81  88
>>>>  46   hdd 3.64499  1.00000 3726G 1851G 1874G 49.69 0.99 107
>>>>  47   hdd 3.64499  1.00000 3726G 1698G 2027G 45.59 0.91  98
>>>>  48   hdd 3.64499  1.00000 3726G 1940G 1785G 52.08 1.04 112
>>>>  49   hdd 3.64499  1.00000 3726G 1746G 1979G 46.88 0.93 101
>>>>  50   hdd 3.64499  1.00000 3726G 1728G 1997G 46.40 0.92 100
>>>>  51   hdd 3.64499  1.00000 3726G 2093G 1632G 56.18 1.12 121
>>>>  52   hdd 3.64499  1.00000 3726G 2235G 1490G 60.01 1.19 129
>>>>  53   hdd 3.64499  1.00000 3726G 1871G 1854G 50.23 1.00 108
>>>>  54   hdd 3.64499  1.00000 3726G 2094G 1631G 56.22 1.12 121
>>>>  55   hdd 3.64499  1.00000 3726G 2422G 1303G 65.02 1.29 140
>>>>  56   hdd 3.64499  1.00000 3726G 1867G 1858G 50.13 1.00 108
>>>>  57   hdd 3.64499  1.00000 3726G 1729G 1996G 46.43 0.92 100
>>>>  58   hdd 3.64499  1.00000 3726G 1976G 1749G 53.05 1.06 114
>>>>  59   hdd 3.64499  1.00000 3726G 1954G 1771G 52.46 1.04 113
>>>>  60   hdd 3.64499  1.00000 3726G 1851G 1874G 49.68 0.99 107
>>>>  61   hdd 3.64499  1.00000 3726G 2042G 1683G 54.82 1.09 118
>>>>  62   hdd 3.64499  1.00000 3726G 1921G 1804G 51.58 1.03 111
>>>>  63   hdd 3.64499  1.00000 3726G 2128G 1597G 57.13 1.14 123
>>>>  64   hdd 3.64499  1.00000 3726G 2037G 1688G 54.67 1.09 118
>>>>  65   hdd 3.64499  1.00000 3726G 1970G 1755G 52.89 1.05 114
>>>>  66   hdd 3.64499  1.00000 3726G 1539G 2186G 41.32 0.82  89
>>>>  67   hdd 3.64499  1.00000 3726G 1867G 1858G 50.13 1.00 108
>>>>  68   hdd 3.64499  1.00000 3726G 1884G 1841G 50.58 1.01 109
>>>>  69   hdd 3.64499  1.00000 3726G 2092G 1633G 56.16 1.12 121
>>>>  70   hdd 3.64499  1.00000 3726G 1800G 1925G 48.33 0.96 104
>>>>  71   hdd 3.64499  1.00000 3726G 2231G 1494G 59.89 1.19 129
>>>>  72   hdd 3.64499  1.00000 3726G 1557G 2168G 41.79 0.83  90
>>>>  73   hdd 3.64499  1.00000 3726G 2300G 1425G 61.75 1.23 133
>>>>  74   hdd 3.64499  1.00000 3726G 2132G 1593G 57.24 1.14 123
>>>>  75   hdd 3.64499  1.00000 3726G 1818G 1907G 48.80 0.97 105
>>>>  76   hdd 3.64499  1.00000 3726G 2231G 1494G 59.89 1.19 129
>>>>  77   hdd 3.64499  1.00000 3726G 1749G 1976G 46.96 0.93 101
>>>>  78   hdd 3.64499  1.00000 3726G 1766G 1959G 47.41 0.94 102
>>>>  79   hdd 3.64499  1.00000 3726G 1678G 2047G 45.04 0.90  97
>>>>  80   hdd 3.63899  1.00000 3726G 1744G 1981G 46.82 0.93 102
>>>>  81   hdd 3.64499  1.00000 3726G 1884G 1841G 50.58 1.01 109
>>>>  82   hdd 3.64499  1.00000 3726G 1711G 2014G 45.93 0.91  99
>>>>  83   hdd 3.64499  1.00000 3726G 1728G 1997G 46.39 0.92 100
>>>>  84   hdd 3.64499  1.00000 3726G 1834G 1891G 49.22 0.98 106
>>>>  85   hdd 3.64499  1.00000 3726G 1709G 2016G 45.88 0.91  99
>>>>  86   hdd 3.64499  1.00000 3726G 1938G 1787G 52.03 1.04 112
>>>>  87   hdd 3.64499  1.00000 3726G 2131G 1594G 57.19 1.14 123
>>>>  88   hdd 3.64499  1.00000 3726G 1559G 2166G 41.85 0.83  90
>>>>  89   hdd 3.64499  1.00000 3726G 1645G 2080G 44.16 0.88  95
>>>>  90   hdd 3.64499  1.00000 3726G 2045G 1680G 54.89 1.09 118
>>>>  91   hdd 3.64499  1.00000 3726G 2164G 1561G 58.09 1.16 125
>>>>  92   hdd 3.64499  1.00000 3726G 1922G 1803G 51.59 1.03 111
>>>>  93   hdd 3.64499  1.00000 3726G 1938G 1787G 52.03 1.04 112
>>>>  94   hdd 3.64499  1.00000 3726G 1971G 1754G 52.91 1.05 114
>>>>  95   hdd 3.64499  1.00000 3726G 1917G 1808G 51.47 1.02 111
>>>>  96   hdd 3.64499  1.00000 3726G 1814G 1911G 48.70 0.97 105
>>>>  97   hdd 3.64499  1.00000 3726G 1660G 2065G 44.57 0.89  96
>>>>  98   hdd 3.64499  1.00000 3726G 1904G 1821G 51.10 1.02 110
>>>>  99   hdd 3.64499  1.00000 3726G 2112G 1613G 56.70 1.13 122
>>>> 100   hdd 3.64499  1.00000 3726G 1869G 1856G 50.18 1.00 108
>>>> 101   hdd 3.64499  1.00000 3726G 2005G 1720G 53.84 1.07 116
>>>> 102   hdd 3.64499  1.00000 3726G 2039G 1686G 54.73 1.09 118
>>>> 103   hdd 3.64499  1.00000 3726G 2093G 1632G 56.20 1.12 121
>>>> 104   hdd 3.64499  1.00000 3726G 1884G 1841G 50.58 1.01 109
>>>> 105   hdd 3.64499  1.00000 3726G 1867G 1858G 50.13 1.00 108
>>>> 106   hdd 3.64499  1.00000 3726G 1956G 1769G 52.51 1.05 113
>>>> 107   hdd 3.64499  1.00000 3726G 2077G 1648G 55.74 1.11 120
>>>> 108   hdd 3.64499  1.00000 3726G 1990G 1735G 53.43 1.06 115
>>>> 109   hdd 3.64499  1.00000 3726G 1781G 1944G 47.80 0.95 103
>>>> 110   hdd 3.64499  1.00000 3726G 1869G 1856G 50.17 1.00 108
>>>> 111   hdd 3.64499  1.00000 3726G 2023G 1702G 54.30 1.08 117
>>>> 112   hdd 3.64499  1.00000 3726G 2007G 1718G 53.89 1.07 116
>>>> 113   hdd 3.64499  1.00000 3726G 1851G 1874G 49.70 0.99 107
>>>> 114   hdd 3.64499  1.00000 3726G 1832G 1893G 49.18 0.98 106
>>>> 115   hdd 3.64499  1.00000 3726G 2098G 1627G 56.32 1.12 121
>>>> 116   hdd 3.64499  1.00000 3726G 1866G 1859G 50.10 1.00 108
>>>> 117   hdd 3.64499  1.00000 3726G 1764G 1961G 47.36 0.94 102
>>>> 118   hdd 3.64499  1.00000 3726G 1816G 1909G 48.76 0.97 105
>>>> 119   hdd 3.64499  1.00000 3726G 1733G 1992G 46.53 0.93 100
>>>> 120   hdd 3.64499  1.00000 3726G 1992G 1733G 53.48 1.06 115
>>>> 121   hdd 3.64499  1.00000 3726G 1657G 2068G 44.48 0.89  96
>>>> 122   hdd 3.64499  1.00000 3726G 1715G 2010G 46.04 0.92  99
>>>> 123   hdd 3.64499  1.00000 3726G 1974G 1751G 53.00 1.05 114
>>>> 124   hdd 3.64499  1.00000 3726G 1731G 1994G 46.48 0.93 100
>>>> 125   hdd 3.64499  1.00000 3726G 1897G 1828G 50.94 1.01 110
>>>> 126   hdd 3.64499  1.00000 3726G 1990G 1735G 53.42 1.06 115
>>>> 127   hdd 3.64499  1.00000 3726G 1972G 1753G 52.93 1.05 114
>>>> 128   hdd 3.64499  1.00000 3726G 1888G 1837G 50.68 1.01 109
>>>> 129   hdd 3.64499  1.00000 3726G 2025G 1700G 54.35 1.08 117
>>>> 130   hdd 3.64499  1.00000 3726G 1747G 1978G 46.90 0.93 101
>>>> 131   hdd 3.64499  1.00000 3726G 2061G 1664G 55.32 1.10 119
>>>> 132   hdd 3.64499  1.00000 3726G 2185G 1540G 58.65 1.17 126
>>>> 133   hdd 3.64499  1.00000 3726G 1764G 1961G 47.35 0.94 102
>>>> 134   hdd 3.64499  1.00000 3726G 1782G 1943G 47.84 0.95 103
>>>> 135   hdd 3.64499  1.00000 3726G 1675G 2050G 44.96 0.89  97
>>>> 136   hdd 3.64499  1.00000 3726G 1889G 1836G 50.70 1.01 109
>>>> 137   hdd 3.64499  1.00000 3726G 1954G 1771G 52.45 1.04 113
>>>> 138   hdd 3.64499  1.00000 3726G 1694G 2031G 45.49 0.91  98
>>>> 139   hdd 3.64499  1.00000 3726G 2114G 1611G 56.76 1.13 122
>>>> 140   hdd 3.64499  1.00000 3726G 1726G 1999G 46.35 0.92 100
>>>> 141   hdd 3.64499  1.00000 3726G 1988G 1737G 53.36 1.06 115
>>>> 142   hdd 3.64499  1.00000 3726G 2181G 1544G 58.55 1.17 126
>>>> 143   hdd 3.64499  1.00000 3726G 1853G 1872G 49.74 0.99 107
>>>> 144   hdd 3.64499  1.00000 3726G 2144G 1581G 57.56 1.15 124
>>>> 145   hdd 3.64499  1.00000 3726G 1868G 1857G 50.14 1.00 108
>>>> 146   hdd 3.64499  1.00000 3726G 2078G 1647G 55.77 1.11 120
>>>> 147   hdd 3.64499  1.00000 3726G 1992G 1733G 53.49 1.06 115
>>>> 148   hdd 3.64499  1.00000 3726G 1728G 1997G 46.39 0.92 100
>>>> 149   hdd 3.64499  1.00000 3726G 2162G 1563G 58.03 1.15 125
>>>> 150   hdd 3.64499  1.00000 3726G 2005G 1720G 53.83 1.07 116
>>>> 151   hdd 3.64499  1.00000 3726G 1918G 1807G 51.49 1.02 111
>>>> 152   hdd 3.64499  1.00000 3726G 1921G 1804G 51.57 1.03 111
>>>> 153   hdd 3.64499  1.00000 3726G 1783G 1942G 47.86 0.95 103
>>>> 154   hdd 3.64499  1.00000 3726G 2005G 1720G 53.82 1.07 116
>>>> 155   hdd 3.64499  1.00000 3726G 1853G 1873G 49.73 0.99 107
>>>> 156   hdd 3.64499  1.00000 3726G 1626G 2099G 43.65 0.87  94
>>>> 157   hdd 3.64499  1.00000 3726G 1802G 1923G 48.38 0.96 104
>>>> 158   hdd 3.64499  1.00000 3726G 1765G 1960G 47.39 0.94 102
>>>> 159   hdd 3.64499  1.00000 3726G 1990G 1735G 53.42 1.06 115
>>>> 160   hdd 3.64499  1.00000 3726G 1901G 1824G 51.02 1.02 110
>>>> 161   hdd 3.64499  1.00000 3726G 1906G 1819G 51.16 1.02 110
>>>> 162   hdd 3.64499  1.00000 3726G 1763G 1962G 47.33 0.94 102
>>>> 163   hdd 3.64499  1.00000 3726G 1800G 1925G 48.33 0.96 104
>>>> 164   hdd 3.64499  1.00000 3726G 1922G 1803G 51.61 1.03 111
>>>> 165   hdd 3.64499  1.00000 3726G 1731G 1994G 46.46 0.92 100
>>>> 166   hdd 3.64499  1.00000 3726G 2027G 1699G 54.40 1.08 117
>>>> 167   hdd 3.64499  1.00000 3726G 1956G 1769G 52.50 1.04 113
>>>> 168   hdd 3.64499  1.00000 3726G 1853G 1872G 49.74 0.99 107
>>>> 169   hdd 3.64499  1.00000 3726G 1884G 1841G 50.59 1.01 109
>>>> 170   hdd 3.64499  1.00000 3726G 1992G 1733G 53.48 1.06 115
>>>> 171   hdd 3.64499  1.00000 3726G 1898G 1827G 50.96 1.01 110
>>>> 172   hdd 3.64499  1.00000 3726G 1781G 1944G 47.82 0.95 103
>>>> 173   hdd 3.64499  1.00000 3726G 1628G 2097G 43.71 0.87  94
>>>> 174   hdd 3.64499  1.00000 3726G 1992G 1733G 53.48 1.06 115
>>>> 175   hdd 3.64499  1.00000 3726G 1905G 1820G 51.15 1.02 110
>>>> 176   hdd 3.64499  1.00000 3726G 1709G 2016G 45.89 0.91  99
>>>> 177   hdd 3.64499  1.00000 3726G 1991G 1734G 53.44 1.06 115
>>>> 178   hdd 3.64499  1.00000 3726G 1973G 1752G 52.97 1.05 114
>>>> 179   hdd 3.64499  1.00000 3726G 1574G 2151G 42.26 0.84  91
>>>> 180   hdd 3.64499  1.00000 3726G 1917G 1808G 51.46 1.02 111
>>>> 181   hdd 3.64499  1.00000 3726G 1937G 1788G 52.00 1.03 112
>>>> 182   hdd 3.64499  1.00000 3726G 1989G 1736G 53.40 1.06 115
>>>> 183   hdd 3.64499  1.00000 3726G 2039G 1686G 54.74 1.09 118
>>>> 184   hdd 3.64499  1.00000 3726G 1868G 1857G 50.16 1.00 108
>>>> 185   hdd 3.64499  1.00000 3726G 1954G 1771G 52.45 1.04 113
>>>> 186   hdd 3.64499  1.00000 3726G 1850G 1875G 49.66 0.99 107
>>>> 187   hdd 3.64499  1.00000 3726G 1851G 1874G 49.68 0.99 107
>>>> 188   hdd 3.64499  1.00000 3726G 1814G 1911G 48.69 0.97 105
>>>> 189   hdd 3.64499  1.00000 3726G 2286G 1439G 61.36 1.22 132
>>>> 190   hdd 3.64499  1.00000 3726G 2060G 1665G 55.31 1.10 119
>>>> 191   hdd 3.64499  1.00000 3726G 1836G 1889G 49.29 0.98 106
>>>> 192   hdd 3.63869  1.00000 3726G 1721G 2004G 46.21 0.92 101
>>>> 193   hdd 3.64499  1.00000 3726G 1953G 1772G 52.43 1.04 113
>>>> 194   hdd 3.64499  1.00000 3726G 1541G 2184G 41.37 0.82  89
>>>> 195   hdd 3.64499  1.00000 3726G 1837G 1888G 49.32 0.98 106
>>>> 196   hdd 3.63899  1.00000 3726G 1705G 2020G 45.79 0.91 100
>>>> 197   hdd 3.63899  1.00000 3726G 1743G 1982G 46.78 0.93 102
>>>> 198   hdd 3.63899  1.00000 3726G 1982G 1743G 53.21 1.06 116
>>>> 199   hdd 3.63899  1.00000 3726G 1382G 2343G 37.11 0.74  81
>>>> 200   hdd 3.63899  1.00000 3726G 1893G 1832G 50.83 1.01 111
>>>> 201   hdd 3.63899  1.00000 3726G 1845G 1880G 49.54 0.99 108
>>>> 202   hdd 3.63899  1.00000 3726G 1658G 2067G 44.51 0.89  97
>>>> 203   hdd 3.63899  1.00000 3726G 2066G 1659G 55.47 1.10 121
>>>> 204   hdd 3.63899  1.00000 3726G 1672G 2053G 44.90 0.89  98
>>>> 205   hdd 3.63899  1.00000 3726G 1521G 2204G 40.84 0.81  89
>>>> 206   hdd 3.63899  1.00000 3726G 1796G 1929G 48.22 0.96 105
>>>> 207   hdd 3.63899  1.00000 3726G 1555G 2170G 41.75 0.83  91
>>>> 208   hdd 3.63899  1.00000 3726G 1742G 1983G 46.77 0.93 102
>>>> 209   hdd 3.63899  1.00000 3726G 2016G 1709G 54.11 1.08 118
>>>> 210   hdd 3.63899  1.00000 3726G 2136G 1589G 57.34 1.14 125
>>>> 211   hdd 3.63899  1.00000 3726G 1656G 2069G 44.46 0.88  97
>>>> 212   hdd 3.63899  1.00000 3726G 1553G 2172G 41.71 0.83  91
>>>> 213   hdd 3.63899  1.00000 3726G 1758G 1967G 47.19 0.94 103
>>>> 214   hdd 3.63899  1.00000 3726G 1603G 2122G 43.03 0.86  94
>>>> 215   hdd 3.63899  1.00000 3726G 1809G 1916G 48.57 0.97 106
>>>> 216   hdd 3.63899  1.00000 3726G 1760G 1965G 47.26 0.94 103
>>>> 217   hdd 3.63899  1.00000 3726G 1742G 1983G 46.77 0.93 102
>>>> 218   hdd 3.63899  1.00000 3726G 1880G 1845G 50.46 1.00 110
>>>> 219   hdd 3.63899  1.00000 3726G 1693G 2032G 45.45 0.90  99
>>>> 220   hdd 3.63899  1.00000 3726G 2032G 1693G 54.56 1.09 119
>>>> 221   hdd 3.63899  1.00000 3726G 1842G 1883G 49.44 0.98 108
>>>> 222   hdd 3.63899  1.00000 3726G 1521G 2204G 40.84 0.81  89
>>>> 223   hdd 3.63899  1.00000 3726G 1470G 2255G 39.46 0.79  86
>>>> 224   hdd 3.63899  1.00000 3726G 1808G 1917G 48.53 0.97 106
>>>> 225   hdd 3.63899  1.00000 3726G 1893G 1832G 50.83 1.01 111
>>>> 226   hdd 3.63899  1.00000 3726G 1776G 1949G 47.69 0.95 104
>>>> 227   hdd 3.63899  1.00000 3726G 1929G 1796G 51.78 1.03 113
>>>> 228   hdd 3.63899  1.00000 3726G 1657G 2068G 44.48 0.89  97
>>>> 229   hdd 3.63899  1.00000 3726G 1843G 1882G 49.47 0.98 108
>>>>                      TOTAL  825T  414T  410T 50.24
>>>> MIN/MAX VAR: 0.01/1.29  STDDEV: 9.22
>>>> --------------------------------------------------------------
>>>>
>>>> Am 20.10.18 um 20:35 schrieb Janne Johansson:
>>>>> Yes, if you have uneven sizes I guess you could end up in a situation
>>>>> where you have
>>>>> lots of 1TB OSDs and a number of 2TB OSD but pool replication forces
>>>>> the pool to have one
>>>>> PG replica on the 1TB OSD, then it would be possible to state "this
>>>>> pool cant write more than X G"
>>>>> but when it is full, there would be free space left on some of the
>>>>> 2TB-OSDs, but which the pool
>>>>> cant utilize. Probably same for uneven OSD hosts if you have those.
>>>>>
>>>>> Den lör 20 okt. 2018 kl 20:28 skrev Oliver Freyermuth
>>>>> <freyermuth at physik.uni-bonn.de>:
>>>>>>
>>>>>> Dear Janne,
>>>>>>
>>>>>> yes, of course. But since we only have two pools here, this can not explain the difference.
>>>>>> The metadata is replicated (3 copies) across ssd drives, and we have < 3 TB of total raw storage for that.
>>>>>> So looking at the raw space usage, we can ignore that.
>>>>>>
>>>>>> All the rest is used for the ceph_data pool. So the ceph_data pool, in terms of raw storage, is about 50 % used.
>>>>>>
>>>>>> But in terms of storage shown for that pool, it's almost 63 % %USED.
>>>>>> So I guess this can purely be from bad balancing, correct?
>>>>>>
>>>>>> Cheers,
>>>>>>         Oliver
>>>>>>
>>>>>> Am 20.10.18 um 19:49 schrieb Janne Johansson:
>>>>>>> Do mind that drives may have more than one pool on them, so RAW space
>>>>>>> is what it says, how much free space there is. Then the avail and
>>>>>>> %USED on per-pool stats will take replication into account, it can
>>>>>>> tell how much data you may write into that particular pool, given that
>>>>>>> pools replication or EC settings.
>>>>>>>
>>>>>>> Den lör 20 okt. 2018 kl 19:09 skrev Oliver Freyermuth
>>>>>>> <freyermuth at physik.uni-bonn.de>:
>>>>>>>>
>>>>>>>> Dear Cephalopodians,
>>>>>>>>
>>>>>>>> as many others, I'm also a bit confused by "ceph df" output
>>>>>>>> in a pretty straightforward configuration.
>>>>>>>>
>>>>>>>> We have a CephFS (12.2.7) running, with 4+2 EC profile.
>>>>>>>>
>>>>>>>> I get:
>>>>>>>> ----------------------------------------------------------------------------
>>>>>>>> # ceph df
>>>>>>>> GLOBAL:
>>>>>>>>     SIZE     AVAIL     RAW USED     %RAW USED
>>>>>>>>     824T      410T         414T         50.26
>>>>>>>> POOLS:
>>>>>>>>     NAME                ID     USED     %USED     MAX AVAIL     OBJECTS
>>>>>>>>     cephfs_metadata     1      452M      0.05          860G       365774
>>>>>>>>     cephfs_data         2      275T     62.68          164T     75056403
>>>>>>>> ----------------------------------------------------------------------------
>>>>>>>>
>>>>>>>> So about 50 % of raw space are used, but already ~63 % of filesystem space are used.
>>>>>>>> Is this purely from imperfect balancing?
>>>>>>>> In "ceph osd df", I do indeed see OSD usages spreading from 65.02 % usage down to 37.12 %.
>>>>>>>>
>>>>>>>> We did not yet use the balancer plugin.
>>>>>>>> We don't have any pre-luminous clients.
>>>>>>>> In that setup, I take it that "upmap" mode would be recommended - correct?
>>>>>>>> Any "gotchas" using that on luminous?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>         Oliver
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> ceph-users mailing list
>>>>>>>> ceph-users at lists.ceph.com
>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>> 
>>> 
>> 
>> 
> 
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5455 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181027/c6221df0/attachment.bin>


More information about the ceph-users mailing list