[ceph-users] ceph df space usage confusion - balancing needed?

Oliver Freyermuth freyermuth at physik.uni-bonn.de
Sat Oct 20 12:57:49 PDT 2018


Ok, I'll try out the balancer end of the upcoming week then (after we've fixed a HW-issue with one of our mons
and the cooling system). 

Until then, any further advice and whether upmap is recommended over crush-compat (all clients are Luminous) are welcome ;-). 

Cheers,
Oliver

Am 20.10.18 um 21:26 schrieb Janne Johansson:
> Ok, can't say "why" then, I'd reweigh them somewhat to even it out,
> 1.22 -vs- 0.74 in variance is a lot, so either a balancer plugin for
> the MGRs, a script or just a few manual tweaks might be in order.
> 
> Den lör 20 okt. 2018 kl 21:02 skrev Oliver Freyermuth
> <freyermuth at physik.uni-bonn.de>:
>>
>> All OSDs are of the very same size. One OSD host has slightly more disks (33 instead of 31), though.
>> So also that that can't explain the hefty difference.
>>
>> I attach the output of "ceph osd tree" and "ceph osd df".
>>
>> The crush rule for the ceph_data pool is:
>> rule cephfs_data {
>>         id 2
>>         type erasure
>>         min_size 3
>>         max_size 6
>>         step set_chooseleaf_tries 5
>>         step set_choose_tries 100
>>         step take default class hdd
>>         step chooseleaf indep 0 type host
>>         step emit
>> }
>> So that only considers the hdd device class. EC is done with k=4 m=2.
>>
>> So I don't see any imbalance on the hardware level, but only a somewhat uneven distribution of PGs.
>> Am I missing something, or is this really just a case for the ceph balancer plugin?
>> I'm just a bit astonished this effect is so huge.
>> Maybe our 4096 PGs for the ceph_data pool are not enough to get an even distribution without balancing?
>> But it yields about 100 PGs per OSD, as you can see...
>>
>> --------------------------------------------------------------
>> # ceph osd tree
>> ID  CLASS WEIGHT    TYPE NAME       STATUS REWEIGHT PRI-AFF
>>  -1       826.26428 root default
>>  -3         0.43700     host mon001
>>   0   ssd   0.21799         osd.0       up  1.00000 1.00000
>>   1   ssd   0.21799         osd.1       up  1.00000 1.00000
>>  -5         0.43700     host mon002
>>   2   ssd   0.21799         osd.2       up  1.00000 1.00000
>>   3   ssd   0.21799         osd.3       up  1.00000 1.00000
>> -31         1.81898     host mon003
>> 230   ssd   0.90999         osd.230     up  1.00000 1.00000
>> 231   ssd   0.90999         osd.231     up  1.00000 1.00000
>> -10       116.64600     host osd001
>>   4   hdd   3.64499         osd.4       up  1.00000 1.00000
>>   5   hdd   3.64499         osd.5       up  1.00000 1.00000
>>   6   hdd   3.64499         osd.6       up  1.00000 1.00000
>>   7   hdd   3.64499         osd.7       up  1.00000 1.00000
>>   8   hdd   3.64499         osd.8       up  1.00000 1.00000
>>   9   hdd   3.64499         osd.9       up  1.00000 1.00000
>>  10   hdd   3.64499         osd.10      up  1.00000 1.00000
>>  11   hdd   3.64499         osd.11      up  1.00000 1.00000
>>  12   hdd   3.64499         osd.12      up  1.00000 1.00000
>>  13   hdd   3.64499         osd.13      up  1.00000 1.00000
>>  14   hdd   3.64499         osd.14      up  1.00000 1.00000
>>  15   hdd   3.64499         osd.15      up  1.00000 1.00000
>>  16   hdd   3.64499         osd.16      up  1.00000 1.00000
>>  17   hdd   3.64499         osd.17      up  1.00000 1.00000
>>  18   hdd   3.64499         osd.18      up  1.00000 1.00000
>>  19   hdd   3.64499         osd.19      up  1.00000 1.00000
>>  20   hdd   3.64499         osd.20      up  1.00000 1.00000
>>  21   hdd   3.64499         osd.21      up  1.00000 1.00000
>>  22   hdd   3.64499         osd.22      up  1.00000 1.00000
>>  23   hdd   3.64499         osd.23      up  1.00000 1.00000
>>  24   hdd   3.64499         osd.24      up  1.00000 1.00000
>>  25   hdd   3.64499         osd.25      up  1.00000 1.00000
>>  26   hdd   3.64499         osd.26      up  1.00000 1.00000
>>  27   hdd   3.64499         osd.27      up  1.00000 1.00000
>>  28   hdd   3.64499         osd.28      up  1.00000 1.00000
>>  29   hdd   3.64499         osd.29      up  1.00000 1.00000
>>  30   hdd   3.64499         osd.30      up  1.00000 1.00000
>>  31   hdd   3.64499         osd.31      up  1.00000 1.00000
>>  32   hdd   3.64499         osd.32      up  1.00000 1.00000
>>  33   hdd   3.64499         osd.33      up  1.00000 1.00000
>>  34   hdd   3.64499         osd.34      up  1.00000 1.00000
>>  35   hdd   3.64499         osd.35      up  1.00000 1.00000
>> -13       116.64600     host osd002
>>  36   hdd   3.64499         osd.36      up  1.00000 1.00000
>>  37   hdd   3.64499         osd.37      up  1.00000 1.00000
>>  38   hdd   3.64499         osd.38      up  1.00000 1.00000
>>  39   hdd   3.64499         osd.39      up  1.00000 1.00000
>>  40   hdd   3.64499         osd.40      up  1.00000 1.00000
>>  41   hdd   3.64499         osd.41      up  1.00000 1.00000
>>  42   hdd   3.64499         osd.42      up  1.00000 1.00000
>>  43   hdd   3.64499         osd.43      up  1.00000 1.00000
>>  44   hdd   3.64499         osd.44      up  1.00000 1.00000
>>  45   hdd   3.64499         osd.45      up  1.00000 1.00000
>>  46   hdd   3.64499         osd.46      up  1.00000 1.00000
>>  47   hdd   3.64499         osd.47      up  1.00000 1.00000
>>  48   hdd   3.64499         osd.48      up  1.00000 1.00000
>>  49   hdd   3.64499         osd.49      up  1.00000 1.00000
>>  50   hdd   3.64499         osd.50      up  1.00000 1.00000
>>  51   hdd   3.64499         osd.51      up  1.00000 1.00000
>>  52   hdd   3.64499         osd.52      up  1.00000 1.00000
>>  53   hdd   3.64499         osd.53      up  1.00000 1.00000
>>  54   hdd   3.64499         osd.54      up  1.00000 1.00000
>>  55   hdd   3.64499         osd.55      up  1.00000 1.00000
>>  56   hdd   3.64499         osd.56      up  1.00000 1.00000
>>  57   hdd   3.64499         osd.57      up  1.00000 1.00000
>>  58   hdd   3.64499         osd.58      up  1.00000 1.00000
>>  59   hdd   3.64499         osd.59      up  1.00000 1.00000
>>  60   hdd   3.64499         osd.60      up  1.00000 1.00000
>>  61   hdd   3.64499         osd.61      up  1.00000 1.00000
>>  62   hdd   3.64499         osd.62      up  1.00000 1.00000
>>  63   hdd   3.64499         osd.63      up  1.00000 1.00000
>>  64   hdd   3.64499         osd.64      up  1.00000 1.00000
>>  65   hdd   3.64499         osd.65      up  1.00000 1.00000
>>  66   hdd   3.64499         osd.66      up  1.00000 1.00000
>>  67   hdd   3.64499         osd.67      up  1.00000 1.00000
>> -16       116.63899     host osd003
>>  68   hdd   3.64499         osd.68      up  1.00000 1.00000
>>  69   hdd   3.64499         osd.69      up  1.00000 1.00000
>>  70   hdd   3.64499         osd.70      up  1.00000 1.00000
>>  71   hdd   3.64499         osd.71      up  1.00000 1.00000
>>  72   hdd   3.64499         osd.72      up  1.00000 1.00000
>>  73   hdd   3.64499         osd.73      up  1.00000 1.00000
>>  74   hdd   3.64499         osd.74      up  1.00000 1.00000
>>  75   hdd   3.64499         osd.75      up  1.00000 1.00000
>>  76   hdd   3.64499         osd.76      up  1.00000 1.00000
>>  77   hdd   3.64499         osd.77      up  1.00000 1.00000
>>  78   hdd   3.64499         osd.78      up  1.00000 1.00000
>>  79   hdd   3.64499         osd.79      up  1.00000 1.00000
>>  80   hdd   3.63899         osd.80      up  1.00000 1.00000
>>  81   hdd   3.64499         osd.81      up  1.00000 1.00000
>>  82   hdd   3.64499         osd.82      up  1.00000 1.00000
>>  83   hdd   3.64499         osd.83      up  1.00000 1.00000
>>  84   hdd   3.64499         osd.84      up  1.00000 1.00000
>>  85   hdd   3.64499         osd.85      up  1.00000 1.00000
>>  86   hdd   3.64499         osd.86      up  1.00000 1.00000
>>  87   hdd   3.64499         osd.87      up  1.00000 1.00000
>>  88   hdd   3.64499         osd.88      up  1.00000 1.00000
>>  89   hdd   3.64499         osd.89      up  1.00000 1.00000
>>  90   hdd   3.64499         osd.90      up  1.00000 1.00000
>>  91   hdd   3.64499         osd.91      up  1.00000 1.00000
>>  92   hdd   3.64499         osd.92      up  1.00000 1.00000
>>  93   hdd   3.64499         osd.93      up  1.00000 1.00000
>>  94   hdd   3.64499         osd.94      up  1.00000 1.00000
>>  95   hdd   3.64499         osd.95      up  1.00000 1.00000
>>  96   hdd   3.64499         osd.96      up  1.00000 1.00000
>>  97   hdd   3.64499         osd.97      up  1.00000 1.00000
>>  98   hdd   3.64499         osd.98      up  1.00000 1.00000
>>  99   hdd   3.64499         osd.99      up  1.00000 1.00000
>> -19       116.64600     host osd004
>> 100   hdd   3.64499         osd.100     up  1.00000 1.00000
>> 101   hdd   3.64499         osd.101     up  1.00000 1.00000
>> 102   hdd   3.64499         osd.102     up  1.00000 1.00000
>> 103   hdd   3.64499         osd.103     up  1.00000 1.00000
>> 104   hdd   3.64499         osd.104     up  1.00000 1.00000
>> 105   hdd   3.64499         osd.105     up  1.00000 1.00000
>> 106   hdd   3.64499         osd.106     up  1.00000 1.00000
>> 107   hdd   3.64499         osd.107     up  1.00000 1.00000
>> 108   hdd   3.64499         osd.108     up  1.00000 1.00000
>> 109   hdd   3.64499         osd.109     up  1.00000 1.00000
>> 110   hdd   3.64499         osd.110     up  1.00000 1.00000
>> 111   hdd   3.64499         osd.111     up  1.00000 1.00000
>> 112   hdd   3.64499         osd.112     up  1.00000 1.00000
>> 113   hdd   3.64499         osd.113     up  1.00000 1.00000
>> 114   hdd   3.64499         osd.114     up  1.00000 1.00000
>> 115   hdd   3.64499         osd.115     up  1.00000 1.00000
>> 116   hdd   3.64499         osd.116     up  1.00000 1.00000
>> 117   hdd   3.64499         osd.117     up  1.00000 1.00000
>> 118   hdd   3.64499         osd.118     up  1.00000 1.00000
>> 119   hdd   3.64499         osd.119     up  1.00000 1.00000
>> 120   hdd   3.64499         osd.120     up  1.00000 1.00000
>> 121   hdd   3.64499         osd.121     up  1.00000 1.00000
>> 122   hdd   3.64499         osd.122     up  1.00000 1.00000
>> 123   hdd   3.64499         osd.123     up  1.00000 1.00000
>> 124   hdd   3.64499         osd.124     up  1.00000 1.00000
>> 125   hdd   3.64499         osd.125     up  1.00000 1.00000
>> 126   hdd   3.64499         osd.126     up  1.00000 1.00000
>> 127   hdd   3.64499         osd.127     up  1.00000 1.00000
>> 128   hdd   3.64499         osd.128     up  1.00000 1.00000
>> 129   hdd   3.64499         osd.129     up  1.00000 1.00000
>> 130   hdd   3.64499         osd.130     up  1.00000 1.00000
>> 131   hdd   3.64499         osd.131     up  1.00000 1.00000
>> -22       116.64600     host osd005
>> 132   hdd   3.64499         osd.132     up  1.00000 1.00000
>> 133   hdd   3.64499         osd.133     up  1.00000 1.00000
>> 134   hdd   3.64499         osd.134     up  1.00000 1.00000
>> 135   hdd   3.64499         osd.135     up  1.00000 1.00000
>> 136   hdd   3.64499         osd.136     up  1.00000 1.00000
>> 137   hdd   3.64499         osd.137     up  1.00000 1.00000
>> 138   hdd   3.64499         osd.138     up  1.00000 1.00000
>> 139   hdd   3.64499         osd.139     up  1.00000 1.00000
>> 140   hdd   3.64499         osd.140     up  1.00000 1.00000
>> 141   hdd   3.64499         osd.141     up  1.00000 1.00000
>> 142   hdd   3.64499         osd.142     up  1.00000 1.00000
>> 143   hdd   3.64499         osd.143     up  1.00000 1.00000
>> 144   hdd   3.64499         osd.144     up  1.00000 1.00000
>> 145   hdd   3.64499         osd.145     up  1.00000 1.00000
>> 146   hdd   3.64499         osd.146     up  1.00000 1.00000
>> 147   hdd   3.64499         osd.147     up  1.00000 1.00000
>> 148   hdd   3.64499         osd.148     up  1.00000 1.00000
>> 149   hdd   3.64499         osd.149     up  1.00000 1.00000
>> 150   hdd   3.64499         osd.150     up  1.00000 1.00000
>> 151   hdd   3.64499         osd.151     up  1.00000 1.00000
>> 152   hdd   3.64499         osd.152     up  1.00000 1.00000
>> 153   hdd   3.64499         osd.153     up  1.00000 1.00000
>> 154   hdd   3.64499         osd.154     up  1.00000 1.00000
>> 155   hdd   3.64499         osd.155     up  1.00000 1.00000
>> 156   hdd   3.64499         osd.156     up  1.00000 1.00000
>> 157   hdd   3.64499         osd.157     up  1.00000 1.00000
>> 158   hdd   3.64499         osd.158     up  1.00000 1.00000
>> 159   hdd   3.64499         osd.159     up  1.00000 1.00000
>> 160   hdd   3.64499         osd.160     up  1.00000 1.00000
>> 161   hdd   3.64499         osd.161     up  1.00000 1.00000
>> 162   hdd   3.64499         osd.162     up  1.00000 1.00000
>> 163   hdd   3.64499         osd.163     up  1.00000 1.00000
>> -25       116.63335     host osd006
>> 164   hdd   3.64499         osd.164     up  1.00000 1.00000
>> 165   hdd   3.64499         osd.165     up  1.00000 1.00000
>> 166   hdd   3.64499         osd.166     up  1.00000 1.00000
>> 167   hdd   3.64499         osd.167     up  1.00000 1.00000
>> 168   hdd   3.64499         osd.168     up  1.00000 1.00000
>> 169   hdd   3.64499         osd.169     up  1.00000 1.00000
>> 170   hdd   3.64499         osd.170     up  1.00000 1.00000
>> 171   hdd   3.64499         osd.171     up  1.00000 1.00000
>> 172   hdd   3.64499         osd.172     up  1.00000 1.00000
>> 173   hdd   3.64499         osd.173     up  1.00000 1.00000
>> 174   hdd   3.64499         osd.174     up  1.00000 1.00000
>> 175   hdd   3.64499         osd.175     up  1.00000 1.00000
>> 176   hdd   3.64499         osd.176     up  1.00000 1.00000
>> 177   hdd   3.64499         osd.177     up  1.00000 1.00000
>> 178   hdd   3.64499         osd.178     up  1.00000 1.00000
>> 179   hdd   3.64499         osd.179     up  1.00000 1.00000
>> 180   hdd   3.64499         osd.180     up  1.00000 1.00000
>> 181   hdd   3.64499         osd.181     up  1.00000 1.00000
>> 182   hdd   3.64499         osd.182     up  1.00000 1.00000
>> 183   hdd   3.64499         osd.183     up  1.00000 1.00000
>> 184   hdd   3.64499         osd.184     up  1.00000 1.00000
>> 185   hdd   3.64499         osd.185     up  1.00000 1.00000
>> 186   hdd   3.64499         osd.186     up  1.00000 1.00000
>> 187   hdd   3.64499         osd.187     up  1.00000 1.00000
>> 188   hdd   3.64499         osd.188     up  1.00000 1.00000
>> 189   hdd   3.64499         osd.189     up  1.00000 1.00000
>> 190   hdd   3.64499         osd.190     up  1.00000 1.00000
>> 191   hdd   3.64499         osd.191     up  1.00000 1.00000
>> 192   hdd   3.63869         osd.192     up  1.00000 1.00000
>> 193   hdd   3.64499         osd.193     up  1.00000 1.00000
>> 194   hdd   3.64499         osd.194     up  1.00000 1.00000
>> 195   hdd   3.64499         osd.195     up  1.00000 1.00000
>> -28       123.71500     host osd007
>> 196   hdd   3.63899         osd.196     up  1.00000 1.00000
>> 197   hdd   3.63899         osd.197     up  1.00000 1.00000
>> 198   hdd   3.63899         osd.198     up  1.00000 1.00000
>> 199   hdd   3.63899         osd.199     up  1.00000 1.00000
>> 200   hdd   3.63899         osd.200     up  1.00000 1.00000
>> 201   hdd   3.63899         osd.201     up  1.00000 1.00000
>> 202   hdd   3.63899         osd.202     up  1.00000 1.00000
>> 203   hdd   3.63899         osd.203     up  1.00000 1.00000
>> 204   hdd   3.63899         osd.204     up  1.00000 1.00000
>> 205   hdd   3.63899         osd.205     up  1.00000 1.00000
>> 206   hdd   3.63899         osd.206     up  1.00000 1.00000
>> 207   hdd   3.63899         osd.207     up  1.00000 1.00000
>> 208   hdd   3.63899         osd.208     up  1.00000 1.00000
>> 209   hdd   3.63899         osd.209     up  1.00000 1.00000
>> 210   hdd   3.63899         osd.210     up  1.00000 1.00000
>> 211   hdd   3.63899         osd.211     up  1.00000 1.00000
>> 212   hdd   3.63899         osd.212     up  1.00000 1.00000
>> 213   hdd   3.63899         osd.213     up  1.00000 1.00000
>> 214   hdd   3.63899         osd.214     up  1.00000 1.00000
>> 215   hdd   3.63899         osd.215     up  1.00000 1.00000
>> 216   hdd   3.63899         osd.216     up  1.00000 1.00000
>> 217   hdd   3.63899         osd.217     up  1.00000 1.00000
>> 218   hdd   3.63899         osd.218     up  1.00000 1.00000
>> 219   hdd   3.63899         osd.219     up  1.00000 1.00000
>> 220   hdd   3.63899         osd.220     up  1.00000 1.00000
>> 221   hdd   3.63899         osd.221     up  1.00000 1.00000
>> 222   hdd   3.63899         osd.222     up  1.00000 1.00000
>> 223   hdd   3.63899         osd.223     up  1.00000 1.00000
>> 224   hdd   3.63899         osd.224     up  1.00000 1.00000
>> 225   hdd   3.63899         osd.225     up  1.00000 1.00000
>> 226   hdd   3.63899         osd.226     up  1.00000 1.00000
>> 227   hdd   3.63899         osd.227     up  1.00000 1.00000
>> 228   hdd   3.63899         osd.228     up  1.00000 1.00000
>> 229   hdd   3.63899         osd.229     up  1.00000 1.00000
>>
>> # ceph osd df
>> ID  CLASS WEIGHT  REWEIGHT SIZE  USE   AVAIL %USE  VAR  PGS
>>   0   ssd 0.21799  1.00000  223G 4305M  219G  1.88 0.04  64
>>   1   ssd 0.21799  1.00000  223G 3353M  220G  1.46 0.03  64
>>   2   ssd 0.21799  1.00000  223G 1836M  221G  0.80 0.02  56
>>   3   ssd 0.21799  1.00000  223G 2276M  221G  0.99 0.02  72
>> 230   ssd 0.90999  1.00000  931G 2775M  928G  0.29 0.01  60
>> 231   ssd 0.90999  1.00000  931G 3279M  928G  0.34 0.01  68
>>   4   hdd 3.64499  1.00000 3726G 1916G 1809G 51.45 1.02 111
>>   5   hdd 3.64499  1.00000 3726G 2144G 1581G 57.54 1.15 124
>>   6   hdd 3.64499  1.00000 3726G 1887G 1838G 50.66 1.01 109
>>   7   hdd 3.64499  1.00000 3726G 1642G 2083G 44.09 0.88  95
>>   8   hdd 3.64499  1.00000 3726G 1969G 1756G 52.86 1.05 114
>>   9   hdd 3.64499  1.00000 3726G 1577G 2148G 42.34 0.84  91
>>  10   hdd 3.64499  1.00000 3726G 2043G 1682G 54.85 1.09 118
>>  11   hdd 3.64499  1.00000 3726G 1957G 1768G 52.54 1.05 113
>>  12   hdd 3.64499  1.00000 3726G 1970G 1755G 52.89 1.05 114
>>  13   hdd 3.64499  1.00000 3726G 1676G 2049G 45.00 0.90  97
>>  14   hdd 3.64499  1.00000 3726G 1764G 1961G 47.35 0.94 102
>>  15   hdd 3.64499  1.00000 3726G 2161G 1564G 58.02 1.15 125
>>  16   hdd 3.64499  1.00000 3726G 1749G 1976G 46.96 0.93 101
>>  17   hdd 3.64499  1.00000 3726G 2022G 1703G 54.29 1.08 117
>>  18   hdd 3.64499  1.00000 3726G 1784G 1941G 47.89 0.95 103
>>  19   hdd 3.64499  1.00000 3726G 1783G 1942G 47.85 0.95 103
>>  20   hdd 3.64499  1.00000 3726G 1731G 1994G 46.47 0.92 100
>>  21   hdd 3.64499  1.00000 3726G 1939G 1786G 52.06 1.04 112
>>  22   hdd 3.64499  1.00000 3726G 2061G 1664G 55.34 1.10 119
>>  23   hdd 3.64499  1.00000 3726G 1538G 2187G 41.29 0.82  89
>>  24   hdd 3.64499  1.00000 3726G 2168G 1557G 58.19 1.16 125
>>  25   hdd 3.64499  1.00000 3726G 2023G 1702G 54.32 1.08 117
>>  26   hdd 3.64499  1.00000 3726G 1900G 1825G 51.00 1.02 110
>>  27   hdd 3.64499  1.00000 3726G 2126G 1599G 57.08 1.14 123
>>  28   hdd 3.64499  1.00000 3726G 1938G 1787G 52.02 1.04 112
>>  29   hdd 3.64499  1.00000 3726G 1663G 2062G 44.65 0.89  96
>>  30   hdd 3.64499  1.00000 3726G 2163G 1562G 58.07 1.16 125
>>  31   hdd 3.64499  1.00000 3726G 1919G 1806G 51.52 1.03 111
>>  32   hdd 3.64499  1.00000 3726G 1696G 2029G 45.52 0.91  98
>>  33   hdd 3.64499  1.00000 3726G 1438G 2287G 38.61 0.77  83
>>  34   hdd 3.64499  1.00000 3726G 2073G 1652G 55.64 1.11 120
>>  35   hdd 3.64499  1.00000 3726G 2079G 1646G 55.81 1.11 120
>>  36   hdd 3.64499  1.00000 3726G 1955G 1770G 52.50 1.04 113
>>  37   hdd 3.64499  1.00000 3726G 2180G 1545G 58.51 1.16 126
>>  38   hdd 3.64499  1.00000 3726G 1664G 2061G 44.66 0.89  96
>>  39   hdd 3.64499  1.00000 3726G 2003G 1722G 53.77 1.07 116
>>  40   hdd 3.64499  1.00000 3726G 1782G 1943G 47.84 0.95 103
>>  41   hdd 3.64499  1.00000 3726G 1903G 1822G 51.08 1.02 110
>>  42   hdd 3.64499  1.00000 3726G 1747G 1978G 46.89 0.93 101
>>  43   hdd 3.64499  1.00000 3726G 2021G 1704G 54.26 1.08 117
>>  44   hdd 3.64499  1.00000 3726G 1852G 1873G 49.71 0.99 107
>>  45   hdd 3.64499  1.00000 3726G 1522G 2203G 40.87 0.81  88
>>  46   hdd 3.64499  1.00000 3726G 1851G 1874G 49.69 0.99 107
>>  47   hdd 3.64499  1.00000 3726G 1698G 2027G 45.59 0.91  98
>>  48   hdd 3.64499  1.00000 3726G 1940G 1785G 52.08 1.04 112
>>  49   hdd 3.64499  1.00000 3726G 1746G 1979G 46.88 0.93 101
>>  50   hdd 3.64499  1.00000 3726G 1728G 1997G 46.40 0.92 100
>>  51   hdd 3.64499  1.00000 3726G 2093G 1632G 56.18 1.12 121
>>  52   hdd 3.64499  1.00000 3726G 2235G 1490G 60.01 1.19 129
>>  53   hdd 3.64499  1.00000 3726G 1871G 1854G 50.23 1.00 108
>>  54   hdd 3.64499  1.00000 3726G 2094G 1631G 56.22 1.12 121
>>  55   hdd 3.64499  1.00000 3726G 2422G 1303G 65.02 1.29 140
>>  56   hdd 3.64499  1.00000 3726G 1867G 1858G 50.13 1.00 108
>>  57   hdd 3.64499  1.00000 3726G 1729G 1996G 46.43 0.92 100
>>  58   hdd 3.64499  1.00000 3726G 1976G 1749G 53.05 1.06 114
>>  59   hdd 3.64499  1.00000 3726G 1954G 1771G 52.46 1.04 113
>>  60   hdd 3.64499  1.00000 3726G 1851G 1874G 49.68 0.99 107
>>  61   hdd 3.64499  1.00000 3726G 2042G 1683G 54.82 1.09 118
>>  62   hdd 3.64499  1.00000 3726G 1921G 1804G 51.58 1.03 111
>>  63   hdd 3.64499  1.00000 3726G 2128G 1597G 57.13 1.14 123
>>  64   hdd 3.64499  1.00000 3726G 2037G 1688G 54.67 1.09 118
>>  65   hdd 3.64499  1.00000 3726G 1970G 1755G 52.89 1.05 114
>>  66   hdd 3.64499  1.00000 3726G 1539G 2186G 41.32 0.82  89
>>  67   hdd 3.64499  1.00000 3726G 1867G 1858G 50.13 1.00 108
>>  68   hdd 3.64499  1.00000 3726G 1884G 1841G 50.58 1.01 109
>>  69   hdd 3.64499  1.00000 3726G 2092G 1633G 56.16 1.12 121
>>  70   hdd 3.64499  1.00000 3726G 1800G 1925G 48.33 0.96 104
>>  71   hdd 3.64499  1.00000 3726G 2231G 1494G 59.89 1.19 129
>>  72   hdd 3.64499  1.00000 3726G 1557G 2168G 41.79 0.83  90
>>  73   hdd 3.64499  1.00000 3726G 2300G 1425G 61.75 1.23 133
>>  74   hdd 3.64499  1.00000 3726G 2132G 1593G 57.24 1.14 123
>>  75   hdd 3.64499  1.00000 3726G 1818G 1907G 48.80 0.97 105
>>  76   hdd 3.64499  1.00000 3726G 2231G 1494G 59.89 1.19 129
>>  77   hdd 3.64499  1.00000 3726G 1749G 1976G 46.96 0.93 101
>>  78   hdd 3.64499  1.00000 3726G 1766G 1959G 47.41 0.94 102
>>  79   hdd 3.64499  1.00000 3726G 1678G 2047G 45.04 0.90  97
>>  80   hdd 3.63899  1.00000 3726G 1744G 1981G 46.82 0.93 102
>>  81   hdd 3.64499  1.00000 3726G 1884G 1841G 50.58 1.01 109
>>  82   hdd 3.64499  1.00000 3726G 1711G 2014G 45.93 0.91  99
>>  83   hdd 3.64499  1.00000 3726G 1728G 1997G 46.39 0.92 100
>>  84   hdd 3.64499  1.00000 3726G 1834G 1891G 49.22 0.98 106
>>  85   hdd 3.64499  1.00000 3726G 1709G 2016G 45.88 0.91  99
>>  86   hdd 3.64499  1.00000 3726G 1938G 1787G 52.03 1.04 112
>>  87   hdd 3.64499  1.00000 3726G 2131G 1594G 57.19 1.14 123
>>  88   hdd 3.64499  1.00000 3726G 1559G 2166G 41.85 0.83  90
>>  89   hdd 3.64499  1.00000 3726G 1645G 2080G 44.16 0.88  95
>>  90   hdd 3.64499  1.00000 3726G 2045G 1680G 54.89 1.09 118
>>  91   hdd 3.64499  1.00000 3726G 2164G 1561G 58.09 1.16 125
>>  92   hdd 3.64499  1.00000 3726G 1922G 1803G 51.59 1.03 111
>>  93   hdd 3.64499  1.00000 3726G 1938G 1787G 52.03 1.04 112
>>  94   hdd 3.64499  1.00000 3726G 1971G 1754G 52.91 1.05 114
>>  95   hdd 3.64499  1.00000 3726G 1917G 1808G 51.47 1.02 111
>>  96   hdd 3.64499  1.00000 3726G 1814G 1911G 48.70 0.97 105
>>  97   hdd 3.64499  1.00000 3726G 1660G 2065G 44.57 0.89  96
>>  98   hdd 3.64499  1.00000 3726G 1904G 1821G 51.10 1.02 110
>>  99   hdd 3.64499  1.00000 3726G 2112G 1613G 56.70 1.13 122
>> 100   hdd 3.64499  1.00000 3726G 1869G 1856G 50.18 1.00 108
>> 101   hdd 3.64499  1.00000 3726G 2005G 1720G 53.84 1.07 116
>> 102   hdd 3.64499  1.00000 3726G 2039G 1686G 54.73 1.09 118
>> 103   hdd 3.64499  1.00000 3726G 2093G 1632G 56.20 1.12 121
>> 104   hdd 3.64499  1.00000 3726G 1884G 1841G 50.58 1.01 109
>> 105   hdd 3.64499  1.00000 3726G 1867G 1858G 50.13 1.00 108
>> 106   hdd 3.64499  1.00000 3726G 1956G 1769G 52.51 1.05 113
>> 107   hdd 3.64499  1.00000 3726G 2077G 1648G 55.74 1.11 120
>> 108   hdd 3.64499  1.00000 3726G 1990G 1735G 53.43 1.06 115
>> 109   hdd 3.64499  1.00000 3726G 1781G 1944G 47.80 0.95 103
>> 110   hdd 3.64499  1.00000 3726G 1869G 1856G 50.17 1.00 108
>> 111   hdd 3.64499  1.00000 3726G 2023G 1702G 54.30 1.08 117
>> 112   hdd 3.64499  1.00000 3726G 2007G 1718G 53.89 1.07 116
>> 113   hdd 3.64499  1.00000 3726G 1851G 1874G 49.70 0.99 107
>> 114   hdd 3.64499  1.00000 3726G 1832G 1893G 49.18 0.98 106
>> 115   hdd 3.64499  1.00000 3726G 2098G 1627G 56.32 1.12 121
>> 116   hdd 3.64499  1.00000 3726G 1866G 1859G 50.10 1.00 108
>> 117   hdd 3.64499  1.00000 3726G 1764G 1961G 47.36 0.94 102
>> 118   hdd 3.64499  1.00000 3726G 1816G 1909G 48.76 0.97 105
>> 119   hdd 3.64499  1.00000 3726G 1733G 1992G 46.53 0.93 100
>> 120   hdd 3.64499  1.00000 3726G 1992G 1733G 53.48 1.06 115
>> 121   hdd 3.64499  1.00000 3726G 1657G 2068G 44.48 0.89  96
>> 122   hdd 3.64499  1.00000 3726G 1715G 2010G 46.04 0.92  99
>> 123   hdd 3.64499  1.00000 3726G 1974G 1751G 53.00 1.05 114
>> 124   hdd 3.64499  1.00000 3726G 1731G 1994G 46.48 0.93 100
>> 125   hdd 3.64499  1.00000 3726G 1897G 1828G 50.94 1.01 110
>> 126   hdd 3.64499  1.00000 3726G 1990G 1735G 53.42 1.06 115
>> 127   hdd 3.64499  1.00000 3726G 1972G 1753G 52.93 1.05 114
>> 128   hdd 3.64499  1.00000 3726G 1888G 1837G 50.68 1.01 109
>> 129   hdd 3.64499  1.00000 3726G 2025G 1700G 54.35 1.08 117
>> 130   hdd 3.64499  1.00000 3726G 1747G 1978G 46.90 0.93 101
>> 131   hdd 3.64499  1.00000 3726G 2061G 1664G 55.32 1.10 119
>> 132   hdd 3.64499  1.00000 3726G 2185G 1540G 58.65 1.17 126
>> 133   hdd 3.64499  1.00000 3726G 1764G 1961G 47.35 0.94 102
>> 134   hdd 3.64499  1.00000 3726G 1782G 1943G 47.84 0.95 103
>> 135   hdd 3.64499  1.00000 3726G 1675G 2050G 44.96 0.89  97
>> 136   hdd 3.64499  1.00000 3726G 1889G 1836G 50.70 1.01 109
>> 137   hdd 3.64499  1.00000 3726G 1954G 1771G 52.45 1.04 113
>> 138   hdd 3.64499  1.00000 3726G 1694G 2031G 45.49 0.91  98
>> 139   hdd 3.64499  1.00000 3726G 2114G 1611G 56.76 1.13 122
>> 140   hdd 3.64499  1.00000 3726G 1726G 1999G 46.35 0.92 100
>> 141   hdd 3.64499  1.00000 3726G 1988G 1737G 53.36 1.06 115
>> 142   hdd 3.64499  1.00000 3726G 2181G 1544G 58.55 1.17 126
>> 143   hdd 3.64499  1.00000 3726G 1853G 1872G 49.74 0.99 107
>> 144   hdd 3.64499  1.00000 3726G 2144G 1581G 57.56 1.15 124
>> 145   hdd 3.64499  1.00000 3726G 1868G 1857G 50.14 1.00 108
>> 146   hdd 3.64499  1.00000 3726G 2078G 1647G 55.77 1.11 120
>> 147   hdd 3.64499  1.00000 3726G 1992G 1733G 53.49 1.06 115
>> 148   hdd 3.64499  1.00000 3726G 1728G 1997G 46.39 0.92 100
>> 149   hdd 3.64499  1.00000 3726G 2162G 1563G 58.03 1.15 125
>> 150   hdd 3.64499  1.00000 3726G 2005G 1720G 53.83 1.07 116
>> 151   hdd 3.64499  1.00000 3726G 1918G 1807G 51.49 1.02 111
>> 152   hdd 3.64499  1.00000 3726G 1921G 1804G 51.57 1.03 111
>> 153   hdd 3.64499  1.00000 3726G 1783G 1942G 47.86 0.95 103
>> 154   hdd 3.64499  1.00000 3726G 2005G 1720G 53.82 1.07 116
>> 155   hdd 3.64499  1.00000 3726G 1853G 1873G 49.73 0.99 107
>> 156   hdd 3.64499  1.00000 3726G 1626G 2099G 43.65 0.87  94
>> 157   hdd 3.64499  1.00000 3726G 1802G 1923G 48.38 0.96 104
>> 158   hdd 3.64499  1.00000 3726G 1765G 1960G 47.39 0.94 102
>> 159   hdd 3.64499  1.00000 3726G 1990G 1735G 53.42 1.06 115
>> 160   hdd 3.64499  1.00000 3726G 1901G 1824G 51.02 1.02 110
>> 161   hdd 3.64499  1.00000 3726G 1906G 1819G 51.16 1.02 110
>> 162   hdd 3.64499  1.00000 3726G 1763G 1962G 47.33 0.94 102
>> 163   hdd 3.64499  1.00000 3726G 1800G 1925G 48.33 0.96 104
>> 164   hdd 3.64499  1.00000 3726G 1922G 1803G 51.61 1.03 111
>> 165   hdd 3.64499  1.00000 3726G 1731G 1994G 46.46 0.92 100
>> 166   hdd 3.64499  1.00000 3726G 2027G 1699G 54.40 1.08 117
>> 167   hdd 3.64499  1.00000 3726G 1956G 1769G 52.50 1.04 113
>> 168   hdd 3.64499  1.00000 3726G 1853G 1872G 49.74 0.99 107
>> 169   hdd 3.64499  1.00000 3726G 1884G 1841G 50.59 1.01 109
>> 170   hdd 3.64499  1.00000 3726G 1992G 1733G 53.48 1.06 115
>> 171   hdd 3.64499  1.00000 3726G 1898G 1827G 50.96 1.01 110
>> 172   hdd 3.64499  1.00000 3726G 1781G 1944G 47.82 0.95 103
>> 173   hdd 3.64499  1.00000 3726G 1628G 2097G 43.71 0.87  94
>> 174   hdd 3.64499  1.00000 3726G 1992G 1733G 53.48 1.06 115
>> 175   hdd 3.64499  1.00000 3726G 1905G 1820G 51.15 1.02 110
>> 176   hdd 3.64499  1.00000 3726G 1709G 2016G 45.89 0.91  99
>> 177   hdd 3.64499  1.00000 3726G 1991G 1734G 53.44 1.06 115
>> 178   hdd 3.64499  1.00000 3726G 1973G 1752G 52.97 1.05 114
>> 179   hdd 3.64499  1.00000 3726G 1574G 2151G 42.26 0.84  91
>> 180   hdd 3.64499  1.00000 3726G 1917G 1808G 51.46 1.02 111
>> 181   hdd 3.64499  1.00000 3726G 1937G 1788G 52.00 1.03 112
>> 182   hdd 3.64499  1.00000 3726G 1989G 1736G 53.40 1.06 115
>> 183   hdd 3.64499  1.00000 3726G 2039G 1686G 54.74 1.09 118
>> 184   hdd 3.64499  1.00000 3726G 1868G 1857G 50.16 1.00 108
>> 185   hdd 3.64499  1.00000 3726G 1954G 1771G 52.45 1.04 113
>> 186   hdd 3.64499  1.00000 3726G 1850G 1875G 49.66 0.99 107
>> 187   hdd 3.64499  1.00000 3726G 1851G 1874G 49.68 0.99 107
>> 188   hdd 3.64499  1.00000 3726G 1814G 1911G 48.69 0.97 105
>> 189   hdd 3.64499  1.00000 3726G 2286G 1439G 61.36 1.22 132
>> 190   hdd 3.64499  1.00000 3726G 2060G 1665G 55.31 1.10 119
>> 191   hdd 3.64499  1.00000 3726G 1836G 1889G 49.29 0.98 106
>> 192   hdd 3.63869  1.00000 3726G 1721G 2004G 46.21 0.92 101
>> 193   hdd 3.64499  1.00000 3726G 1953G 1772G 52.43 1.04 113
>> 194   hdd 3.64499  1.00000 3726G 1541G 2184G 41.37 0.82  89
>> 195   hdd 3.64499  1.00000 3726G 1837G 1888G 49.32 0.98 106
>> 196   hdd 3.63899  1.00000 3726G 1705G 2020G 45.79 0.91 100
>> 197   hdd 3.63899  1.00000 3726G 1743G 1982G 46.78 0.93 102
>> 198   hdd 3.63899  1.00000 3726G 1982G 1743G 53.21 1.06 116
>> 199   hdd 3.63899  1.00000 3726G 1382G 2343G 37.11 0.74  81
>> 200   hdd 3.63899  1.00000 3726G 1893G 1832G 50.83 1.01 111
>> 201   hdd 3.63899  1.00000 3726G 1845G 1880G 49.54 0.99 108
>> 202   hdd 3.63899  1.00000 3726G 1658G 2067G 44.51 0.89  97
>> 203   hdd 3.63899  1.00000 3726G 2066G 1659G 55.47 1.10 121
>> 204   hdd 3.63899  1.00000 3726G 1672G 2053G 44.90 0.89  98
>> 205   hdd 3.63899  1.00000 3726G 1521G 2204G 40.84 0.81  89
>> 206   hdd 3.63899  1.00000 3726G 1796G 1929G 48.22 0.96 105
>> 207   hdd 3.63899  1.00000 3726G 1555G 2170G 41.75 0.83  91
>> 208   hdd 3.63899  1.00000 3726G 1742G 1983G 46.77 0.93 102
>> 209   hdd 3.63899  1.00000 3726G 2016G 1709G 54.11 1.08 118
>> 210   hdd 3.63899  1.00000 3726G 2136G 1589G 57.34 1.14 125
>> 211   hdd 3.63899  1.00000 3726G 1656G 2069G 44.46 0.88  97
>> 212   hdd 3.63899  1.00000 3726G 1553G 2172G 41.71 0.83  91
>> 213   hdd 3.63899  1.00000 3726G 1758G 1967G 47.19 0.94 103
>> 214   hdd 3.63899  1.00000 3726G 1603G 2122G 43.03 0.86  94
>> 215   hdd 3.63899  1.00000 3726G 1809G 1916G 48.57 0.97 106
>> 216   hdd 3.63899  1.00000 3726G 1760G 1965G 47.26 0.94 103
>> 217   hdd 3.63899  1.00000 3726G 1742G 1983G 46.77 0.93 102
>> 218   hdd 3.63899  1.00000 3726G 1880G 1845G 50.46 1.00 110
>> 219   hdd 3.63899  1.00000 3726G 1693G 2032G 45.45 0.90  99
>> 220   hdd 3.63899  1.00000 3726G 2032G 1693G 54.56 1.09 119
>> 221   hdd 3.63899  1.00000 3726G 1842G 1883G 49.44 0.98 108
>> 222   hdd 3.63899  1.00000 3726G 1521G 2204G 40.84 0.81  89
>> 223   hdd 3.63899  1.00000 3726G 1470G 2255G 39.46 0.79  86
>> 224   hdd 3.63899  1.00000 3726G 1808G 1917G 48.53 0.97 106
>> 225   hdd 3.63899  1.00000 3726G 1893G 1832G 50.83 1.01 111
>> 226   hdd 3.63899  1.00000 3726G 1776G 1949G 47.69 0.95 104
>> 227   hdd 3.63899  1.00000 3726G 1929G 1796G 51.78 1.03 113
>> 228   hdd 3.63899  1.00000 3726G 1657G 2068G 44.48 0.89  97
>> 229   hdd 3.63899  1.00000 3726G 1843G 1882G 49.47 0.98 108
>>                      TOTAL  825T  414T  410T 50.24
>> MIN/MAX VAR: 0.01/1.29  STDDEV: 9.22
>> --------------------------------------------------------------
>>
>> Am 20.10.18 um 20:35 schrieb Janne Johansson:
>>> Yes, if you have uneven sizes I guess you could end up in a situation
>>> where you have
>>> lots of 1TB OSDs and a number of 2TB OSD but pool replication forces
>>> the pool to have one
>>> PG replica on the 1TB OSD, then it would be possible to state "this
>>> pool cant write more than X G"
>>> but when it is full, there would be free space left on some of the
>>> 2TB-OSDs, but which the pool
>>> cant utilize. Probably same for uneven OSD hosts if you have those.
>>>
>>> Den lör 20 okt. 2018 kl 20:28 skrev Oliver Freyermuth
>>> <freyermuth at physik.uni-bonn.de>:
>>>>
>>>> Dear Janne,
>>>>
>>>> yes, of course. But since we only have two pools here, this can not explain the difference.
>>>> The metadata is replicated (3 copies) across ssd drives, and we have < 3 TB of total raw storage for that.
>>>> So looking at the raw space usage, we can ignore that.
>>>>
>>>> All the rest is used for the ceph_data pool. So the ceph_data pool, in terms of raw storage, is about 50 % used.
>>>>
>>>> But in terms of storage shown for that pool, it's almost 63 % %USED.
>>>> So I guess this can purely be from bad balancing, correct?
>>>>
>>>> Cheers,
>>>>         Oliver
>>>>
>>>> Am 20.10.18 um 19:49 schrieb Janne Johansson:
>>>>> Do mind that drives may have more than one pool on them, so RAW space
>>>>> is what it says, how much free space there is. Then the avail and
>>>>> %USED on per-pool stats will take replication into account, it can
>>>>> tell how much data you may write into that particular pool, given that
>>>>> pools replication or EC settings.
>>>>>
>>>>> Den lör 20 okt. 2018 kl 19:09 skrev Oliver Freyermuth
>>>>> <freyermuth at physik.uni-bonn.de>:
>>>>>>
>>>>>> Dear Cephalopodians,
>>>>>>
>>>>>> as many others, I'm also a bit confused by "ceph df" output
>>>>>> in a pretty straightforward configuration.
>>>>>>
>>>>>> We have a CephFS (12.2.7) running, with 4+2 EC profile.
>>>>>>
>>>>>> I get:
>>>>>> ----------------------------------------------------------------------------
>>>>>> # ceph df
>>>>>> GLOBAL:
>>>>>>     SIZE     AVAIL     RAW USED     %RAW USED
>>>>>>     824T      410T         414T         50.26
>>>>>> POOLS:
>>>>>>     NAME                ID     USED     %USED     MAX AVAIL     OBJECTS
>>>>>>     cephfs_metadata     1      452M      0.05          860G       365774
>>>>>>     cephfs_data         2      275T     62.68          164T     75056403
>>>>>> ----------------------------------------------------------------------------
>>>>>>
>>>>>> So about 50 % of raw space are used, but already ~63 % of filesystem space are used.
>>>>>> Is this purely from imperfect balancing?
>>>>>> In "ceph osd df", I do indeed see OSD usages spreading from 65.02 % usage down to 37.12 %.
>>>>>>
>>>>>> We did not yet use the balancer plugin.
>>>>>> We don't have any pre-luminous clients.
>>>>>> In that setup, I take it that "upmap" mode would be recommended - correct?
>>>>>> Any "gotchas" using that on luminous?
>>>>>>
>>>>>> Cheers,
>>>>>>         Oliver
>>>>>>
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list
>>>>>> ceph-users at lists.ceph.com
>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
> 
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5455 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181020/edfe043b/attachment.bin>


More information about the ceph-users mailing list