[ceph-users] ceph df space usage confusion - balancing needed?

David Turner drakonstein at gmail.com
Mon Oct 22 11:28:02 PDT 2018


I haven't had crush-compat do anything helpful for balancing my clusters.
upmap has been amazing and balanced my clusters far better than anything
else I've ever seen.  I would go so far as to say that upmap can achieve a
perfect balance.

It seems to evenly distribute the PGs for each pool onto all OSDs that pool
is on.  It does that with a maximum difference of 1PG based on how
divisible the number of PGs are with the number of OSDs you have.  As a
side note, your OSD CRUSH weights should be the default weights for their
size for upmap to be as effective as it can be.

On Sat, Oct 20, 2018 at 3:58 PM Oliver Freyermuth <
freyermuth at physik.uni-bonn.de> wrote:

> Ok, I'll try out the balancer end of the upcoming week then (after we've
> fixed a HW-issue with one of our mons
> and the cooling system).
>
> Until then, any further advice and whether upmap is recommended over
> crush-compat (all clients are Luminous) are welcome ;-).
>
> Cheers,
> Oliver
>
> Am 20.10.18 um 21:26 schrieb Janne Johansson:
> > Ok, can't say "why" then, I'd reweigh them somewhat to even it out,
> > 1.22 -vs- 0.74 in variance is a lot, so either a balancer plugin for
> > the MGRs, a script or just a few manual tweaks might be in order.
> >
> > Den lör 20 okt. 2018 kl 21:02 skrev Oliver Freyermuth
> > <freyermuth at physik.uni-bonn.de>:
> >>
> >> All OSDs are of the very same size. One OSD host has slightly more
> disks (33 instead of 31), though.
> >> So also that that can't explain the hefty difference.
> >>
> >> I attach the output of "ceph osd tree" and "ceph osd df".
> >>
> >> The crush rule for the ceph_data pool is:
> >> rule cephfs_data {
> >>         id 2
> >>         type erasure
> >>         min_size 3
> >>         max_size 6
> >>         step set_chooseleaf_tries 5
> >>         step set_choose_tries 100
> >>         step take default class hdd
> >>         step chooseleaf indep 0 type host
> >>         step emit
> >> }
> >> So that only considers the hdd device class. EC is done with k=4 m=2.
> >>
> >> So I don't see any imbalance on the hardware level, but only a somewhat
> uneven distribution of PGs.
> >> Am I missing something, or is this really just a case for the ceph
> balancer plugin?
> >> I'm just a bit astonished this effect is so huge.
> >> Maybe our 4096 PGs for the ceph_data pool are not enough to get an even
> distribution without balancing?
> >> But it yields about 100 PGs per OSD, as you can see...
> >>
> >> --------------------------------------------------------------
> >> # ceph osd tree
> >> ID  CLASS WEIGHT    TYPE NAME       STATUS REWEIGHT PRI-AFF
> >>  -1       826.26428 root default
> >>  -3         0.43700     host mon001
> >>   0   ssd   0.21799         osd.0       up  1.00000 1.00000
> >>   1   ssd   0.21799         osd.1       up  1.00000 1.00000
> >>  -5         0.43700     host mon002
> >>   2   ssd   0.21799         osd.2       up  1.00000 1.00000
> >>   3   ssd   0.21799         osd.3       up  1.00000 1.00000
> >> -31         1.81898     host mon003
> >> 230   ssd   0.90999         osd.230     up  1.00000 1.00000
> >> 231   ssd   0.90999         osd.231     up  1.00000 1.00000
> >> -10       116.64600     host osd001
> >>   4   hdd   3.64499         osd.4       up  1.00000 1.00000
> >>   5   hdd   3.64499         osd.5       up  1.00000 1.00000
> >>   6   hdd   3.64499         osd.6       up  1.00000 1.00000
> >>   7   hdd   3.64499         osd.7       up  1.00000 1.00000
> >>   8   hdd   3.64499         osd.8       up  1.00000 1.00000
> >>   9   hdd   3.64499         osd.9       up  1.00000 1.00000
> >>  10   hdd   3.64499         osd.10      up  1.00000 1.00000
> >>  11   hdd   3.64499         osd.11      up  1.00000 1.00000
> >>  12   hdd   3.64499         osd.12      up  1.00000 1.00000
> >>  13   hdd   3.64499         osd.13      up  1.00000 1.00000
> >>  14   hdd   3.64499         osd.14      up  1.00000 1.00000
> >>  15   hdd   3.64499         osd.15      up  1.00000 1.00000
> >>  16   hdd   3.64499         osd.16      up  1.00000 1.00000
> >>  17   hdd   3.64499         osd.17      up  1.00000 1.00000
> >>  18   hdd   3.64499         osd.18      up  1.00000 1.00000
> >>  19   hdd   3.64499         osd.19      up  1.00000 1.00000
> >>  20   hdd   3.64499         osd.20      up  1.00000 1.00000
> >>  21   hdd   3.64499         osd.21      up  1.00000 1.00000
> >>  22   hdd   3.64499         osd.22      up  1.00000 1.00000
> >>  23   hdd   3.64499         osd.23      up  1.00000 1.00000
> >>  24   hdd   3.64499         osd.24      up  1.00000 1.00000
> >>  25   hdd   3.64499         osd.25      up  1.00000 1.00000
> >>  26   hdd   3.64499         osd.26      up  1.00000 1.00000
> >>  27   hdd   3.64499         osd.27      up  1.00000 1.00000
> >>  28   hdd   3.64499         osd.28      up  1.00000 1.00000
> >>  29   hdd   3.64499         osd.29      up  1.00000 1.00000
> >>  30   hdd   3.64499         osd.30      up  1.00000 1.00000
> >>  31   hdd   3.64499         osd.31      up  1.00000 1.00000
> >>  32   hdd   3.64499         osd.32      up  1.00000 1.00000
> >>  33   hdd   3.64499         osd.33      up  1.00000 1.00000
> >>  34   hdd   3.64499         osd.34      up  1.00000 1.00000
> >>  35   hdd   3.64499         osd.35      up  1.00000 1.00000
> >> -13       116.64600     host osd002
> >>  36   hdd   3.64499         osd.36      up  1.00000 1.00000
> >>  37   hdd   3.64499         osd.37      up  1.00000 1.00000
> >>  38   hdd   3.64499         osd.38      up  1.00000 1.00000
> >>  39   hdd   3.64499         osd.39      up  1.00000 1.00000
> >>  40   hdd   3.64499         osd.40      up  1.00000 1.00000
> >>  41   hdd   3.64499         osd.41      up  1.00000 1.00000
> >>  42   hdd   3.64499         osd.42      up  1.00000 1.00000
> >>  43   hdd   3.64499         osd.43      up  1.00000 1.00000
> >>  44   hdd   3.64499         osd.44      up  1.00000 1.00000
> >>  45   hdd   3.64499         osd.45      up  1.00000 1.00000
> >>  46   hdd   3.64499         osd.46      up  1.00000 1.00000
> >>  47   hdd   3.64499         osd.47      up  1.00000 1.00000
> >>  48   hdd   3.64499         osd.48      up  1.00000 1.00000
> >>  49   hdd   3.64499         osd.49      up  1.00000 1.00000
> >>  50   hdd   3.64499         osd.50      up  1.00000 1.00000
> >>  51   hdd   3.64499         osd.51      up  1.00000 1.00000
> >>  52   hdd   3.64499         osd.52      up  1.00000 1.00000
> >>  53   hdd   3.64499         osd.53      up  1.00000 1.00000
> >>  54   hdd   3.64499         osd.54      up  1.00000 1.00000
> >>  55   hdd   3.64499         osd.55      up  1.00000 1.00000
> >>  56   hdd   3.64499         osd.56      up  1.00000 1.00000
> >>  57   hdd   3.64499         osd.57      up  1.00000 1.00000
> >>  58   hdd   3.64499         osd.58      up  1.00000 1.00000
> >>  59   hdd   3.64499         osd.59      up  1.00000 1.00000
> >>  60   hdd   3.64499         osd.60      up  1.00000 1.00000
> >>  61   hdd   3.64499         osd.61      up  1.00000 1.00000
> >>  62   hdd   3.64499         osd.62      up  1.00000 1.00000
> >>  63   hdd   3.64499         osd.63      up  1.00000 1.00000
> >>  64   hdd   3.64499         osd.64      up  1.00000 1.00000
> >>  65   hdd   3.64499         osd.65      up  1.00000 1.00000
> >>  66   hdd   3.64499         osd.66      up  1.00000 1.00000
> >>  67   hdd   3.64499         osd.67      up  1.00000 1.00000
> >> -16       116.63899     host osd003
> >>  68   hdd   3.64499         osd.68      up  1.00000 1.00000
> >>  69   hdd   3.64499         osd.69      up  1.00000 1.00000
> >>  70   hdd   3.64499         osd.70      up  1.00000 1.00000
> >>  71   hdd   3.64499         osd.71      up  1.00000 1.00000
> >>  72   hdd   3.64499         osd.72      up  1.00000 1.00000
> >>  73   hdd   3.64499         osd.73      up  1.00000 1.00000
> >>  74   hdd   3.64499         osd.74      up  1.00000 1.00000
> >>  75   hdd   3.64499         osd.75      up  1.00000 1.00000
> >>  76   hdd   3.64499         osd.76      up  1.00000 1.00000
> >>  77   hdd   3.64499         osd.77      up  1.00000 1.00000
> >>  78   hdd   3.64499         osd.78      up  1.00000 1.00000
> >>  79   hdd   3.64499         osd.79      up  1.00000 1.00000
> >>  80   hdd   3.63899         osd.80      up  1.00000 1.00000
> >>  81   hdd   3.64499         osd.81      up  1.00000 1.00000
> >>  82   hdd   3.64499         osd.82      up  1.00000 1.00000
> >>  83   hdd   3.64499         osd.83      up  1.00000 1.00000
> >>  84   hdd   3.64499         osd.84      up  1.00000 1.00000
> >>  85   hdd   3.64499         osd.85      up  1.00000 1.00000
> >>  86   hdd   3.64499         osd.86      up  1.00000 1.00000
> >>  87   hdd   3.64499         osd.87      up  1.00000 1.00000
> >>  88   hdd   3.64499         osd.88      up  1.00000 1.00000
> >>  89   hdd   3.64499         osd.89      up  1.00000 1.00000
> >>  90   hdd   3.64499         osd.90      up  1.00000 1.00000
> >>  91   hdd   3.64499         osd.91      up  1.00000 1.00000
> >>  92   hdd   3.64499         osd.92      up  1.00000 1.00000
> >>  93   hdd   3.64499         osd.93      up  1.00000 1.00000
> >>  94   hdd   3.64499         osd.94      up  1.00000 1.00000
> >>  95   hdd   3.64499         osd.95      up  1.00000 1.00000
> >>  96   hdd   3.64499         osd.96      up  1.00000 1.00000
> >>  97   hdd   3.64499         osd.97      up  1.00000 1.00000
> >>  98   hdd   3.64499         osd.98      up  1.00000 1.00000
> >>  99   hdd   3.64499         osd.99      up  1.00000 1.00000
> >> -19       116.64600     host osd004
> >> 100   hdd   3.64499         osd.100     up  1.00000 1.00000
> >> 101   hdd   3.64499         osd.101     up  1.00000 1.00000
> >> 102   hdd   3.64499         osd.102     up  1.00000 1.00000
> >> 103   hdd   3.64499         osd.103     up  1.00000 1.00000
> >> 104   hdd   3.64499         osd.104     up  1.00000 1.00000
> >> 105   hdd   3.64499         osd.105     up  1.00000 1.00000
> >> 106   hdd   3.64499         osd.106     up  1.00000 1.00000
> >> 107   hdd   3.64499         osd.107     up  1.00000 1.00000
> >> 108   hdd   3.64499         osd.108     up  1.00000 1.00000
> >> 109   hdd   3.64499         osd.109     up  1.00000 1.00000
> >> 110   hdd   3.64499         osd.110     up  1.00000 1.00000
> >> 111   hdd   3.64499         osd.111     up  1.00000 1.00000
> >> 112   hdd   3.64499         osd.112     up  1.00000 1.00000
> >> 113   hdd   3.64499         osd.113     up  1.00000 1.00000
> >> 114   hdd   3.64499         osd.114     up  1.00000 1.00000
> >> 115   hdd   3.64499         osd.115     up  1.00000 1.00000
> >> 116   hdd   3.64499         osd.116     up  1.00000 1.00000
> >> 117   hdd   3.64499         osd.117     up  1.00000 1.00000
> >> 118   hdd   3.64499         osd.118     up  1.00000 1.00000
> >> 119   hdd   3.64499         osd.119     up  1.00000 1.00000
> >> 120   hdd   3.64499         osd.120     up  1.00000 1.00000
> >> 121   hdd   3.64499         osd.121     up  1.00000 1.00000
> >> 122   hdd   3.64499         osd.122     up  1.00000 1.00000
> >> 123   hdd   3.64499         osd.123     up  1.00000 1.00000
> >> 124   hdd   3.64499         osd.124     up  1.00000 1.00000
> >> 125   hdd   3.64499         osd.125     up  1.00000 1.00000
> >> 126   hdd   3.64499         osd.126     up  1.00000 1.00000
> >> 127   hdd   3.64499         osd.127     up  1.00000 1.00000
> >> 128   hdd   3.64499         osd.128     up  1.00000 1.00000
> >> 129   hdd   3.64499         osd.129     up  1.00000 1.00000
> >> 130   hdd   3.64499         osd.130     up  1.00000 1.00000
> >> 131   hdd   3.64499         osd.131     up  1.00000 1.00000
> >> -22       116.64600     host osd005
> >> 132   hdd   3.64499         osd.132     up  1.00000 1.00000
> >> 133   hdd   3.64499         osd.133     up  1.00000 1.00000
> >> 134   hdd   3.64499         osd.134     up  1.00000 1.00000
> >> 135   hdd   3.64499         osd.135     up  1.00000 1.00000
> >> 136   hdd   3.64499         osd.136     up  1.00000 1.00000
> >> 137   hdd   3.64499         osd.137     up  1.00000 1.00000
> >> 138   hdd   3.64499         osd.138     up  1.00000 1.00000
> >> 139   hdd   3.64499         osd.139     up  1.00000 1.00000
> >> 140   hdd   3.64499         osd.140     up  1.00000 1.00000
> >> 141   hdd   3.64499         osd.141     up  1.00000 1.00000
> >> 142   hdd   3.64499         osd.142     up  1.00000 1.00000
> >> 143   hdd   3.64499         osd.143     up  1.00000 1.00000
> >> 144   hdd   3.64499         osd.144     up  1.00000 1.00000
> >> 145   hdd   3.64499         osd.145     up  1.00000 1.00000
> >> 146   hdd   3.64499         osd.146     up  1.00000 1.00000
> >> 147   hdd   3.64499         osd.147     up  1.00000 1.00000
> >> 148   hdd   3.64499         osd.148     up  1.00000 1.00000
> >> 149   hdd   3.64499         osd.149     up  1.00000 1.00000
> >> 150   hdd   3.64499         osd.150     up  1.00000 1.00000
> >> 151   hdd   3.64499         osd.151     up  1.00000 1.00000
> >> 152   hdd   3.64499         osd.152     up  1.00000 1.00000
> >> 153   hdd   3.64499         osd.153     up  1.00000 1.00000
> >> 154   hdd   3.64499         osd.154     up  1.00000 1.00000
> >> 155   hdd   3.64499         osd.155     up  1.00000 1.00000
> >> 156   hdd   3.64499         osd.156     up  1.00000 1.00000
> >> 157   hdd   3.64499         osd.157     up  1.00000 1.00000
> >> 158   hdd   3.64499         osd.158     up  1.00000 1.00000
> >> 159   hdd   3.64499         osd.159     up  1.00000 1.00000
> >> 160   hdd   3.64499         osd.160     up  1.00000 1.00000
> >> 161   hdd   3.64499         osd.161     up  1.00000 1.00000
> >> 162   hdd   3.64499         osd.162     up  1.00000 1.00000
> >> 163   hdd   3.64499         osd.163     up  1.00000 1.00000
> >> -25       116.63335     host osd006
> >> 164   hdd   3.64499         osd.164     up  1.00000 1.00000
> >> 165   hdd   3.64499         osd.165     up  1.00000 1.00000
> >> 166   hdd   3.64499         osd.166     up  1.00000 1.00000
> >> 167   hdd   3.64499         osd.167     up  1.00000 1.00000
> >> 168   hdd   3.64499         osd.168     up  1.00000 1.00000
> >> 169   hdd   3.64499         osd.169     up  1.00000 1.00000
> >> 170   hdd   3.64499         osd.170     up  1.00000 1.00000
> >> 171   hdd   3.64499         osd.171     up  1.00000 1.00000
> >> 172   hdd   3.64499         osd.172     up  1.00000 1.00000
> >> 173   hdd   3.64499         osd.173     up  1.00000 1.00000
> >> 174   hdd   3.64499         osd.174     up  1.00000 1.00000
> >> 175   hdd   3.64499         osd.175     up  1.00000 1.00000
> >> 176   hdd   3.64499         osd.176     up  1.00000 1.00000
> >> 177   hdd   3.64499         osd.177     up  1.00000 1.00000
> >> 178   hdd   3.64499         osd.178     up  1.00000 1.00000
> >> 179   hdd   3.64499         osd.179     up  1.00000 1.00000
> >> 180   hdd   3.64499         osd.180     up  1.00000 1.00000
> >> 181   hdd   3.64499         osd.181     up  1.00000 1.00000
> >> 182   hdd   3.64499         osd.182     up  1.00000 1.00000
> >> 183   hdd   3.64499         osd.183     up  1.00000 1.00000
> >> 184   hdd   3.64499         osd.184     up  1.00000 1.00000
> >> 185   hdd   3.64499         osd.185     up  1.00000 1.00000
> >> 186   hdd   3.64499         osd.186     up  1.00000 1.00000
> >> 187   hdd   3.64499         osd.187     up  1.00000 1.00000
> >> 188   hdd   3.64499         osd.188     up  1.00000 1.00000
> >> 189   hdd   3.64499         osd.189     up  1.00000 1.00000
> >> 190   hdd   3.64499         osd.190     up  1.00000 1.00000
> >> 191   hdd   3.64499         osd.191     up  1.00000 1.00000
> >> 192   hdd   3.63869         osd.192     up  1.00000 1.00000
> >> 193   hdd   3.64499         osd.193     up  1.00000 1.00000
> >> 194   hdd   3.64499         osd.194     up  1.00000 1.00000
> >> 195   hdd   3.64499         osd.195     up  1.00000 1.00000
> >> -28       123.71500     host osd007
> >> 196   hdd   3.63899         osd.196     up  1.00000 1.00000
> >> 197   hdd   3.63899         osd.197     up  1.00000 1.00000
> >> 198   hdd   3.63899         osd.198     up  1.00000 1.00000
> >> 199   hdd   3.63899         osd.199     up  1.00000 1.00000
> >> 200   hdd   3.63899         osd.200     up  1.00000 1.00000
> >> 201   hdd   3.63899         osd.201     up  1.00000 1.00000
> >> 202   hdd   3.63899         osd.202     up  1.00000 1.00000
> >> 203   hdd   3.63899         osd.203     up  1.00000 1.00000
> >> 204   hdd   3.63899         osd.204     up  1.00000 1.00000
> >> 205   hdd   3.63899         osd.205     up  1.00000 1.00000
> >> 206   hdd   3.63899         osd.206     up  1.00000 1.00000
> >> 207   hdd   3.63899         osd.207     up  1.00000 1.00000
> >> 208   hdd   3.63899         osd.208     up  1.00000 1.00000
> >> 209   hdd   3.63899         osd.209     up  1.00000 1.00000
> >> 210   hdd   3.63899         osd.210     up  1.00000 1.00000
> >> 211   hdd   3.63899         osd.211     up  1.00000 1.00000
> >> 212   hdd   3.63899         osd.212     up  1.00000 1.00000
> >> 213   hdd   3.63899         osd.213     up  1.00000 1.00000
> >> 214   hdd   3.63899         osd.214     up  1.00000 1.00000
> >> 215   hdd   3.63899         osd.215     up  1.00000 1.00000
> >> 216   hdd   3.63899         osd.216     up  1.00000 1.00000
> >> 217   hdd   3.63899         osd.217     up  1.00000 1.00000
> >> 218   hdd   3.63899         osd.218     up  1.00000 1.00000
> >> 219   hdd   3.63899         osd.219     up  1.00000 1.00000
> >> 220   hdd   3.63899         osd.220     up  1.00000 1.00000
> >> 221   hdd   3.63899         osd.221     up  1.00000 1.00000
> >> 222   hdd   3.63899         osd.222     up  1.00000 1.00000
> >> 223   hdd   3.63899         osd.223     up  1.00000 1.00000
> >> 224   hdd   3.63899         osd.224     up  1.00000 1.00000
> >> 225   hdd   3.63899         osd.225     up  1.00000 1.00000
> >> 226   hdd   3.63899         osd.226     up  1.00000 1.00000
> >> 227   hdd   3.63899         osd.227     up  1.00000 1.00000
> >> 228   hdd   3.63899         osd.228     up  1.00000 1.00000
> >> 229   hdd   3.63899         osd.229     up  1.00000 1.00000
> >>
> >> # ceph osd df
> >> ID  CLASS WEIGHT  REWEIGHT SIZE  USE   AVAIL %USE  VAR  PGS
> >>   0   ssd 0.21799  1.00000  223G 4305M  219G  1.88 0.04  64
> >>   1   ssd 0.21799  1.00000  223G 3353M  220G  1.46 0.03  64
> >>   2   ssd 0.21799  1.00000  223G 1836M  221G  0.80 0.02  56
> >>   3   ssd 0.21799  1.00000  223G 2276M  221G  0.99 0.02  72
> >> 230   ssd 0.90999  1.00000  931G 2775M  928G  0.29 0.01  60
> >> 231   ssd 0.90999  1.00000  931G 3279M  928G  0.34 0.01  68
> >>   4   hdd 3.64499  1.00000 3726G 1916G 1809G 51.45 1.02 111
> >>   5   hdd 3.64499  1.00000 3726G 2144G 1581G 57.54 1.15 124
> >>   6   hdd 3.64499  1.00000 3726G 1887G 1838G 50.66 1.01 109
> >>   7   hdd 3.64499  1.00000 3726G 1642G 2083G 44.09 0.88  95
> >>   8   hdd 3.64499  1.00000 3726G 1969G 1756G 52.86 1.05 114
> >>   9   hdd 3.64499  1.00000 3726G 1577G 2148G 42.34 0.84  91
> >>  10   hdd 3.64499  1.00000 3726G 2043G 1682G 54.85 1.09 118
> >>  11   hdd 3.64499  1.00000 3726G 1957G 1768G 52.54 1.05 113
> >>  12   hdd 3.64499  1.00000 3726G 1970G 1755G 52.89 1.05 114
> >>  13   hdd 3.64499  1.00000 3726G 1676G 2049G 45.00 0.90  97
> >>  14   hdd 3.64499  1.00000 3726G 1764G 1961G 47.35 0.94 102
> >>  15   hdd 3.64499  1.00000 3726G 2161G 1564G 58.02 1.15 125
> >>  16   hdd 3.64499  1.00000 3726G 1749G 1976G 46.96 0.93 101
> >>  17   hdd 3.64499  1.00000 3726G 2022G 1703G 54.29 1.08 117
> >>  18   hdd 3.64499  1.00000 3726G 1784G 1941G 47.89 0.95 103
> >>  19   hdd 3.64499  1.00000 3726G 1783G 1942G 47.85 0.95 103
> >>  20   hdd 3.64499  1.00000 3726G 1731G 1994G 46.47 0.92 100
> >>  21   hdd 3.64499  1.00000 3726G 1939G 1786G 52.06 1.04 112
> >>  22   hdd 3.64499  1.00000 3726G 2061G 1664G 55.34 1.10 119
> >>  23   hdd 3.64499  1.00000 3726G 1538G 2187G 41.29 0.82  89
> >>  24   hdd 3.64499  1.00000 3726G 2168G 1557G 58.19 1.16 125
> >>  25   hdd 3.64499  1.00000 3726G 2023G 1702G 54.32 1.08 117
> >>  26   hdd 3.64499  1.00000 3726G 1900G 1825G 51.00 1.02 110
> >>  27   hdd 3.64499  1.00000 3726G 2126G 1599G 57.08 1.14 123
> >>  28   hdd 3.64499  1.00000 3726G 1938G 1787G 52.02 1.04 112
> >>  29   hdd 3.64499  1.00000 3726G 1663G 2062G 44.65 0.89  96
> >>  30   hdd 3.64499  1.00000 3726G 2163G 1562G 58.07 1.16 125
> >>  31   hdd 3.64499  1.00000 3726G 1919G 1806G 51.52 1.03 111
> >>  32   hdd 3.64499  1.00000 3726G 1696G 2029G 45.52 0.91  98
> >>  33   hdd 3.64499  1.00000 3726G 1438G 2287G 38.61 0.77  83
> >>  34   hdd 3.64499  1.00000 3726G 2073G 1652G 55.64 1.11 120
> >>  35   hdd 3.64499  1.00000 3726G 2079G 1646G 55.81 1.11 120
> >>  36   hdd 3.64499  1.00000 3726G 1955G 1770G 52.50 1.04 113
> >>  37   hdd 3.64499  1.00000 3726G 2180G 1545G 58.51 1.16 126
> >>  38   hdd 3.64499  1.00000 3726G 1664G 2061G 44.66 0.89  96
> >>  39   hdd 3.64499  1.00000 3726G 2003G 1722G 53.77 1.07 116
> >>  40   hdd 3.64499  1.00000 3726G 1782G 1943G 47.84 0.95 103
> >>  41   hdd 3.64499  1.00000 3726G 1903G 1822G 51.08 1.02 110
> >>  42   hdd 3.64499  1.00000 3726G 1747G 1978G 46.89 0.93 101
> >>  43   hdd 3.64499  1.00000 3726G 2021G 1704G 54.26 1.08 117
> >>  44   hdd 3.64499  1.00000 3726G 1852G 1873G 49.71 0.99 107
> >>  45   hdd 3.64499  1.00000 3726G 1522G 2203G 40.87 0.81  88
> >>  46   hdd 3.64499  1.00000 3726G 1851G 1874G 49.69 0.99 107
> >>  47   hdd 3.64499  1.00000 3726G 1698G 2027G 45.59 0.91  98
> >>  48   hdd 3.64499  1.00000 3726G 1940G 1785G 52.08 1.04 112
> >>  49   hdd 3.64499  1.00000 3726G 1746G 1979G 46.88 0.93 101
> >>  50   hdd 3.64499  1.00000 3726G 1728G 1997G 46.40 0.92 100
> >>  51   hdd 3.64499  1.00000 3726G 2093G 1632G 56.18 1.12 121
> >>  52   hdd 3.64499  1.00000 3726G 2235G 1490G 60.01 1.19 129
> >>  53   hdd 3.64499  1.00000 3726G 1871G 1854G 50.23 1.00 108
> >>  54   hdd 3.64499  1.00000 3726G 2094G 1631G 56.22 1.12 121
> >>  55   hdd 3.64499  1.00000 3726G 2422G 1303G 65.02 1.29 140
> >>  56   hdd 3.64499  1.00000 3726G 1867G 1858G 50.13 1.00 108
> >>  57   hdd 3.64499  1.00000 3726G 1729G 1996G 46.43 0.92 100
> >>  58   hdd 3.64499  1.00000 3726G 1976G 1749G 53.05 1.06 114
> >>  59   hdd 3.64499  1.00000 3726G 1954G 1771G 52.46 1.04 113
> >>  60   hdd 3.64499  1.00000 3726G 1851G 1874G 49.68 0.99 107
> >>  61   hdd 3.64499  1.00000 3726G 2042G 1683G 54.82 1.09 118
> >>  62   hdd 3.64499  1.00000 3726G 1921G 1804G 51.58 1.03 111
> >>  63   hdd 3.64499  1.00000 3726G 2128G 1597G 57.13 1.14 123
> >>  64   hdd 3.64499  1.00000 3726G 2037G 1688G 54.67 1.09 118
> >>  65   hdd 3.64499  1.00000 3726G 1970G 1755G 52.89 1.05 114
> >>  66   hdd 3.64499  1.00000 3726G 1539G 2186G 41.32 0.82  89
> >>  67   hdd 3.64499  1.00000 3726G 1867G 1858G 50.13 1.00 108
> >>  68   hdd 3.64499  1.00000 3726G 1884G 1841G 50.58 1.01 109
> >>  69   hdd 3.64499  1.00000 3726G 2092G 1633G 56.16 1.12 121
> >>  70   hdd 3.64499  1.00000 3726G 1800G 1925G 48.33 0.96 104
> >>  71   hdd 3.64499  1.00000 3726G 2231G 1494G 59.89 1.19 129
> >>  72   hdd 3.64499  1.00000 3726G 1557G 2168G 41.79 0.83  90
> >>  73   hdd 3.64499  1.00000 3726G 2300G 1425G 61.75 1.23 133
> >>  74   hdd 3.64499  1.00000 3726G 2132G 1593G 57.24 1.14 123
> >>  75   hdd 3.64499  1.00000 3726G 1818G 1907G 48.80 0.97 105
> >>  76   hdd 3.64499  1.00000 3726G 2231G 1494G 59.89 1.19 129
> >>  77   hdd 3.64499  1.00000 3726G 1749G 1976G 46.96 0.93 101
> >>  78   hdd 3.64499  1.00000 3726G 1766G 1959G 47.41 0.94 102
> >>  79   hdd 3.64499  1.00000 3726G 1678G 2047G 45.04 0.90  97
> >>  80   hdd 3.63899  1.00000 3726G 1744G 1981G 46.82 0.93 102
> >>  81   hdd 3.64499  1.00000 3726G 1884G 1841G 50.58 1.01 109
> >>  82   hdd 3.64499  1.00000 3726G 1711G 2014G 45.93 0.91  99
> >>  83   hdd 3.64499  1.00000 3726G 1728G 1997G 46.39 0.92 100
> >>  84   hdd 3.64499  1.00000 3726G 1834G 1891G 49.22 0.98 106
> >>  85   hdd 3.64499  1.00000 3726G 1709G 2016G 45.88 0.91  99
> >>  86   hdd 3.64499  1.00000 3726G 1938G 1787G 52.03 1.04 112
> >>  87   hdd 3.64499  1.00000 3726G 2131G 1594G 57.19 1.14 123
> >>  88   hdd 3.64499  1.00000 3726G 1559G 2166G 41.85 0.83  90
> >>  89   hdd 3.64499  1.00000 3726G 1645G 2080G 44.16 0.88  95
> >>  90   hdd 3.64499  1.00000 3726G 2045G 1680G 54.89 1.09 118
> >>  91   hdd 3.64499  1.00000 3726G 2164G 1561G 58.09 1.16 125
> >>  92   hdd 3.64499  1.00000 3726G 1922G 1803G 51.59 1.03 111
> >>  93   hdd 3.64499  1.00000 3726G 1938G 1787G 52.03 1.04 112
> >>  94   hdd 3.64499  1.00000 3726G 1971G 1754G 52.91 1.05 114
> >>  95   hdd 3.64499  1.00000 3726G 1917G 1808G 51.47 1.02 111
> >>  96   hdd 3.64499  1.00000 3726G 1814G 1911G 48.70 0.97 105
> >>  97   hdd 3.64499  1.00000 3726G 1660G 2065G 44.57 0.89  96
> >>  98   hdd 3.64499  1.00000 3726G 1904G 1821G 51.10 1.02 110
> >>  99   hdd 3.64499  1.00000 3726G 2112G 1613G 56.70 1.13 122
> >> 100   hdd 3.64499  1.00000 3726G 1869G 1856G 50.18 1.00 108
> >> 101   hdd 3.64499  1.00000 3726G 2005G 1720G 53.84 1.07 116
> >> 102   hdd 3.64499  1.00000 3726G 2039G 1686G 54.73 1.09 118
> >> 103   hdd 3.64499  1.00000 3726G 2093G 1632G 56.20 1.12 121
> >> 104   hdd 3.64499  1.00000 3726G 1884G 1841G 50.58 1.01 109
> >> 105   hdd 3.64499  1.00000 3726G 1867G 1858G 50.13 1.00 108
> >> 106   hdd 3.64499  1.00000 3726G 1956G 1769G 52.51 1.05 113
> >> 107   hdd 3.64499  1.00000 3726G 2077G 1648G 55.74 1.11 120
> >> 108   hdd 3.64499  1.00000 3726G 1990G 1735G 53.43 1.06 115
> >> 109   hdd 3.64499  1.00000 3726G 1781G 1944G 47.80 0.95 103
> >> 110   hdd 3.64499  1.00000 3726G 1869G 1856G 50.17 1.00 108
> >> 111   hdd 3.64499  1.00000 3726G 2023G 1702G 54.30 1.08 117
> >> 112   hdd 3.64499  1.00000 3726G 2007G 1718G 53.89 1.07 116
> >> 113   hdd 3.64499  1.00000 3726G 1851G 1874G 49.70 0.99 107
> >> 114   hdd 3.64499  1.00000 3726G 1832G 1893G 49.18 0.98 106
> >> 115   hdd 3.64499  1.00000 3726G 2098G 1627G 56.32 1.12 121
> >> 116   hdd 3.64499  1.00000 3726G 1866G 1859G 50.10 1.00 108
> >> 117   hdd 3.64499  1.00000 3726G 1764G 1961G 47.36 0.94 102
> >> 118   hdd 3.64499  1.00000 3726G 1816G 1909G 48.76 0.97 105
> >> 119   hdd 3.64499  1.00000 3726G 1733G 1992G 46.53 0.93 100
> >> 120   hdd 3.64499  1.00000 3726G 1992G 1733G 53.48 1.06 115
> >> 121   hdd 3.64499  1.00000 3726G 1657G 2068G 44.48 0.89  96
> >> 122   hdd 3.64499  1.00000 3726G 1715G 2010G 46.04 0.92  99
> >> 123   hdd 3.64499  1.00000 3726G 1974G 1751G 53.00 1.05 114
> >> 124   hdd 3.64499  1.00000 3726G 1731G 1994G 46.48 0.93 100
> >> 125   hdd 3.64499  1.00000 3726G 1897G 1828G 50.94 1.01 110
> >> 126   hdd 3.64499  1.00000 3726G 1990G 1735G 53.42 1.06 115
> >> 127   hdd 3.64499  1.00000 3726G 1972G 1753G 52.93 1.05 114
> >> 128   hdd 3.64499  1.00000 3726G 1888G 1837G 50.68 1.01 109
> >> 129   hdd 3.64499  1.00000 3726G 2025G 1700G 54.35 1.08 117
> >> 130   hdd 3.64499  1.00000 3726G 1747G 1978G 46.90 0.93 101
> >> 131   hdd 3.64499  1.00000 3726G 2061G 1664G 55.32 1.10 119
> >> 132   hdd 3.64499  1.00000 3726G 2185G 1540G 58.65 1.17 126
> >> 133   hdd 3.64499  1.00000 3726G 1764G 1961G 47.35 0.94 102
> >> 134   hdd 3.64499  1.00000 3726G 1782G 1943G 47.84 0.95 103
> >> 135   hdd 3.64499  1.00000 3726G 1675G 2050G 44.96 0.89  97
> >> 136   hdd 3.64499  1.00000 3726G 1889G 1836G 50.70 1.01 109
> >> 137   hdd 3.64499  1.00000 3726G 1954G 1771G 52.45 1.04 113
> >> 138   hdd 3.64499  1.00000 3726G 1694G 2031G 45.49 0.91  98
> >> 139   hdd 3.64499  1.00000 3726G 2114G 1611G 56.76 1.13 122
> >> 140   hdd 3.64499  1.00000 3726G 1726G 1999G 46.35 0.92 100
> >> 141   hdd 3.64499  1.00000 3726G 1988G 1737G 53.36 1.06 115
> >> 142   hdd 3.64499  1.00000 3726G 2181G 1544G 58.55 1.17 126
> >> 143   hdd 3.64499  1.00000 3726G 1853G 1872G 49.74 0.99 107
> >> 144   hdd 3.64499  1.00000 3726G 2144G 1581G 57.56 1.15 124
> >> 145   hdd 3.64499  1.00000 3726G 1868G 1857G 50.14 1.00 108
> >> 146   hdd 3.64499  1.00000 3726G 2078G 1647G 55.77 1.11 120
> >> 147   hdd 3.64499  1.00000 3726G 1992G 1733G 53.49 1.06 115
> >> 148   hdd 3.64499  1.00000 3726G 1728G 1997G 46.39 0.92 100
> >> 149   hdd 3.64499  1.00000 3726G 2162G 1563G 58.03 1.15 125
> >> 150   hdd 3.64499  1.00000 3726G 2005G 1720G 53.83 1.07 116
> >> 151   hdd 3.64499  1.00000 3726G 1918G 1807G 51.49 1.02 111
> >> 152   hdd 3.64499  1.00000 3726G 1921G 1804G 51.57 1.03 111
> >> 153   hdd 3.64499  1.00000 3726G 1783G 1942G 47.86 0.95 103
> >> 154   hdd 3.64499  1.00000 3726G 2005G 1720G 53.82 1.07 116
> >> 155   hdd 3.64499  1.00000 3726G 1853G 1873G 49.73 0.99 107
> >> 156   hdd 3.64499  1.00000 3726G 1626G 2099G 43.65 0.87  94
> >> 157   hdd 3.64499  1.00000 3726G 1802G 1923G 48.38 0.96 104
> >> 158   hdd 3.64499  1.00000 3726G 1765G 1960G 47.39 0.94 102
> >> 159   hdd 3.64499  1.00000 3726G 1990G 1735G 53.42 1.06 115
> >> 160   hdd 3.64499  1.00000 3726G 1901G 1824G 51.02 1.02 110
> >> 161   hdd 3.64499  1.00000 3726G 1906G 1819G 51.16 1.02 110
> >> 162   hdd 3.64499  1.00000 3726G 1763G 1962G 47.33 0.94 102
> >> 163   hdd 3.64499  1.00000 3726G 1800G 1925G 48.33 0.96 104
> >> 164   hdd 3.64499  1.00000 3726G 1922G 1803G 51.61 1.03 111
> >> 165   hdd 3.64499  1.00000 3726G 1731G 1994G 46.46 0.92 100
> >> 166   hdd 3.64499  1.00000 3726G 2027G 1699G 54.40 1.08 117
> >> 167   hdd 3.64499  1.00000 3726G 1956G 1769G 52.50 1.04 113
> >> 168   hdd 3.64499  1.00000 3726G 1853G 1872G 49.74 0.99 107
> >> 169   hdd 3.64499  1.00000 3726G 1884G 1841G 50.59 1.01 109
> >> 170   hdd 3.64499  1.00000 3726G 1992G 1733G 53.48 1.06 115
> >> 171   hdd 3.64499  1.00000 3726G 1898G 1827G 50.96 1.01 110
> >> 172   hdd 3.64499  1.00000 3726G 1781G 1944G 47.82 0.95 103
> >> 173   hdd 3.64499  1.00000 3726G 1628G 2097G 43.71 0.87  94
> >> 174   hdd 3.64499  1.00000 3726G 1992G 1733G 53.48 1.06 115
> >> 175   hdd 3.64499  1.00000 3726G 1905G 1820G 51.15 1.02 110
> >> 176   hdd 3.64499  1.00000 3726G 1709G 2016G 45.89 0.91  99
> >> 177   hdd 3.64499  1.00000 3726G 1991G 1734G 53.44 1.06 115
> >> 178   hdd 3.64499  1.00000 3726G 1973G 1752G 52.97 1.05 114
> >> 179   hdd 3.64499  1.00000 3726G 1574G 2151G 42.26 0.84  91
> >> 180   hdd 3.64499  1.00000 3726G 1917G 1808G 51.46 1.02 111
> >> 181   hdd 3.64499  1.00000 3726G 1937G 1788G 52.00 1.03 112
> >> 182   hdd 3.64499  1.00000 3726G 1989G 1736G 53.40 1.06 115
> >> 183   hdd 3.64499  1.00000 3726G 2039G 1686G 54.74 1.09 118
> >> 184   hdd 3.64499  1.00000 3726G 1868G 1857G 50.16 1.00 108
> >> 185   hdd 3.64499  1.00000 3726G 1954G 1771G 52.45 1.04 113
> >> 186   hdd 3.64499  1.00000 3726G 1850G 1875G 49.66 0.99 107
> >> 187   hdd 3.64499  1.00000 3726G 1851G 1874G 49.68 0.99 107
> >> 188   hdd 3.64499  1.00000 3726G 1814G 1911G 48.69 0.97 105
> >> 189   hdd 3.64499  1.00000 3726G 2286G 1439G 61.36 1.22 132
> >> 190   hdd 3.64499  1.00000 3726G 2060G 1665G 55.31 1.10 119
> >> 191   hdd 3.64499  1.00000 3726G 1836G 1889G 49.29 0.98 106
> >> 192   hdd 3.63869  1.00000 3726G 1721G 2004G 46.21 0.92 101
> >> 193   hdd 3.64499  1.00000 3726G 1953G 1772G 52.43 1.04 113
> >> 194   hdd 3.64499  1.00000 3726G 1541G 2184G 41.37 0.82  89
> >> 195   hdd 3.64499  1.00000 3726G 1837G 1888G 49.32 0.98 106
> >> 196   hdd 3.63899  1.00000 3726G 1705G 2020G 45.79 0.91 100
> >> 197   hdd 3.63899  1.00000 3726G 1743G 1982G 46.78 0.93 102
> >> 198   hdd 3.63899  1.00000 3726G 1982G 1743G 53.21 1.06 116
> >> 199   hdd 3.63899  1.00000 3726G 1382G 2343G 37.11 0.74  81
> >> 200   hdd 3.63899  1.00000 3726G 1893G 1832G 50.83 1.01 111
> >> 201   hdd 3.63899  1.00000 3726G 1845G 1880G 49.54 0.99 108
> >> 202   hdd 3.63899  1.00000 3726G 1658G 2067G 44.51 0.89  97
> >> 203   hdd 3.63899  1.00000 3726G 2066G 1659G 55.47 1.10 121
> >> 204   hdd 3.63899  1.00000 3726G 1672G 2053G 44.90 0.89  98
> >> 205   hdd 3.63899  1.00000 3726G 1521G 2204G 40.84 0.81  89
> >> 206   hdd 3.63899  1.00000 3726G 1796G 1929G 48.22 0.96 105
> >> 207   hdd 3.63899  1.00000 3726G 1555G 2170G 41.75 0.83  91
> >> 208   hdd 3.63899  1.00000 3726G 1742G 1983G 46.77 0.93 102
> >> 209   hdd 3.63899  1.00000 3726G 2016G 1709G 54.11 1.08 118
> >> 210   hdd 3.63899  1.00000 3726G 2136G 1589G 57.34 1.14 125
> >> 211   hdd 3.63899  1.00000 3726G 1656G 2069G 44.46 0.88  97
> >> 212   hdd 3.63899  1.00000 3726G 1553G 2172G 41.71 0.83  91
> >> 213   hdd 3.63899  1.00000 3726G 1758G 1967G 47.19 0.94 103
> >> 214   hdd 3.63899  1.00000 3726G 1603G 2122G 43.03 0.86  94
> >> 215   hdd 3.63899  1.00000 3726G 1809G 1916G 48.57 0.97 106
> >> 216   hdd 3.63899  1.00000 3726G 1760G 1965G 47.26 0.94 103
> >> 217   hdd 3.63899  1.00000 3726G 1742G 1983G 46.77 0.93 102
> >> 218   hdd 3.63899  1.00000 3726G 1880G 1845G 50.46 1.00 110
> >> 219   hdd 3.63899  1.00000 3726G 1693G 2032G 45.45 0.90  99
> >> 220   hdd 3.63899  1.00000 3726G 2032G 1693G 54.56 1.09 119
> >> 221   hdd 3.63899  1.00000 3726G 1842G 1883G 49.44 0.98 108
> >> 222   hdd 3.63899  1.00000 3726G 1521G 2204G 40.84 0.81  89
> >> 223   hdd 3.63899  1.00000 3726G 1470G 2255G 39.46 0.79  86
> >> 224   hdd 3.63899  1.00000 3726G 1808G 1917G 48.53 0.97 106
> >> 225   hdd 3.63899  1.00000 3726G 1893G 1832G 50.83 1.01 111
> >> 226   hdd 3.63899  1.00000 3726G 1776G 1949G 47.69 0.95 104
> >> 227   hdd 3.63899  1.00000 3726G 1929G 1796G 51.78 1.03 113
> >> 228   hdd 3.63899  1.00000 3726G 1657G 2068G 44.48 0.89  97
> >> 229   hdd 3.63899  1.00000 3726G 1843G 1882G 49.47 0.98 108
> >>                      TOTAL  825T  414T  410T 50.24
> >> MIN/MAX VAR: 0.01/1.29  STDDEV: 9.22
> >> --------------------------------------------------------------
> >>
> >> Am 20.10.18 um 20:35 schrieb Janne Johansson:
> >>> Yes, if you have uneven sizes I guess you could end up in a situation
> >>> where you have
> >>> lots of 1TB OSDs and a number of 2TB OSD but pool replication forces
> >>> the pool to have one
> >>> PG replica on the 1TB OSD, then it would be possible to state "this
> >>> pool cant write more than X G"
> >>> but when it is full, there would be free space left on some of the
> >>> 2TB-OSDs, but which the pool
> >>> cant utilize. Probably same for uneven OSD hosts if you have those.
> >>>
> >>> Den lör 20 okt. 2018 kl 20:28 skrev Oliver Freyermuth
> >>> <freyermuth at physik.uni-bonn.de>:
> >>>>
> >>>> Dear Janne,
> >>>>
> >>>> yes, of course. But since we only have two pools here, this can not
> explain the difference.
> >>>> The metadata is replicated (3 copies) across ssd drives, and we have
> < 3 TB of total raw storage for that.
> >>>> So looking at the raw space usage, we can ignore that.
> >>>>
> >>>> All the rest is used for the ceph_data pool. So the ceph_data pool,
> in terms of raw storage, is about 50 % used.
> >>>>
> >>>> But in terms of storage shown for that pool, it's almost 63 % %USED.
> >>>> So I guess this can purely be from bad balancing, correct?
> >>>>
> >>>> Cheers,
> >>>>         Oliver
> >>>>
> >>>> Am 20.10.18 um 19:49 schrieb Janne Johansson:
> >>>>> Do mind that drives may have more than one pool on them, so RAW space
> >>>>> is what it says, how much free space there is. Then the avail and
> >>>>> %USED on per-pool stats will take replication into account, it can
> >>>>> tell how much data you may write into that particular pool, given
> that
> >>>>> pools replication or EC settings.
> >>>>>
> >>>>> Den lör 20 okt. 2018 kl 19:09 skrev Oliver Freyermuth
> >>>>> <freyermuth at physik.uni-bonn.de>:
> >>>>>>
> >>>>>> Dear Cephalopodians,
> >>>>>>
> >>>>>> as many others, I'm also a bit confused by "ceph df" output
> >>>>>> in a pretty straightforward configuration.
> >>>>>>
> >>>>>> We have a CephFS (12.2.7) running, with 4+2 EC profile.
> >>>>>>
> >>>>>> I get:
> >>>>>>
> ----------------------------------------------------------------------------
> >>>>>> # ceph df
> >>>>>> GLOBAL:
> >>>>>>     SIZE     AVAIL     RAW USED     %RAW USED
> >>>>>>     824T      410T         414T         50.26
> >>>>>> POOLS:
> >>>>>>     NAME                ID     USED     %USED     MAX AVAIL
>  OBJECTS
> >>>>>>     cephfs_metadata     1      452M      0.05          860G
>  365774
> >>>>>>     cephfs_data         2      275T     62.68          164T
>  75056403
> >>>>>>
> ----------------------------------------------------------------------------
> >>>>>>
> >>>>>> So about 50 % of raw space are used, but already ~63 % of
> filesystem space are used.
> >>>>>> Is this purely from imperfect balancing?
> >>>>>> In "ceph osd df", I do indeed see OSD usages spreading from 65.02 %
> usage down to 37.12 %.
> >>>>>>
> >>>>>> We did not yet use the balancer plugin.
> >>>>>> We don't have any pre-luminous clients.
> >>>>>> In that setup, I take it that "upmap" mode would be recommended -
> correct?
> >>>>>> Any "gotchas" using that on luminous?
> >>>>>>
> >>>>>> Cheers,
> >>>>>>         Oliver
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> ceph-users mailing list
> >>>>>> ceph-users at lists.ceph.com
> >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >
> >
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181022/4696bc33/attachment.html>


More information about the ceph-users mailing list