[ceph-users] Switch to replica 3

Christian Balzer chibi at gol.com
Mon Nov 20 14:15:09 PST 2017


On Mon, 20 Nov 2017 10:35:36 -0800 Chris Taylor wrote:

> On 2017-11-20 3:39 am, Matteo Dacrema wrote:
> > Yes I mean the existing Cluster.
> > SSDs are on a fully separate pool.
> > Cluster is not busy during recovery and deep scrubs but I think it’s
> > better to limit replication in some way when switching to replica 3.
> > 
> > My question is to understand if I need to set some options parameters
> > to limit the impact of the creation of new objects.I’m also concerned
> > about disk filling up during recovery because of inefficient data
> > balancing.  
> 
> You can try using osd_recovery_sleep to slow down the backfilling so it 
> does not cause the client io to hang.
> 
> ceph tell osd.* injectargs "--osd_recovery_sleep 0.1"
> 

Which is one of the things that is version specific and we don't know the
version yet.

The above will work with Hammer and should again with Luminous, but not so
much with the unified queue bits inbetween. 

Christian

> 
> > 
> > Here osd tree
> > 
> > ID  WEIGHT    TYPE NAME        UP/DOWN REWEIGHT PRIMARY-AFFINITY
> > -10  19.69994 root ssd
> > -11   5.06998     host ceph101
> > 166   0.98999         osd.166       up  1.00000          1.00000
> > 167   1.00000         osd.167       up  1.00000          1.00000
> > 168   1.00000         osd.168       up  1.00000          1.00000
> > 169   1.07999         osd.169       up  1.00000          1.00000
> > 170   1.00000         osd.170       up  1.00000          1.00000
> > -12   4.92998     host ceph102
> > 171   0.98000         osd.171       up  1.00000          1.00000
> > 172   0.92999         osd.172       up  1.00000          1.00000
> > 173   0.98000         osd.173       up  1.00000          1.00000
> > 174   1.00000         osd.174       up  1.00000          1.00000
> > 175   1.03999         osd.175       up  1.00000          1.00000
> > -13   4.69998     host ceph103
> > 176   0.84999         osd.176       up  1.00000          1.00000
> > 177   0.84999         osd.177       up  1.00000          1.00000
> > 178   1.00000         osd.178       up  1.00000          1.00000
> > 179   1.00000         osd.179       up  1.00000          1.00000
> > 180   1.00000         osd.180       up  1.00000          1.00000
> > -14   5.00000     host ceph104
> > 181   1.00000         osd.181       up  1.00000          1.00000
> > 182   1.00000         osd.182       up  1.00000          1.00000
> > 183   1.00000         osd.183       up  1.00000          1.00000
> > 184   1.00000         osd.184       up  1.00000          1.00000
> > 185   1.00000         osd.185       up  1.00000          1.00000
> >  -1 185.19835 root default
> >  -2  18.39980     host ceph001
> >  63   0.79999         osd.63        up  1.00000          1.00000
> >  64   0.79999         osd.64        up  1.00000          1.00000
> >  65   0.79999         osd.65        up  1.00000          1.00000
> > 146   0.70000         osd.146       up  1.00000          1.00000
> > 147   0.70000         osd.147       up  1.00000          1.00000
> > 148   0.90999         osd.148       up  1.00000          1.00000
> > 149   0.79999         osd.149       up  1.00000          1.00000
> > 150   0.79999         osd.150       up  1.00000          1.00000
> > 151   0.79999         osd.151       up  1.00000          1.00000
> > 152   0.79999         osd.152       up  1.00000          1.00000
> > 153   0.70000         osd.153       up  1.00000          1.00000
> > 154   0.70000         osd.154       up  1.00000          1.00000
> > 155   0.89999         osd.155       up  1.00000          1.00000
> > 156   0.84999         osd.156       up  1.00000          1.00000
> > 157   0.70000         osd.157       up  1.00000          1.00000
> > 158   0.70000         osd.158       up  1.00000          1.00000
> > 159   0.84999         osd.159       up  1.00000          1.00000
> > 160   0.90999         osd.160       up  1.00000          1.00000
> > 161   0.90999         osd.161       up  1.00000          1.00000
> > 162   0.90999         osd.162       up  1.00000          1.00000
> > 163   0.79999         osd.163       up  1.00000          1.00000
> > 164   0.90999         osd.164       up  1.00000          1.00000
> > 165   0.64999         osd.165       up  1.00000          1.00000
> >  -3  19.41982     host ceph002
> >  23   0.79999         osd.23        up  1.00000          1.00000
> >  24   0.70000         osd.24        up  1.00000          1.00000
> >  25   0.90999         osd.25        up  1.00000          1.00000
> >  26   0.59999         osd.26        up  1.00000          1.00000
> >  27   0.95000         osd.27        up  1.00000          1.00000
> >  28   0.64999         osd.28        up  1.00000          1.00000
> >  29   0.75000         osd.29        up  1.00000          1.00000
> >  30   0.89999         osd.30        up  1.00000          1.00000
> >  31   0.90999         osd.31        up  1.00000          1.00000
> >  32   0.90999         osd.32        up  1.00000          1.00000
> >  33   0.89999         osd.33        up  1.00000          1.00000
> >  34   0.90999         osd.34        up  1.00000          1.00000
> >  35   0.90999         osd.35        up  1.00000          1.00000
> >  36   0.84999         osd.36        up  1.00000          1.00000
> >  37   0.89999         osd.37        up  1.00000          1.00000
> >  38   1.00000         osd.38        up  1.00000          1.00000
> >  39   0.79999         osd.39        up  1.00000          1.00000
> >  40   0.90999         osd.40        up  1.00000          1.00000
> >  41   0.84999         osd.41        up  1.00000          1.00000
> >  42   0.84999         osd.42        up  1.00000          1.00000
> >  43   0.90999         osd.43        up  1.00000          1.00000
> >  44   0.75000         osd.44        up  1.00000          1.00000
> >  45   0.79999         osd.45        up  1.00000          1.00000
> >  -4  16.78984     host ceph003
> >  20   1.00000         osd.20        up  1.00000          1.00000
> >  21   0.90999         osd.21        up  1.00000          1.00000
> >  22   0.90999         osd.22        up  1.00000          1.00000
> >  46   0.79999         osd.46        up  1.00000          1.00000
> >  47   0.84999         osd.47        up  1.00000          1.00000
> >  48   0.75000         osd.48        up  1.00000          1.00000
> >  49   0.90999         osd.49        up  1.00000          1.00000
> >  50   0.90999         osd.50        up  1.00000          1.00000
> >  51   0.90999         osd.51        up  1.00000          1.00000
> >  52   0.90999         osd.52        up  1.00000          1.00000
> >  53   0.70000         osd.53        up  1.00000          1.00000
> >  54   0.70000         osd.54        up  1.00000          1.00000
> >  55   0.90999         osd.55        up  1.00000          1.00000
> >  56   0.90999         osd.56        up  1.00000          1.00000
> >  57   0.75000         osd.57        up  1.00000          1.00000
> >  58   0.79999         osd.58        up  1.00000          1.00000
> >  59   0.70000         osd.59        up  1.00000          1.00000
> >  60   0.90999         osd.60        up  1.00000          1.00000
> >  61   0.75000         osd.61        up  1.00000          1.00000
> >  62   0.79999         osd.62        up  1.00000          1.00000
> >  -5  16.30986     host ceph004
> >  66   0.95000         osd.66        up  1.00000          1.00000
> >  67   0.79999         osd.67        up  1.00000          1.00000
> >  68   0.84999         osd.68        up  1.00000          1.00000
> >  69   0.84999         osd.69        up  1.00000          1.00000
> >  70   0.70000         osd.70        up  1.00000          1.00000
> >  71   0.75000         osd.71        up  1.00000          1.00000
> >  72   0.64999         osd.72        up  1.00000          1.00000
> >  73   0.90999         osd.73        up  1.00000          1.00000
> >  74   0.75000         osd.74        up  1.00000          1.00000
> >  75   0.81999         osd.75        up  1.00000          1.00000
> >  76   0.90999         osd.76        up  1.00000          1.00000
> >  77   0.79999         osd.77        up  1.00000          1.00000
> >  78   0.59999         osd.78        up  1.00000          1.00000
> >  79   1.00000         osd.79        up  1.00000          1.00000
> >  80   0.90999         osd.80        up  1.00000          1.00000
> >  81   0.79999         osd.81        up  1.00000          1.00000
> >  82   0.89999         osd.82        up  1.00000          1.00000
> >  83   0.70000         osd.83        up  1.00000          1.00000
> >  84   0.75000         osd.84        up  1.00000          1.00000
> >  85   0.90999         osd.85        up  1.00000          1.00000
> >  -6  16.71983     host ceph005
> >  86   0.75000         osd.86        up  1.00000          1.00000
> >  87   0.90999         osd.87        up  1.00000          1.00000
> >  88   0.89999         osd.88        up  1.00000          1.00000
> >  89   0.90999         osd.89        up  1.00000          1.00000
> >  90   0.90999         osd.90        up  1.00000          1.00000
> >  91   0.89999         osd.91        up  1.00000          1.00000
> >  92   0.89999         osd.92        up  1.00000          1.00000
> >  93   0.90999         osd.93        up  1.00000          1.00000
> >  94   0.84999         osd.94        up  1.00000          1.00000
> >  95   0.90999         osd.95        up  1.00000          1.00000
> >  96   0.50000         osd.96        up  1.00000          1.00000
> >  97   0.79999         osd.97        up  1.00000          1.00000
> >  98   0.59999         osd.98        up  1.00000          1.00000
> >  99   0.75000         osd.99        up  1.00000          1.00000
> > 100   1.00000         osd.100       up  1.00000          1.00000
> > 101   0.79999         osd.101       up  1.00000          1.00000
> > 102   0.90999         osd.102       up  1.00000          1.00000
> > 103   0.79999         osd.103       up  1.00000          1.00000
> > 104   0.90999         osd.104       up  1.00000          1.00000
> > 105   0.79999         osd.105       up  1.00000          1.00000
> >  -7  17.19980     host ceph006
> > 106   0.90999         osd.106       up  1.00000          1.00000
> > 107   0.89999         osd.107       up  1.00000          1.00000
> > 108   0.70000         osd.108       up  1.00000          1.00000
> > 109   0.84999         osd.109       up  1.00000          1.00000
> > 110   0.79999         osd.110       up  1.00000          1.00000
> > 111   0.79999         osd.111       up  1.00000          1.00000
> > 112   0.79999         osd.112       up  1.00000          1.00000
> > 113   0.90999         osd.113       up  1.00000          1.00000
> > 115   0.90999         osd.115       up  1.00000          1.00000
> > 116   0.90999         osd.116       up  1.00000          1.00000
> > 117   0.90999         osd.117       up  1.00000          1.00000
> > 118   0.90999         osd.118       up  1.00000          1.00000
> > 119   0.59999         osd.119       up  1.00000          1.00000
> > 120   0.90999         osd.120       up  1.00000          1.00000
> > 121   0.90999         osd.121       up  1.00000          1.00000
> > 122   0.90999         osd.122       up  1.00000          1.00000
> > 123   0.79999         osd.123       up  1.00000          1.00000
> > 124   0.75000         osd.124       up  1.00000          1.00000
> > 125   1.09999         osd.125       up  1.00000          1.00000
> > 114   0.90999         osd.114       up  1.00000          1.00000
> >  -8  16.61984     host ceph007
> > 126   0.90999         osd.126       up  1.00000          1.00000
> > 127   0.75000         osd.127       up  1.00000          1.00000
> > 128   0.84999         osd.128       up  1.00000          1.00000
> > 129   0.90999         osd.129       up  1.00000          1.00000
> > 130   0.70000         osd.130       up  1.00000          1.00000
> > 131   0.78000         osd.131       up  1.00000          1.00000
> > 132   0.79999         osd.132       up  1.00000          1.00000
> > 133   0.54999         osd.133       up  1.00000          1.00000
> > 134   0.79999         osd.134       up  1.00000          1.00000
> > 135   0.87000         osd.135       up  1.00000          1.00000
> > 136   0.90999         osd.136       up  1.00000          1.00000
> > 137   0.90999         osd.137       up  1.00000          1.00000
> > 138   0.90999         osd.138       up  1.00000          1.00000
> > 139   0.89999         osd.139       up  1.00000          1.00000
> > 140   0.75000         osd.140       up  1.00000          1.00000
> > 141   0.75000         osd.141       up  1.00000          1.00000
> > 142   0.90999         osd.142       up  1.00000          1.00000
> > 143   0.84999         osd.143       up  1.00000          1.00000
> > 145   0.90999         osd.145       up  1.00000          1.00000
> > 144   0.89999         osd.144       up  1.00000          1.00000
> >  -9  16.93982     host ceph008
> >   0   0.90999         osd.0         up  1.00000          1.00000
> >   1   0.70000         osd.1         up  1.00000          1.00000
> >   2   0.90999         osd.2         up  1.00000          1.00000
> >   3   0.90999         osd.3         up  1.00000          1.00000
> >   4   0.79999         osd.4         up  1.00000          1.00000
> >   5   0.64999         osd.5         up  1.00000          1.00000
> >   6   0.64999         osd.6         up  1.00000          1.00000
> >   7   0.90999         osd.7         up  1.00000          1.00000
> >   8   0.79999         osd.8         up  1.00000          1.00000
> >   9   0.90999         osd.9         up  1.00000          1.00000
> >  10   0.75000         osd.10        up  1.00000          1.00000
> >  11   0.90999         osd.11        up  1.00000          1.00000
> >  12   0.89999         osd.12        up  1.00000          1.00000
> >  13   0.75000         osd.13        up  1.00000          1.00000
> >  14   1.09999         osd.14        up  1.00000          1.00000
> >  15   0.84999         osd.15        up  1.00000          1.00000
> >  16   0.90999         osd.16        up  1.00000          1.00000
> >  17   0.90999         osd.17        up  1.00000          1.00000
> >  18   0.79999         osd.18        up  1.00000          1.00000
> >  19   0.90999         osd.19        up  1.00000          1.00000
> > -15  28.79987     host ceph009
> > 186   1.00000         osd.186       up  1.00000          1.00000
> > 187   1.00000         osd.187       up  1.00000          1.00000
> > 188   1.00000         osd.188       up  1.00000          1.00000
> > 189   1.00000         osd.189       up  1.00000          1.00000
> > 190   1.00000         osd.190       up  1.00000          1.00000
> > 191   1.00000         osd.191       up  1.00000          1.00000
> > 192   1.00000         osd.192       up  1.00000          1.00000
> > 193   1.00000         osd.193       up  1.00000          1.00000
> > 194   1.00000         osd.194       up  1.00000          1.00000
> > 195   1.79999         osd.195       up  1.00000          1.00000
> > 196   1.79999         osd.196       up  1.00000          1.00000
> > 197   1.79999         osd.197       up  1.00000          1.00000
> > 198   1.79999         osd.198       up  1.00000          1.00000
> > 199   1.79999         osd.199       up  1.00000          1.00000
> > 200   1.79999         osd.200       up  1.00000          1.00000
> > 201   1.79999         osd.201       up  1.00000          1.00000
> > 202   1.79999         osd.202       up  1.00000          1.00000
> > 203   1.79999         osd.203       up  1.00000          1.00000
> > 204   1.79999         osd.204       up  1.00000          1.00000
> > 205   1.79999         osd.205       up  1.00000          1.00000
> > -16  17.99988     host ceph010
> > 206   1.79999         osd.206       up  1.00000          1.00000
> > 207   1.79999         osd.207       up  1.00000          1.00000
> > 208   1.79999         osd.208       up  1.00000          1.00000
> > 209   1.79999         osd.209       up  1.00000          1.00000
> > 210   1.79999         osd.210       up  1.00000          1.00000
> > 211   1.79999         osd.211       up  1.00000          1.00000
> > 212   1.79999         osd.212       up  1.00000          1.00000
> > 213   1.79999         osd.213       up  1.00000          1.00000
> > 214   1.79999         osd.214       up  1.00000          1.00000
> > 215   1.79999         osd.215       up  1.00000          1.00000
> > 
> > 
> >   
> >> Il giorno 20 nov 2017, alle ore 12:17, Christian Balzer 
> >> <chibi at gol.com> ha scritto:
> >> 
> >> 
> >> Hello,
> >> 
> >> On Mon, 20 Nov 2017 11:56:31 +0100 Matteo Dacrema wrote:
> >>   
> >>> Hi,
> >>> 
> >>> I need to switch a cluster of over 200 OSDs from replica 2 to replica 
> >>> 3  
> >> I presume this means the existing cluster and not adding 100 OSDs...
> >>   
> >>> There are two different crush maps for HDD and SSDs also mapped to 
> >>> two different pools.
> >>> 
> >>> Is there a best practice to use? Can this provoke troubles?
> >>>   
> >> Are your SSDs a cache-tier or are they a fully separate pool?
> >> 
> >> As for troubles, how busy is your cluster during the recovery of 
> >> failed
> >> OSDs or deep scrubs?
> >> 
> >> There are 2 things to consider here:
> >> 
> >> 1. The re-balancing and additional replication of all the data, which 
> >> you
> >> can control/ease by the various knobs present. Ceph version matters to
> >> which are relevant/useful. It shouldn't impact things too much, unless
> >> your cluster was at the very edge of it's capacity anyway.
> >> 
> >> 2. The little detail that after 1) is done, your cluster will be
> >> noticeably slower than before, especially in the latency department.
> >> In short, you don't just need to have the disk space to go 3x, but 
> >> also
> >> enough IOPS/bandwidth reserves.
> >> 
> >> Christian
> >>   
> >>> Thank you
> >>> Matteo
> >>> _______________________________________________
> >>> ceph-users mailing list
> >>> ceph-users at lists.ceph.com
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>>   
> >> 
> >> 
> >> --
> >> Christian Balzer        Network/Systems Engineer
> >> chibi at gol.com   	Rakuten Communications
> >> 
> >> --
> >> Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato 
> >> non infetto.
> >> Seguire il link qui sotto per segnalarlo come spam:
> >> http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=524464756E.A33EC
> >> 
> >>   
> > 
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users at lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com  
> 


-- 
Christian Balzer        Network/Systems Engineer                
chibi at gol.com   	Rakuten Communications


More information about the ceph-users mailing list