[ceph-users] Switch to replica 3

Chris Taylor ctaylor at eyonic.com
Mon Nov 20 10:35:36 PST 2017


On 2017-11-20 3:39 am, Matteo Dacrema wrote:
> Yes I mean the existing Cluster.
> SSDs are on a fully separate pool.
> Cluster is not busy during recovery and deep scrubs but I think it’s
> better to limit replication in some way when switching to replica 3.
> 
> My question is to understand if I need to set some options parameters
> to limit the impact of the creation of new objects.I’m also concerned
> about disk filling up during recovery because of inefficient data
> balancing.

You can try using osd_recovery_sleep to slow down the backfilling so it 
does not cause the client io to hang.

ceph tell osd.* injectargs "--osd_recovery_sleep 0.1"


> 
> Here osd tree
> 
> ID  WEIGHT    TYPE NAME        UP/DOWN REWEIGHT PRIMARY-AFFINITY
> -10  19.69994 root ssd
> -11   5.06998     host ceph101
> 166   0.98999         osd.166       up  1.00000          1.00000
> 167   1.00000         osd.167       up  1.00000          1.00000
> 168   1.00000         osd.168       up  1.00000          1.00000
> 169   1.07999         osd.169       up  1.00000          1.00000
> 170   1.00000         osd.170       up  1.00000          1.00000
> -12   4.92998     host ceph102
> 171   0.98000         osd.171       up  1.00000          1.00000
> 172   0.92999         osd.172       up  1.00000          1.00000
> 173   0.98000         osd.173       up  1.00000          1.00000
> 174   1.00000         osd.174       up  1.00000          1.00000
> 175   1.03999         osd.175       up  1.00000          1.00000
> -13   4.69998     host ceph103
> 176   0.84999         osd.176       up  1.00000          1.00000
> 177   0.84999         osd.177       up  1.00000          1.00000
> 178   1.00000         osd.178       up  1.00000          1.00000
> 179   1.00000         osd.179       up  1.00000          1.00000
> 180   1.00000         osd.180       up  1.00000          1.00000
> -14   5.00000     host ceph104
> 181   1.00000         osd.181       up  1.00000          1.00000
> 182   1.00000         osd.182       up  1.00000          1.00000
> 183   1.00000         osd.183       up  1.00000          1.00000
> 184   1.00000         osd.184       up  1.00000          1.00000
> 185   1.00000         osd.185       up  1.00000          1.00000
>  -1 185.19835 root default
>  -2  18.39980     host ceph001
>  63   0.79999         osd.63        up  1.00000          1.00000
>  64   0.79999         osd.64        up  1.00000          1.00000
>  65   0.79999         osd.65        up  1.00000          1.00000
> 146   0.70000         osd.146       up  1.00000          1.00000
> 147   0.70000         osd.147       up  1.00000          1.00000
> 148   0.90999         osd.148       up  1.00000          1.00000
> 149   0.79999         osd.149       up  1.00000          1.00000
> 150   0.79999         osd.150       up  1.00000          1.00000
> 151   0.79999         osd.151       up  1.00000          1.00000
> 152   0.79999         osd.152       up  1.00000          1.00000
> 153   0.70000         osd.153       up  1.00000          1.00000
> 154   0.70000         osd.154       up  1.00000          1.00000
> 155   0.89999         osd.155       up  1.00000          1.00000
> 156   0.84999         osd.156       up  1.00000          1.00000
> 157   0.70000         osd.157       up  1.00000          1.00000
> 158   0.70000         osd.158       up  1.00000          1.00000
> 159   0.84999         osd.159       up  1.00000          1.00000
> 160   0.90999         osd.160       up  1.00000          1.00000
> 161   0.90999         osd.161       up  1.00000          1.00000
> 162   0.90999         osd.162       up  1.00000          1.00000
> 163   0.79999         osd.163       up  1.00000          1.00000
> 164   0.90999         osd.164       up  1.00000          1.00000
> 165   0.64999         osd.165       up  1.00000          1.00000
>  -3  19.41982     host ceph002
>  23   0.79999         osd.23        up  1.00000          1.00000
>  24   0.70000         osd.24        up  1.00000          1.00000
>  25   0.90999         osd.25        up  1.00000          1.00000
>  26   0.59999         osd.26        up  1.00000          1.00000
>  27   0.95000         osd.27        up  1.00000          1.00000
>  28   0.64999         osd.28        up  1.00000          1.00000
>  29   0.75000         osd.29        up  1.00000          1.00000
>  30   0.89999         osd.30        up  1.00000          1.00000
>  31   0.90999         osd.31        up  1.00000          1.00000
>  32   0.90999         osd.32        up  1.00000          1.00000
>  33   0.89999         osd.33        up  1.00000          1.00000
>  34   0.90999         osd.34        up  1.00000          1.00000
>  35   0.90999         osd.35        up  1.00000          1.00000
>  36   0.84999         osd.36        up  1.00000          1.00000
>  37   0.89999         osd.37        up  1.00000          1.00000
>  38   1.00000         osd.38        up  1.00000          1.00000
>  39   0.79999         osd.39        up  1.00000          1.00000
>  40   0.90999         osd.40        up  1.00000          1.00000
>  41   0.84999         osd.41        up  1.00000          1.00000
>  42   0.84999         osd.42        up  1.00000          1.00000
>  43   0.90999         osd.43        up  1.00000          1.00000
>  44   0.75000         osd.44        up  1.00000          1.00000
>  45   0.79999         osd.45        up  1.00000          1.00000
>  -4  16.78984     host ceph003
>  20   1.00000         osd.20        up  1.00000          1.00000
>  21   0.90999         osd.21        up  1.00000          1.00000
>  22   0.90999         osd.22        up  1.00000          1.00000
>  46   0.79999         osd.46        up  1.00000          1.00000
>  47   0.84999         osd.47        up  1.00000          1.00000
>  48   0.75000         osd.48        up  1.00000          1.00000
>  49   0.90999         osd.49        up  1.00000          1.00000
>  50   0.90999         osd.50        up  1.00000          1.00000
>  51   0.90999         osd.51        up  1.00000          1.00000
>  52   0.90999         osd.52        up  1.00000          1.00000
>  53   0.70000         osd.53        up  1.00000          1.00000
>  54   0.70000         osd.54        up  1.00000          1.00000
>  55   0.90999         osd.55        up  1.00000          1.00000
>  56   0.90999         osd.56        up  1.00000          1.00000
>  57   0.75000         osd.57        up  1.00000          1.00000
>  58   0.79999         osd.58        up  1.00000          1.00000
>  59   0.70000         osd.59        up  1.00000          1.00000
>  60   0.90999         osd.60        up  1.00000          1.00000
>  61   0.75000         osd.61        up  1.00000          1.00000
>  62   0.79999         osd.62        up  1.00000          1.00000
>  -5  16.30986     host ceph004
>  66   0.95000         osd.66        up  1.00000          1.00000
>  67   0.79999         osd.67        up  1.00000          1.00000
>  68   0.84999         osd.68        up  1.00000          1.00000
>  69   0.84999         osd.69        up  1.00000          1.00000
>  70   0.70000         osd.70        up  1.00000          1.00000
>  71   0.75000         osd.71        up  1.00000          1.00000
>  72   0.64999         osd.72        up  1.00000          1.00000
>  73   0.90999         osd.73        up  1.00000          1.00000
>  74   0.75000         osd.74        up  1.00000          1.00000
>  75   0.81999         osd.75        up  1.00000          1.00000
>  76   0.90999         osd.76        up  1.00000          1.00000
>  77   0.79999         osd.77        up  1.00000          1.00000
>  78   0.59999         osd.78        up  1.00000          1.00000
>  79   1.00000         osd.79        up  1.00000          1.00000
>  80   0.90999         osd.80        up  1.00000          1.00000
>  81   0.79999         osd.81        up  1.00000          1.00000
>  82   0.89999         osd.82        up  1.00000          1.00000
>  83   0.70000         osd.83        up  1.00000          1.00000
>  84   0.75000         osd.84        up  1.00000          1.00000
>  85   0.90999         osd.85        up  1.00000          1.00000
>  -6  16.71983     host ceph005
>  86   0.75000         osd.86        up  1.00000          1.00000
>  87   0.90999         osd.87        up  1.00000          1.00000
>  88   0.89999         osd.88        up  1.00000          1.00000
>  89   0.90999         osd.89        up  1.00000          1.00000
>  90   0.90999         osd.90        up  1.00000          1.00000
>  91   0.89999         osd.91        up  1.00000          1.00000
>  92   0.89999         osd.92        up  1.00000          1.00000
>  93   0.90999         osd.93        up  1.00000          1.00000
>  94   0.84999         osd.94        up  1.00000          1.00000
>  95   0.90999         osd.95        up  1.00000          1.00000
>  96   0.50000         osd.96        up  1.00000          1.00000
>  97   0.79999         osd.97        up  1.00000          1.00000
>  98   0.59999         osd.98        up  1.00000          1.00000
>  99   0.75000         osd.99        up  1.00000          1.00000
> 100   1.00000         osd.100       up  1.00000          1.00000
> 101   0.79999         osd.101       up  1.00000          1.00000
> 102   0.90999         osd.102       up  1.00000          1.00000
> 103   0.79999         osd.103       up  1.00000          1.00000
> 104   0.90999         osd.104       up  1.00000          1.00000
> 105   0.79999         osd.105       up  1.00000          1.00000
>  -7  17.19980     host ceph006
> 106   0.90999         osd.106       up  1.00000          1.00000
> 107   0.89999         osd.107       up  1.00000          1.00000
> 108   0.70000         osd.108       up  1.00000          1.00000
> 109   0.84999         osd.109       up  1.00000          1.00000
> 110   0.79999         osd.110       up  1.00000          1.00000
> 111   0.79999         osd.111       up  1.00000          1.00000
> 112   0.79999         osd.112       up  1.00000          1.00000
> 113   0.90999         osd.113       up  1.00000          1.00000
> 115   0.90999         osd.115       up  1.00000          1.00000
> 116   0.90999         osd.116       up  1.00000          1.00000
> 117   0.90999         osd.117       up  1.00000          1.00000
> 118   0.90999         osd.118       up  1.00000          1.00000
> 119   0.59999         osd.119       up  1.00000          1.00000
> 120   0.90999         osd.120       up  1.00000          1.00000
> 121   0.90999         osd.121       up  1.00000          1.00000
> 122   0.90999         osd.122       up  1.00000          1.00000
> 123   0.79999         osd.123       up  1.00000          1.00000
> 124   0.75000         osd.124       up  1.00000          1.00000
> 125   1.09999         osd.125       up  1.00000          1.00000
> 114   0.90999         osd.114       up  1.00000          1.00000
>  -8  16.61984     host ceph007
> 126   0.90999         osd.126       up  1.00000          1.00000
> 127   0.75000         osd.127       up  1.00000          1.00000
> 128   0.84999         osd.128       up  1.00000          1.00000
> 129   0.90999         osd.129       up  1.00000          1.00000
> 130   0.70000         osd.130       up  1.00000          1.00000
> 131   0.78000         osd.131       up  1.00000          1.00000
> 132   0.79999         osd.132       up  1.00000          1.00000
> 133   0.54999         osd.133       up  1.00000          1.00000
> 134   0.79999         osd.134       up  1.00000          1.00000
> 135   0.87000         osd.135       up  1.00000          1.00000
> 136   0.90999         osd.136       up  1.00000          1.00000
> 137   0.90999         osd.137       up  1.00000          1.00000
> 138   0.90999         osd.138       up  1.00000          1.00000
> 139   0.89999         osd.139       up  1.00000          1.00000
> 140   0.75000         osd.140       up  1.00000          1.00000
> 141   0.75000         osd.141       up  1.00000          1.00000
> 142   0.90999         osd.142       up  1.00000          1.00000
> 143   0.84999         osd.143       up  1.00000          1.00000
> 145   0.90999         osd.145       up  1.00000          1.00000
> 144   0.89999         osd.144       up  1.00000          1.00000
>  -9  16.93982     host ceph008
>   0   0.90999         osd.0         up  1.00000          1.00000
>   1   0.70000         osd.1         up  1.00000          1.00000
>   2   0.90999         osd.2         up  1.00000          1.00000
>   3   0.90999         osd.3         up  1.00000          1.00000
>   4   0.79999         osd.4         up  1.00000          1.00000
>   5   0.64999         osd.5         up  1.00000          1.00000
>   6   0.64999         osd.6         up  1.00000          1.00000
>   7   0.90999         osd.7         up  1.00000          1.00000
>   8   0.79999         osd.8         up  1.00000          1.00000
>   9   0.90999         osd.9         up  1.00000          1.00000
>  10   0.75000         osd.10        up  1.00000          1.00000
>  11   0.90999         osd.11        up  1.00000          1.00000
>  12   0.89999         osd.12        up  1.00000          1.00000
>  13   0.75000         osd.13        up  1.00000          1.00000
>  14   1.09999         osd.14        up  1.00000          1.00000
>  15   0.84999         osd.15        up  1.00000          1.00000
>  16   0.90999         osd.16        up  1.00000          1.00000
>  17   0.90999         osd.17        up  1.00000          1.00000
>  18   0.79999         osd.18        up  1.00000          1.00000
>  19   0.90999         osd.19        up  1.00000          1.00000
> -15  28.79987     host ceph009
> 186   1.00000         osd.186       up  1.00000          1.00000
> 187   1.00000         osd.187       up  1.00000          1.00000
> 188   1.00000         osd.188       up  1.00000          1.00000
> 189   1.00000         osd.189       up  1.00000          1.00000
> 190   1.00000         osd.190       up  1.00000          1.00000
> 191   1.00000         osd.191       up  1.00000          1.00000
> 192   1.00000         osd.192       up  1.00000          1.00000
> 193   1.00000         osd.193       up  1.00000          1.00000
> 194   1.00000         osd.194       up  1.00000          1.00000
> 195   1.79999         osd.195       up  1.00000          1.00000
> 196   1.79999         osd.196       up  1.00000          1.00000
> 197   1.79999         osd.197       up  1.00000          1.00000
> 198   1.79999         osd.198       up  1.00000          1.00000
> 199   1.79999         osd.199       up  1.00000          1.00000
> 200   1.79999         osd.200       up  1.00000          1.00000
> 201   1.79999         osd.201       up  1.00000          1.00000
> 202   1.79999         osd.202       up  1.00000          1.00000
> 203   1.79999         osd.203       up  1.00000          1.00000
> 204   1.79999         osd.204       up  1.00000          1.00000
> 205   1.79999         osd.205       up  1.00000          1.00000
> -16  17.99988     host ceph010
> 206   1.79999         osd.206       up  1.00000          1.00000
> 207   1.79999         osd.207       up  1.00000          1.00000
> 208   1.79999         osd.208       up  1.00000          1.00000
> 209   1.79999         osd.209       up  1.00000          1.00000
> 210   1.79999         osd.210       up  1.00000          1.00000
> 211   1.79999         osd.211       up  1.00000          1.00000
> 212   1.79999         osd.212       up  1.00000          1.00000
> 213   1.79999         osd.213       up  1.00000          1.00000
> 214   1.79999         osd.214       up  1.00000          1.00000
> 215   1.79999         osd.215       up  1.00000          1.00000
> 
> 
> 
>> Il giorno 20 nov 2017, alle ore 12:17, Christian Balzer 
>> <chibi at gol.com> ha scritto:
>> 
>> 
>> Hello,
>> 
>> On Mon, 20 Nov 2017 11:56:31 +0100 Matteo Dacrema wrote:
>> 
>>> Hi,
>>> 
>>> I need to switch a cluster of over 200 OSDs from replica 2 to replica 
>>> 3
>> I presume this means the existing cluster and not adding 100 OSDs...
>> 
>>> There are two different crush maps for HDD and SSDs also mapped to 
>>> two different pools.
>>> 
>>> Is there a best practice to use? Can this provoke troubles?
>>> 
>> Are your SSDs a cache-tier or are they a fully separate pool?
>> 
>> As for troubles, how busy is your cluster during the recovery of 
>> failed
>> OSDs or deep scrubs?
>> 
>> There are 2 things to consider here:
>> 
>> 1. The re-balancing and additional replication of all the data, which 
>> you
>> can control/ease by the various knobs present. Ceph version matters to
>> which are relevant/useful. It shouldn't impact things too much, unless
>> your cluster was at the very edge of it's capacity anyway.
>> 
>> 2. The little detail that after 1) is done, your cluster will be
>> noticeably slower than before, especially in the latency department.
>> In short, you don't just need to have the disk space to go 3x, but 
>> also
>> enough IOPS/bandwidth reserves.
>> 
>> Christian
>> 
>>> Thank you
>>> Matteo
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users at lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> 
>> 
>> 
>> --
>> Christian Balzer        Network/Systems Engineer
>> chibi at gol.com   	Rakuten Communications
>> 
>> --
>> Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato 
>> non infetto.
>> Seguire il link qui sotto per segnalarlo come spam:
>> http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=524464756E.A33EC
>> 
>> 
> 
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


More information about the ceph-users mailing list