[ceph-users] Switch to replica 3

Matteo Dacrema mdacrema at enter.eu
Mon Nov 20 23:41:51 PST 2017


Ok, thank you guys

The version is 10.2.10

Matteo

> Il giorno 20 nov 2017, alle ore 23:15, Christian Balzer <chibi at gol.com> ha scritto:
> 
> On Mon, 20 Nov 2017 10:35:36 -0800 Chris Taylor wrote:
> 
>> On 2017-11-20 3:39 am, Matteo Dacrema wrote:
>>> Yes I mean the existing Cluster.
>>> SSDs are on a fully separate pool.
>>> Cluster is not busy during recovery and deep scrubs but I think it’s
>>> better to limit replication in some way when switching to replica 3.
>>> 
>>> My question is to understand if I need to set some options parameters
>>> to limit the impact of the creation of new objects.I’m also concerned
>>> about disk filling up during recovery because of inefficient data
>>> balancing.  
>> 
>> You can try using osd_recovery_sleep to slow down the backfilling so it 
>> does not cause the client io to hang.
>> 
>> ceph tell osd.* injectargs "--osd_recovery_sleep 0.1"
>> 
> 
> Which is one of the things that is version specific and we don't know the
> version yet.
> 
> The above will work with Hammer and should again with Luminous, but not so
> much with the unified queue bits inbetween. 
> 
> Christian
> 
>> 
>>> 
>>> Here osd tree
>>> 
>>> ID  WEIGHT    TYPE NAME        UP/DOWN REWEIGHT PRIMARY-AFFINITY
>>> -10  19.69994 root ssd
>>> -11   5.06998     host ceph101
>>> 166   0.98999         osd.166       up  1.00000          1.00000
>>> 167   1.00000         osd.167       up  1.00000          1.00000
>>> 168   1.00000         osd.168       up  1.00000          1.00000
>>> 169   1.07999         osd.169       up  1.00000          1.00000
>>> 170   1.00000         osd.170       up  1.00000          1.00000
>>> -12   4.92998     host ceph102
>>> 171   0.98000         osd.171       up  1.00000          1.00000
>>> 172   0.92999         osd.172       up  1.00000          1.00000
>>> 173   0.98000         osd.173       up  1.00000          1.00000
>>> 174   1.00000         osd.174       up  1.00000          1.00000
>>> 175   1.03999         osd.175       up  1.00000          1.00000
>>> -13   4.69998     host ceph103
>>> 176   0.84999         osd.176       up  1.00000          1.00000
>>> 177   0.84999         osd.177       up  1.00000          1.00000
>>> 178   1.00000         osd.178       up  1.00000          1.00000
>>> 179   1.00000         osd.179       up  1.00000          1.00000
>>> 180   1.00000         osd.180       up  1.00000          1.00000
>>> -14   5.00000     host ceph104
>>> 181   1.00000         osd.181       up  1.00000          1.00000
>>> 182   1.00000         osd.182       up  1.00000          1.00000
>>> 183   1.00000         osd.183       up  1.00000          1.00000
>>> 184   1.00000         osd.184       up  1.00000          1.00000
>>> 185   1.00000         osd.185       up  1.00000          1.00000
>>> -1 185.19835 root default
>>> -2  18.39980     host ceph001
>>> 63   0.79999         osd.63        up  1.00000          1.00000
>>> 64   0.79999         osd.64        up  1.00000          1.00000
>>> 65   0.79999         osd.65        up  1.00000          1.00000
>>> 146   0.70000         osd.146       up  1.00000          1.00000
>>> 147   0.70000         osd.147       up  1.00000          1.00000
>>> 148   0.90999         osd.148       up  1.00000          1.00000
>>> 149   0.79999         osd.149       up  1.00000          1.00000
>>> 150   0.79999         osd.150       up  1.00000          1.00000
>>> 151   0.79999         osd.151       up  1.00000          1.00000
>>> 152   0.79999         osd.152       up  1.00000          1.00000
>>> 153   0.70000         osd.153       up  1.00000          1.00000
>>> 154   0.70000         osd.154       up  1.00000          1.00000
>>> 155   0.89999         osd.155       up  1.00000          1.00000
>>> 156   0.84999         osd.156       up  1.00000          1.00000
>>> 157   0.70000         osd.157       up  1.00000          1.00000
>>> 158   0.70000         osd.158       up  1.00000          1.00000
>>> 159   0.84999         osd.159       up  1.00000          1.00000
>>> 160   0.90999         osd.160       up  1.00000          1.00000
>>> 161   0.90999         osd.161       up  1.00000          1.00000
>>> 162   0.90999         osd.162       up  1.00000          1.00000
>>> 163   0.79999         osd.163       up  1.00000          1.00000
>>> 164   0.90999         osd.164       up  1.00000          1.00000
>>> 165   0.64999         osd.165       up  1.00000          1.00000
>>> -3  19.41982     host ceph002
>>> 23   0.79999         osd.23        up  1.00000          1.00000
>>> 24   0.70000         osd.24        up  1.00000          1.00000
>>> 25   0.90999         osd.25        up  1.00000          1.00000
>>> 26   0.59999         osd.26        up  1.00000          1.00000
>>> 27   0.95000         osd.27        up  1.00000          1.00000
>>> 28   0.64999         osd.28        up  1.00000          1.00000
>>> 29   0.75000         osd.29        up  1.00000          1.00000
>>> 30   0.89999         osd.30        up  1.00000          1.00000
>>> 31   0.90999         osd.31        up  1.00000          1.00000
>>> 32   0.90999         osd.32        up  1.00000          1.00000
>>> 33   0.89999         osd.33        up  1.00000          1.00000
>>> 34   0.90999         osd.34        up  1.00000          1.00000
>>> 35   0.90999         osd.35        up  1.00000          1.00000
>>> 36   0.84999         osd.36        up  1.00000          1.00000
>>> 37   0.89999         osd.37        up  1.00000          1.00000
>>> 38   1.00000         osd.38        up  1.00000          1.00000
>>> 39   0.79999         osd.39        up  1.00000          1.00000
>>> 40   0.90999         osd.40        up  1.00000          1.00000
>>> 41   0.84999         osd.41        up  1.00000          1.00000
>>> 42   0.84999         osd.42        up  1.00000          1.00000
>>> 43   0.90999         osd.43        up  1.00000          1.00000
>>> 44   0.75000         osd.44        up  1.00000          1.00000
>>> 45   0.79999         osd.45        up  1.00000          1.00000
>>> -4  16.78984     host ceph003
>>> 20   1.00000         osd.20        up  1.00000          1.00000
>>> 21   0.90999         osd.21        up  1.00000          1.00000
>>> 22   0.90999         osd.22        up  1.00000          1.00000
>>> 46   0.79999         osd.46        up  1.00000          1.00000
>>> 47   0.84999         osd.47        up  1.00000          1.00000
>>> 48   0.75000         osd.48        up  1.00000          1.00000
>>> 49   0.90999         osd.49        up  1.00000          1.00000
>>> 50   0.90999         osd.50        up  1.00000          1.00000
>>> 51   0.90999         osd.51        up  1.00000          1.00000
>>> 52   0.90999         osd.52        up  1.00000          1.00000
>>> 53   0.70000         osd.53        up  1.00000          1.00000
>>> 54   0.70000         osd.54        up  1.00000          1.00000
>>> 55   0.90999         osd.55        up  1.00000          1.00000
>>> 56   0.90999         osd.56        up  1.00000          1.00000
>>> 57   0.75000         osd.57        up  1.00000          1.00000
>>> 58   0.79999         osd.58        up  1.00000          1.00000
>>> 59   0.70000         osd.59        up  1.00000          1.00000
>>> 60   0.90999         osd.60        up  1.00000          1.00000
>>> 61   0.75000         osd.61        up  1.00000          1.00000
>>> 62   0.79999         osd.62        up  1.00000          1.00000
>>> -5  16.30986     host ceph004
>>> 66   0.95000         osd.66        up  1.00000          1.00000
>>> 67   0.79999         osd.67        up  1.00000          1.00000
>>> 68   0.84999         osd.68        up  1.00000          1.00000
>>> 69   0.84999         osd.69        up  1.00000          1.00000
>>> 70   0.70000         osd.70        up  1.00000          1.00000
>>> 71   0.75000         osd.71        up  1.00000          1.00000
>>> 72   0.64999         osd.72        up  1.00000          1.00000
>>> 73   0.90999         osd.73        up  1.00000          1.00000
>>> 74   0.75000         osd.74        up  1.00000          1.00000
>>> 75   0.81999         osd.75        up  1.00000          1.00000
>>> 76   0.90999         osd.76        up  1.00000          1.00000
>>> 77   0.79999         osd.77        up  1.00000          1.00000
>>> 78   0.59999         osd.78        up  1.00000          1.00000
>>> 79   1.00000         osd.79        up  1.00000          1.00000
>>> 80   0.90999         osd.80        up  1.00000          1.00000
>>> 81   0.79999         osd.81        up  1.00000          1.00000
>>> 82   0.89999         osd.82        up  1.00000          1.00000
>>> 83   0.70000         osd.83        up  1.00000          1.00000
>>> 84   0.75000         osd.84        up  1.00000          1.00000
>>> 85   0.90999         osd.85        up  1.00000          1.00000
>>> -6  16.71983     host ceph005
>>> 86   0.75000         osd.86        up  1.00000          1.00000
>>> 87   0.90999         osd.87        up  1.00000          1.00000
>>> 88   0.89999         osd.88        up  1.00000          1.00000
>>> 89   0.90999         osd.89        up  1.00000          1.00000
>>> 90   0.90999         osd.90        up  1.00000          1.00000
>>> 91   0.89999         osd.91        up  1.00000          1.00000
>>> 92   0.89999         osd.92        up  1.00000          1.00000
>>> 93   0.90999         osd.93        up  1.00000          1.00000
>>> 94   0.84999         osd.94        up  1.00000          1.00000
>>> 95   0.90999         osd.95        up  1.00000          1.00000
>>> 96   0.50000         osd.96        up  1.00000          1.00000
>>> 97   0.79999         osd.97        up  1.00000          1.00000
>>> 98   0.59999         osd.98        up  1.00000          1.00000
>>> 99   0.75000         osd.99        up  1.00000          1.00000
>>> 100   1.00000         osd.100       up  1.00000          1.00000
>>> 101   0.79999         osd.101       up  1.00000          1.00000
>>> 102   0.90999         osd.102       up  1.00000          1.00000
>>> 103   0.79999         osd.103       up  1.00000          1.00000
>>> 104   0.90999         osd.104       up  1.00000          1.00000
>>> 105   0.79999         osd.105       up  1.00000          1.00000
>>> -7  17.19980     host ceph006
>>> 106   0.90999         osd.106       up  1.00000          1.00000
>>> 107   0.89999         osd.107       up  1.00000          1.00000
>>> 108   0.70000         osd.108       up  1.00000          1.00000
>>> 109   0.84999         osd.109       up  1.00000          1.00000
>>> 110   0.79999         osd.110       up  1.00000          1.00000
>>> 111   0.79999         osd.111       up  1.00000          1.00000
>>> 112   0.79999         osd.112       up  1.00000          1.00000
>>> 113   0.90999         osd.113       up  1.00000          1.00000
>>> 115   0.90999         osd.115       up  1.00000          1.00000
>>> 116   0.90999         osd.116       up  1.00000          1.00000
>>> 117   0.90999         osd.117       up  1.00000          1.00000
>>> 118   0.90999         osd.118       up  1.00000          1.00000
>>> 119   0.59999         osd.119       up  1.00000          1.00000
>>> 120   0.90999         osd.120       up  1.00000          1.00000
>>> 121   0.90999         osd.121       up  1.00000          1.00000
>>> 122   0.90999         osd.122       up  1.00000          1.00000
>>> 123   0.79999         osd.123       up  1.00000          1.00000
>>> 124   0.75000         osd.124       up  1.00000          1.00000
>>> 125   1.09999         osd.125       up  1.00000          1.00000
>>> 114   0.90999         osd.114       up  1.00000          1.00000
>>> -8  16.61984     host ceph007
>>> 126   0.90999         osd.126       up  1.00000          1.00000
>>> 127   0.75000         osd.127       up  1.00000          1.00000
>>> 128   0.84999         osd.128       up  1.00000          1.00000
>>> 129   0.90999         osd.129       up  1.00000          1.00000
>>> 130   0.70000         osd.130       up  1.00000          1.00000
>>> 131   0.78000         osd.131       up  1.00000          1.00000
>>> 132   0.79999         osd.132       up  1.00000          1.00000
>>> 133   0.54999         osd.133       up  1.00000          1.00000
>>> 134   0.79999         osd.134       up  1.00000          1.00000
>>> 135   0.87000         osd.135       up  1.00000          1.00000
>>> 136   0.90999         osd.136       up  1.00000          1.00000
>>> 137   0.90999         osd.137       up  1.00000          1.00000
>>> 138   0.90999         osd.138       up  1.00000          1.00000
>>> 139   0.89999         osd.139       up  1.00000          1.00000
>>> 140   0.75000         osd.140       up  1.00000          1.00000
>>> 141   0.75000         osd.141       up  1.00000          1.00000
>>> 142   0.90999         osd.142       up  1.00000          1.00000
>>> 143   0.84999         osd.143       up  1.00000          1.00000
>>> 145   0.90999         osd.145       up  1.00000          1.00000
>>> 144   0.89999         osd.144       up  1.00000          1.00000
>>> -9  16.93982     host ceph008
>>>  0   0.90999         osd.0         up  1.00000          1.00000
>>>  1   0.70000         osd.1         up  1.00000          1.00000
>>>  2   0.90999         osd.2         up  1.00000          1.00000
>>>  3   0.90999         osd.3         up  1.00000          1.00000
>>>  4   0.79999         osd.4         up  1.00000          1.00000
>>>  5   0.64999         osd.5         up  1.00000          1.00000
>>>  6   0.64999         osd.6         up  1.00000          1.00000
>>>  7   0.90999         osd.7         up  1.00000          1.00000
>>>  8   0.79999         osd.8         up  1.00000          1.00000
>>>  9   0.90999         osd.9         up  1.00000          1.00000
>>> 10   0.75000         osd.10        up  1.00000          1.00000
>>> 11   0.90999         osd.11        up  1.00000          1.00000
>>> 12   0.89999         osd.12        up  1.00000          1.00000
>>> 13   0.75000         osd.13        up  1.00000          1.00000
>>> 14   1.09999         osd.14        up  1.00000          1.00000
>>> 15   0.84999         osd.15        up  1.00000          1.00000
>>> 16   0.90999         osd.16        up  1.00000          1.00000
>>> 17   0.90999         osd.17        up  1.00000          1.00000
>>> 18   0.79999         osd.18        up  1.00000          1.00000
>>> 19   0.90999         osd.19        up  1.00000          1.00000
>>> -15  28.79987     host ceph009
>>> 186   1.00000         osd.186       up  1.00000          1.00000
>>> 187   1.00000         osd.187       up  1.00000          1.00000
>>> 188   1.00000         osd.188       up  1.00000          1.00000
>>> 189   1.00000         osd.189       up  1.00000          1.00000
>>> 190   1.00000         osd.190       up  1.00000          1.00000
>>> 191   1.00000         osd.191       up  1.00000          1.00000
>>> 192   1.00000         osd.192       up  1.00000          1.00000
>>> 193   1.00000         osd.193       up  1.00000          1.00000
>>> 194   1.00000         osd.194       up  1.00000          1.00000
>>> 195   1.79999         osd.195       up  1.00000          1.00000
>>> 196   1.79999         osd.196       up  1.00000          1.00000
>>> 197   1.79999         osd.197       up  1.00000          1.00000
>>> 198   1.79999         osd.198       up  1.00000          1.00000
>>> 199   1.79999         osd.199       up  1.00000          1.00000
>>> 200   1.79999         osd.200       up  1.00000          1.00000
>>> 201   1.79999         osd.201       up  1.00000          1.00000
>>> 202   1.79999         osd.202       up  1.00000          1.00000
>>> 203   1.79999         osd.203       up  1.00000          1.00000
>>> 204   1.79999         osd.204       up  1.00000          1.00000
>>> 205   1.79999         osd.205       up  1.00000          1.00000
>>> -16  17.99988     host ceph010
>>> 206   1.79999         osd.206       up  1.00000          1.00000
>>> 207   1.79999         osd.207       up  1.00000          1.00000
>>> 208   1.79999         osd.208       up  1.00000          1.00000
>>> 209   1.79999         osd.209       up  1.00000          1.00000
>>> 210   1.79999         osd.210       up  1.00000          1.00000
>>> 211   1.79999         osd.211       up  1.00000          1.00000
>>> 212   1.79999         osd.212       up  1.00000          1.00000
>>> 213   1.79999         osd.213       up  1.00000          1.00000
>>> 214   1.79999         osd.214       up  1.00000          1.00000
>>> 215   1.79999         osd.215       up  1.00000          1.00000
>>> 
>>> 
>>> 
>>>> Il giorno 20 nov 2017, alle ore 12:17, Christian Balzer 
>>>> <chibi at gol.com> ha scritto:
>>>> 
>>>> 
>>>> Hello,
>>>> 
>>>> On Mon, 20 Nov 2017 11:56:31 +0100 Matteo Dacrema wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> I need to switch a cluster of over 200 OSDs from replica 2 to replica 
>>>>> 3  
>>>> I presume this means the existing cluster and not adding 100 OSDs...
>>>> 
>>>>> There are two different crush maps for HDD and SSDs also mapped to 
>>>>> two different pools.
>>>>> 
>>>>> Is there a best practice to use? Can this provoke troubles?
>>>>> 
>>>> Are your SSDs a cache-tier or are they a fully separate pool?
>>>> 
>>>> As for troubles, how busy is your cluster during the recovery of 
>>>> failed
>>>> OSDs or deep scrubs?
>>>> 
>>>> There are 2 things to consider here:
>>>> 
>>>> 1. The re-balancing and additional replication of all the data, which 
>>>> you
>>>> can control/ease by the various knobs present. Ceph version matters to
>>>> which are relevant/useful. It shouldn't impact things too much, unless
>>>> your cluster was at the very edge of it's capacity anyway.
>>>> 
>>>> 2. The little detail that after 1) is done, your cluster will be
>>>> noticeably slower than before, especially in the latency department.
>>>> In short, you don't just need to have the disk space to go 3x, but 
>>>> also
>>>> enough IOPS/bandwidth reserves.
>>>> 
>>>> Christian
>>>> 
>>>>> Thank you
>>>>> Matteo
>>>>> _______________________________________________
>>>>> ceph-users mailing list
>>>>> ceph-users at lists.ceph.com
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Christian Balzer        Network/Systems Engineer
>>>> chibi at gol.com   	Rakuten Communications
>>>> 
>>>> --
>>>> Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato 
>>>> non infetto.
>>>> Seguire il link qui sotto per segnalarlo come spam:
>>>> http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=524464756E.A33EC
>>>> 
>>>> 
>>> 
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users at lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com  
>> 
> 
> 
> -- 
> Christian Balzer        Network/Systems Engineer                
> chibi at gol.com <mailto:chibi at gol.com>   	Rakuten Communications
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> 
> --
> Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato non infetto.
> Seguire il link qui sotto per segnalarlo come spam: 
> http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=78F40403AD.A839B <http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=78F40403AD.A839B>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171121/7d2e3765/attachment.html>


More information about the ceph-users mailing list