[ceph-users] OSD is near full and slow in accessing storage from client

Cassiano Pilipavicius cassiano at tips.com.br
Sun Nov 12 07:23:17 PST 2017


I think that more pgs help to distribute the data more evenly, but I 
dont know if its recommended with a low OSDs number. I remember read 
somewhere in the docs a guideline for the mas pgs number/OSD, but was 
from an really old ceph version, maybe things has changed.


Em 11/12/2017 12:39 PM, gjprabu escreveu:
> Hi Cassiano,
>
>        Thanks for your valuable feedback and will wait for some time 
> till new osd sync get complete. Also for by increasing pg count it is 
> the issue will solve? our setup pool size for data and metadata pg 
> number is 250. Is this correct for 7 OSD with 2 replica. Also 
> currently stored data size is 17TB.
>
> ceph osd df
> ID WEIGHT  REWEIGHT SIZE   USE    AVAIL %USE  VAR  PGS
> 0 3.29749  1.00000  3376G  2814G  562G 83.35 1.23 165
> 1 3.26869  1.00000  3347G  1923G 1423G 57.48 0.85 152
> 2 3.27339  1.00000  3351G  1980G 1371G 59.10 0.88 161
> 3 3.24089  1.00000  3318G  2131G 1187G 64.23 0.95 168
> 4 3.24089  1.00000  3318G  2998G  319G 90.36 1.34 176
> 5 3.32669  1.00000  3406G  2476G  930G 72.68 1.08 165
> 6 3.27800  1.00000  3356G  1518G 1838G 45.24 0.67 166
>               TOTAL 23476G 15843G 7632G 67.49
> MIN/MAX VAR: 0.67/1.34  STDDEV: 14.53
>
> ceph osd tree
> ID WEIGHT   TYPE NAME            UP/DOWN REWEIGHT PRIMARY-AFFINITY
> -1 22.92604 root default
> -2  3.29749     host intcfs-osd1
> 0  3.29749         osd.0             up  1.00000 1.00000
> -3  3.26869     host intcfs-osd2
> 1  3.26869         osd.1             up  1.00000 1.00000
> -4  3.27339     host intcfs-osd3
> 2  3.27339         osd.2             up  1.00000 1.00000
> -5  3.24089     host intcfs-osd4
> 3  3.24089         osd.3             up  1.00000 1.00000
> -6  3.24089     host intcfs-osd5
> 4  3.24089         osd.4             up  1.00000 1.00000
> -7  3.32669     host intcfs-osd6
> 5  3.32669         osd.5             up  1.00000 1.00000
> -8  3.27800     host intcfs-osd7
> 6  3.27800         osd.6             up  1.00000 1.00000
>
> *ceph osd pool ls detail*
>
> pool 0 'rbd' replicated size 2 min_size 1 crush_ruleset 0 object_hash 
> rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool 
> stripe_width 0
> pool 3 '*downloads_data*' replicated size 2 min_size 1 crush_ruleset 0 
> object_hash rjenkins*pg_num 250 pgp_num 250* last_change 39 flags 
> hashpspool crash_replay_interval 45 stripe_width 0
> pool 4 '*downloads_metadata*' replicated size 2 min_size 1 
> crush_ruleset 0 object_hash rjenkins *pg_num 250 pgp_num 250 
> *last_change 36 flags hashpspool stripe_width 0
>
> Regards
> Prabu GJ
>
> ---- On Sun, 12 Nov 2017 19:20:34 +0530 *Cassiano Pilipavicius 
> <cassiano at tips.com.br>* wrote ----
>
>     I am also not an expert, but it looks like you have big data
>     volumes on few PGs, from what I've seen, the pg data is only
>     deleted from the old OSD when is completed copied to the new osd.
>
>     So, if 1 pg have 100G por example, only when it is fully copied to
>     the new OSD, the space will be released on the old OSD.
>
>     If you have a busy cluster/network, it may take a good while.
>     Maybe just wait a litle and check from time to time and the space
>     will eventually be released.
>
>
>     Em 11/12/2017 11:44 AM, Sébastien VIGNERON escreveu:
>
>
>     _______________________________________________
>     ceph-users mailing list
>     ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com>
>     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>         I’m not an expert either so if someone in the list have some
>         ideas on this problem, don’t be shy, share them with us.
>
>         For now, I only have hypothese that the OSD space will be
>         recovered as soon as the recovery process is complete.
>         Hope everything will get back in order soon (before reaching
>         95% or above).
>
>         I saw some messages on the list about the fstrim tool which
>         can help reclaim unused free space, but i don’t know if it’s
>         apply to your case.
>
>         Cordialement / Best regards,
>
>         Sébastien VIGNERON
>         CRIANN,
>         Ingénieur / Engineer
>         Technopôle du Madrillet
>         745, avenue de l'Université
>         76800 Saint-Etienne du Rouvray - France
>         tél. +33 2 32 91 42 91
>         fax. +33 2 32 91 42 92
>         http://www.criann.fr
>         mailto:sebastien.vigneron at criann.fr
>         support: support at criann.fr <mailto:support at criann.fr>
>
>             Le 12 nov. 2017 à 13:29, gjprabu <gjprabu at zohocorp.com
>             <mailto:gjprabu at zohocorp.com>> a écrit :
>
>             Hi Sebastien,
>
>                 Below is the query details. I am not that much expert
>             and still learning . pg's are not stuck stat before adding
>             osd and pg are slowly clearing stat to active-clean. Today
>             morning there was around
>             53 active+undersized+degraded+remapped+wait_backfill and
>             now it is 21 only, hope its going on and i am seeing the
>             space keep increasing in newly added OSD (osd.6)
>
>
>             ID WEIGHT  REWEIGHT SIZE   USE    AVAIL %USE  VAR  PGS
>             *0 3.29749  1.00000  3376G  2814G  562G 83.35 1.23 165  (
>             Available Spaces not reduced after adding new OSD)*
>             1 3.26869  1.00000  3347G  1923G 1423G 57.48 0.85 152
>             2 3.27339  1.00000  3351G  1980G 1371G 59.10 0.88 161
>             3 3.24089  1.00000  3318G  2131G 1187G 64.23 0.95 168
>             *4 3.24089  1.00000  3318G  2998G  319G 90.36 1.34 176
>              (**Available *Spaces not reduced after adding new OSD)**
>             *5 3.32669  1.00000  3406G  2476G  930G 72.68 1.08 165
>             *(**Available *Spaces not reduced after adding new OSD)***
>             6 3.27800  1.00000  3356G  1518G 1838G 45.24 0.67 166
>                           TOTAL 23476G 15843G 7632G 67.49
>             MIN/MAX VAR: 0.67/1.34  STDDEV: 14.53
>
>         ...
>
>
>
>         _______________________________________________
>         ceph-users mailing list
>         ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com>
>         http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171112/c5945fa0/attachment.html>


More information about the ceph-users mailing list