[ceph-users] OSD is near full and slow in accessing storage from client

gjprabu gjprabu at zohocorp.com
Sun Nov 12 08:41:03 PST 2017


        

        
            Hi David,Thanks for your valuable reply , once complete the backfilling for new osd and will consider by increasing replica value asap. Is it possible to decrease the metadata pg count ?  if the pg count for metadata for value same as data count what kind of issue may occur ? RegardsPrabuGJ---- On Sun, 12 Nov 2017 21:25:05 +0530  David Turner<drakonstein at gmail.com> wrote ----What's the output of `ceph df` to see if your PG counts are good or not?  Like everyone else has said, the space on the original osds can't be expected to free up until the backfill from adding the new osd has finished. You don't have anything in your cluster health to indicate that your cluster will not be able to finish this backfilling operation on its own. You might find this URL helpful in calculating your PG counts. http://ceph.com/pgcalc/  As a side note. It is generally better to keep your PG counts as base 2 numbers (16, 64, 256, etc). When you do not have a base 2 number then some of your PGs will take up twice as much space as others. In your case with 250, you have 244 PGs that are the same size and 6 PGs that are twice the size of those 244 PGs.  Bumping that up to 256 will even things out. Assuming that the metadata pool is for a CephFS volume, you do not need nearly so many PGs for that pool. Also, I would recommend changing at least the metadata pool to 3 replica_size. If we can talk you into 3 replica for everything else, great! But if not, at least do the metadata pool. If you lose an object in the data pool, you just lose that file. If you lose an object in the metadata pool, you might lose access to the entire CephFS volume. On Sun, Nov 12, 2017, 9:39 AM gjprabu <gjprabu at zohocorp.com> wrote:Hi Cassiano,       Thanks for your valuable feedback and will wait for some time till new osd sync get complete. Also for by increasing pg count it is the issue will solve? our setup pool size for data and metadata pg number is 250. Is this correct for 7 OSD with 2 replica. Also currently stored data size is 17TB.ceph osd dfID WEIGHT  REWEIGHT SIZE   USE    AVAIL %USE  VAR  PGS0 3.29749  1.00000  3376G  2814G  562G 83.35 1.23 1651 3.26869  1.00000  3347G  1923G 1423G 57.48 0.85 1522 3.27339  1.00000  3351G  1980G 1371G 59.10 0.88 1613 3.24089  1.00000  3318G  2131G 1187G 64.23 0.95 1684 3.24089  1.00000  3318G  2998G  319G 90.36 1.34 1765 3.32669  1.00000  3406G  2476G  930G 72.68 1.08 1656 3.27800  1.00000  3356G  1518G 1838G 45.24 0.67 166              TOTAL 23476G 15843G 7632G 67.49         MIN/MAX VAR: 0.67/1.34  STDDEV: 14.53ceph osd treeID WEIGHT   TYPE NAME            UP/DOWN REWEIGHT PRIMARY-AFFINITY-1 22.92604 root default                                          -2  3.29749     host intcfs-osd1                                  0  3.29749         osd.0             up  1.00000          1.00000-3  3.26869     host intcfs-osd2                                  1  3.26869         osd.1             up  1.00000          1.00000-4  3.27339     host intcfs-osd3                                  2  3.27339         osd.2             up  1.00000          1.00000-5  3.24089     host intcfs-osd4                                  3  3.24089         osd.3             up  1.00000          1.00000-6  3.24089     host intcfs-osd5                                  4  3.24089         osd.4             up  1.00000          1.00000-7  3.32669     host intcfs-osd6                                  5  3.32669         osd.5             up  1.00000          1.00000-8  3.27800     host intcfs-osd7                                  6  3.27800         osd.6             up  1.00000          1.00000ceph osd pool ls detailpool 0 'rbd' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool stripe_width 0pool 3 'downloads_data' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 250 pgp_num 250 last_change 39 flags hashpspool crash_replay_interval 45 stripe_width 0pool 4 'downloads_metadata' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 250 pgp_num 250 last_change 36 flags hashpspool stripe_width 0RegardsPrabu GJ---- On Sun, 12 Nov 2017 19:20:34 +0530 Cassiano Pilipavicius <cassiano at tips.com.br> wrote ----I am also not an expert, but it looks like you have big data       volumes on few PGs, from what I've seen, the pg data is only       deleted from the old OSD when is completed copied to the new osd.So, if 1 pg have 100G por example, only when it is fully copied       to the new OSD, the space will be released on the old OSD.If you have a busy cluster/network, it may take a good while.       Maybe just wait a litle and check from time to time and the space       will eventually be released.Em 11/12/2017 11:44 AM, Sébastien       VIGNERON escreveu:_______________________________________________ceph-users mailing list ceph-users at lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com I’m not an expert either so if someone in the list have some ideas       on this problem, don’t be shy, share them with us. For now, I only have hypothese that the OSD space         will be recovered as soon as the recovery process is complete. Hope everything will get back in order soon (before         reaching 95% or above).I saw some messages on the list about the fstrim         tool which can help reclaim unused free space, but i don’t know         if it’s apply to your case.Cordialement / Best regards,  Sébastien VIGNERON  CRIANN,  Ingénieur / Engineer Technopôle du Madrillet  745, avenue de l'Université  76800 Saint-Etienne du Rouvray - France  tél. +33 2 32 91 42 91  fax. +33 2 32 91 42 92  http://www.criann.fr  mailto:sebastien.vigneron at criann.fr support: support at criann.frLe 12 nov. 2017 à 13:29, gjprabu <gjprabu at zohocorp.com> a             écrit :Hi Sebastien,    Below is the query details. I am not                   that much expert and still learning . pg's are not                   stuck stat before adding osd and pg are slowly                   clearing stat to active-clean. Today morning there was                   around                   53 active+undersized+degraded+remapped+wait_backfill                   and now it is 21 only, hope its going on and i am                   seeing the space keep increasing in newly added OSD                   (osd.6) ID WEIGHT  REWEIGHT SIZE   USE    AVAIL                   %USE  VAR  PGS 0 3.29749  1.00000  3376G                      2814G  562G 83.35 1.23 165  ( Available Spaces not                     reduced after adding new OSD)1 3.26869  1.00000  3347G  1923G 1423G                   57.48 0.85 1522 3.27339  1.00000  3351G  1980G 1371G                   59.10 0.88 1613 3.24089  1.00000  3318G  2131G 1187G                   64.23 0.95 1684 3.24089  1.00000  3318G                      2998G  319G 90.36 1.34 176  ( Available Spaces not reduced after                       adding new OSD)5 3.32669  1.00000  3406G                      2476G  930G 72.68 1.08 165  ( Available Spaces not reduced                         after adding new OSD)6 3.27800  1.00000  3356G  1518G 1838G                   45.24 0.67 166              TOTAL 23476G 15843G 7632G                   67.49         MIN/MAX VAR: 0.67/1.34  STDDEV: 14.53..._______________________________________________ ceph-users mailing list ceph-users at lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users at lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com  
        
        

    
    

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171112/7dc1ec06/attachment.html>


More information about the ceph-users mailing list