[ceph-users] Poor ceph cluster performance

Stefan Kooman stefan at bit.nl
Mon Nov 26 23:19:44 PST 2018


Quoting Cody (codeology.lab at gmail.com):
> The Ceph OSD part of the cluster uses 3 identical servers with the
> following specifications:
> 
> CPU: 2 x E5-2603 @1.8GHz
> RAM: 16GB
> Network: 1G port shared for Ceph public and cluster traffics

This will hamper throughput a lot. 

> Journaling device: 1 x 120GB SSD (SATA3, consumer grade)
> OSD device: 2 x 2TB 7200rpm spindle (SATA3, consumer grade)

OK, let's stop here first: Consumer grade SSD. Percona did a nice
writeup about "fsync" speed on consumer grade SSDs [1]. As I don't know
what drives you use this might or might not be the issue.

> 
> This is not beefy enough in any way, but I am running for PoC only,
> with minimum utilization.
> 
> Ceph-mon and ceph-mgr daemons are hosted on the OpenStack Controller
> nodes. Ceph-ansible version is 3.1 and is using Filestore with
> non-colocated scenario (1 SSD for every 2 OSDs). Connection speed
> among Controllers, Computes, and OSD nodes can reach ~900Mbps tested
> using iperf.

Why filestore if I may ask? I guess bluestore with bluestore journal on
SSD and data on SATA should give you better performance. If the SSDs are
suitable for the job at all.

What version of Ceph are use using? Metrics can give you a lot of
insight. Did you take a look at those? In fFor example Ceph mgr dashboard?

> 
> I followed the Red Hat Ceph 3 benchmarking procedure [1] and received
> following results:
> 
> Write Test:
> 
> Total time run:         80.313004
> Total writes made:      17
> Write size:             4194304
> Object size:            4194304
> Bandwidth (MB/sec):     0.846687
> Stddev Bandwidth:       0.320051
> Max bandwidth (MB/sec): 2
> Min bandwidth (MB/sec): 0
> Average IOPS:           0
> Stddev IOPS:            0
> Max IOPS:               0
> Min IOPS:               0
> Average Latency(s):     66.6582
> Stddev Latency(s):      15.5529
> Max latency(s):         80.3122
> Min latency(s):         29.7059
> 
> Sequencial Read Test:
> 
> Total time run:       25.951049
> Total reads made:     17
> Read size:            4194304
> Object size:          4194304
> Bandwidth (MB/sec):   2.62032
> Average IOPS:         0
> Stddev IOPS:          0
> Max IOPS:             1
> Min IOPS:             0
> Average Latency(s):   24.4129
> Max latency(s):       25.9492
> Min latency(s):       0.117732
> 
> Random Read Test:
> 
> Total time run:       66.355433
> Total reads made:     46
> Read size:            4194304
> Object size:          4194304
> Bandwidth (MB/sec):   2.77295
> Average IOPS:         0
> Stddev IOPS:          3
> Max IOPS:             27
> Min IOPS:             0
> Average Latency(s):   21.4531
> Max latency(s):       66.1885
> Min latency(s):       0.0395266
> 
> Apparently, the results are pathetic...
> 
> As I moved on to test block devices, I got a following error message:
> 
> # rbd map image01 --pool testbench --name client.admin
> rbd: failed to add secret 'client.admin' to kernel

What replication factor are you using?

Make sure you have the client.admin keyring on the node you are issuing
this command. If you have the keyring present like Ceph expects it to
be, then you can omit the --name client.admin. On a monitor node you can
extract the admin keyring: ceph auth export client.admin. Put the output
of that in /etc/ceph/ceph.client.admin.keyring and this should work.

> Any suggestions on the above error and/or debugging would be greatly
> appreciated!

Gr. Stefan

[1]:
https://www.percona.com/blog/2018/07/18/why-consumer-ssd-reviews-are-useless-for-database-performance-use-case/
> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html-single/administration_guide/#benchmarking_performance
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
| BIT BV  http://www.bit.nl/        Kamer van Koophandel 09090351
| GPG: 0xD14839C6                   +31 318 648 688 / info at bit.nl


More information about the ceph-users mailing list