[ceph-users] ceph all-nvme mysql performance tuning

Zoltan Arnold Nagy zoltan at linux.vnet.ibm.com
Wed Nov 29 14:14:52 PST 2017

On 2017-11-27 14:02, German Anders wrote:
> 4x 2U servers:
>   1x 82599ES 10-Gigabit SFI/SFP+ Network Connection
>   1x Mellanox ConnectX-3 InfiniBand FDR 56Gb/s Adapter (dual port)
so I assume you are using IPoIB as the cluster network for the 

> 1x OneConnect 10Gb NIC (quad-port) - in a bond configuration
> (active/active) with 3 vlans
... and the 10GbE network for the front-end network?

At 4k writes your network latency will be very high (see the flame 
graphs at the Intel NVMe presentation from the Boston OpenStack Summit - 
not sure if there is a newer deck that somebody could link ;)) and the 
time will be spent in the kernel. You could give RDMAMessenger a try but 
it's not stable at the current LTS release.

If I were you I'd be looking at 100GbE - we've recently pulled in a 
bunch of 100GbE links and it's been wonderful to see 100+GB/s going over 
the network for just storage.

Some people suggested mounting multiple RBD volumes - unless I'm 
mistaken and you're using very recent qemu/libvirt combinations with the 
proper libvirt disk settings all IO will still be single threaded 
towards librbd thus not making any speedup.

More information about the ceph-users mailing list