[ceph-users] ceph all-nvme mysql performance tuning
Zoltan Arnold Nagy
zoltan at linux.vnet.ibm.com
Wed Nov 29 14:14:52 PST 2017
On 2017-11-27 14:02, German Anders wrote:
> 4x 2U servers:
> 1x 82599ES 10-Gigabit SFI/SFP+ Network Connection
> 1x Mellanox ConnectX-3 InfiniBand FDR 56Gb/s Adapter (dual port)
so I assume you are using IPoIB as the cluster network for the
> 1x OneConnect 10Gb NIC (quad-port) - in a bond configuration
> (active/active) with 3 vlans
... and the 10GbE network for the front-end network?
At 4k writes your network latency will be very high (see the flame
graphs at the Intel NVMe presentation from the Boston OpenStack Summit -
not sure if there is a newer deck that somebody could link ;)) and the
time will be spent in the kernel. You could give RDMAMessenger a try but
it's not stable at the current LTS release.
If I were you I'd be looking at 100GbE - we've recently pulled in a
bunch of 100GbE links and it's been wonderful to see 100+GB/s going over
the network for just storage.
Some people suggested mounting multiple RBD volumes - unless I'm
mistaken and you're using very recent qemu/libvirt combinations with the
proper libvirt disk settings all IO will still be single threaded
towards librbd thus not making any speedup.
More information about the ceph-users