[ceph-users] cephfs kernel client - page cache being invaildated.

jesper at krogh.cc jesper at krogh.cc
Sun Oct 14 05:21:44 PDT 2018


Hi

We have a dataset of ~300 GB on CephFS which as being used for computations
over and over agian .. being refreshed daily or similar.

When hosting it on NFS after refresh, they are transferred, but from
there - they would be sitting in the kernel page cache of the client
until they are refreshed serverside.

On CephFS it look "similar" but "different". Where the "steady state"
operation over NFS would give a client/server traffic of < 1MB/s ..
CephFS contantly pulls 50-100MB/s over the network.  This has
implications for the clients that end up spending unnessary time waiting
for IO in the execution.

This is in a setting where the CephFS client mem look like this:

$ free -h
              total        used        free      shared  buff/cache  
available
Mem:           377G         17G        340G        1.2G         19G       
354G
Swap:          8.8G        430M        8.4G


If I just repeatedly run (within a few minute) something that is using the
files, then
it is fully served out of client page cache (2GB'ish / s) ..  but it looks
like
it is being evicted way faster than in the NFS setting?

This is not scientific .. but the CMD is a cat /file/on/ceph > /dev/null -
type on a total of 24GB data in 300'ish files.

$ free -h; time CMD ; sleep 1800; free -h; time CMD ; free -h; sleep 3600;
time CMD ;

              total        used        free      shared  buff/cache  
available
Mem:           377G         16G        312G        1.2G         48G       
355G
Swap:          8.8G        430M        8.4G

real    0m8.997s
user    0m2.036s
sys     0m6.915s
              total        used        free      shared  buff/cache  
available
Mem:           377G         17G        277G        1.2G         82G       
354G
Swap:          8.8G        430M        8.4G

real    3m25.904s
user    0m2.794s
sys     0m9.028s
              total        used        free      shared  buff/cache  
available
Mem:           377G         17G        283G        1.2G         76G       
353G
Swap:          8.8G        430M        8.4G

real    6m18.358s
user    0m2.847s
sys     0m10.651s


Munin graphs of the system confirms that there has been zero memory
pressure over the period.

Is there things in the CephFS case that can cause the page-cache to be
invailated?
Could less agressive "read-ahead" play a role?

Other thoughts on what root cause on the different behaviour could be?

Clients are using 4.15 kernel.. Anyone aware of newer patches in this area
that could impact ?

Jesper



More information about the ceph-users mailing list