[ceph-users] slow export of cephfs through samba

Gregory Farnum gfarnum at redhat.com
Wed Oct 3 08:53:30 PDT 2018


On Thu, Sep 27, 2018 at 7:37 AM Chad W Seys <cwseys at physics.wisc.edu> wrote:

> Hi all,
>    I am exporting cephfs using samba.  It is much slower over samba than
> direct. Anyone know how to speed it up?
>    Benchmarked using bonnie++ 5 times either directly to cephfs mounted
> by kernel (v4.18.6) module:
>     bonnie++ -> kcephfs
> or through a cifs kernel-module-mounted (protocol version 3.02) Samba
> (v4.8.5) share on the same machine.
>     bonnie++ -> Samba -> kcephfs
>
> Abbreviated results for 5 runs:
> kcephfs:                  min      max
>   file created             555      619       files/sec
>   sequential block input:  106.44   108.13    MB/sec
>   sequential block output: 102.82   110.61    MB/sec
>
> (There is a gigabit network between the client and the ceph cluster, so
> the block in/out is pleasing.)
>
> samba -> kcephfs:         min      max
>   file created             4        5       files/sec
>   sequential block input:  22.85    29.5    MB/sec
>   sequential block output: 27.95    30.01   MB/sec
>
> The block input/output is okay fast, but the files created per second is
> low.  Anyone know how to tweak Samba to speed it up?
>

That seems to be a samba tuning issue, which this isn't the right list for.
:/


>    Would Samba vfs_ceph speed up access?  At the moment vfs_ceph in
> Debian depends on libceph1 10.2.5, so not too modern.
>

Well, good news and bad news here. Good news is that since vfs_ceph uses
the native cephfs client library, it ought to be faster (though I don't
have any data on how much going through samba itself costs against our
client).
Bad news is that samba's connection model doesn't play very nicely with
Ceph's — if you use vfs_ceph, every samba connection will turn into a brand
new Ceph connection — smb literally runs a fork on every incoming
connection — so you will see RAM/network connections/etc scale on a
per-smb-client basis. This works fine for a small number of clients but not
so great if you're planning to attach a bunch of them. :(
-Greg


>
> Current Samba settings:
> [global]
>          dns proxy = No
>          hostname lookups = Yes
>          kerberos method = secrets and keytab
>          logging = syslog at 1 /var/log/samba/log.%m
>          max log size = 100000
>          panic action = /usr/share/samba/panic-action %d
>          realm = PHYSICS.WISC.EDU
>          security = USER
>          server signing = required
>          server string = %h server
>          workgroup = PHYSICS
>          fruit:nfs_aces = no
>          idmap config * : backend = tdb
> [smb]
>          ea support = Yes
>          inherit acls = Yes
>          inherit permissions = Yes
>          msdfs root = Yes
>          path = /srv/smb
>          read only = No
>          smb encrypt = desired
>          vfs objects = catia fruit streams_xattr
>          fruit:encoding = native
>
> Thanks!
> Chad.
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181003/aa1af688/attachment.html>


More information about the ceph-users mailing list