[ceph-users] "failed to open ino"
jmozdzen at nde.ag
Tue Nov 28 05:50:38 PST 2017
Zitat von David C <dcsysengineer at gmail.com>:
> On 27 Nov 2017 1:06 p.m., "Jens-U. Mozdzen" <jmozdzen at nde.ag> wrote:
> Hi David,
> Zitat von David C <dcsysengineer at gmail.com>:
> Hi Jens
>> We also see these messages quite frequently, mainly the "replicating
>> dir...". Only seen "failed to open ino" a few times so didn't do any real
>> investigation. Our set up is very similar to yours, 12.2.1, active/standby
>> MDS and exporting cephfs through KNFS (hoping to replace with Ganesha
> been there, done that - using Ganesha more than doubled the run-time of our
> jobs, while with knfsd, the run-time is about the same for CephFS-based and
> "local disk"-based files. But YMMV, so if you see speeds with Ganesha that
> are similar to knfsd, please report back with details...
> I'd be interested to know if you tested Ganesha over a cephfs kernel mount
> (ie using the VFS fsal) or if you used the Ceph fsal. Also the server and
> client versions you tested.
I had tested Ganesha only via the Ceph FSAL. Our Ceph nodes (including
the one used as a Ganesha server) are running
ceph-12.2.1+git.1507910930.aea79b8b7a on OpenSUSE 42.3, SUSE's kernel
4.4.76-1-default (which has a number of back-ports in it), Ganesha is
at version nfs-ganesha-184.108.40.206+git.1504275777.a9d23b98f.
The NFS clients are a broad mix of current and older systems.
> Prior to Luminous, Ganesha writes were terrible due to a bug with fsync
> calls in the mds code. The fix went into the mds and client code. If you're
> doing Ganesha over the top of the kernel mount you'll need a pretty recent
> kernel to see the write improvements.
As we were testing the Ceph FSAL, this should not be the cause.
> From my limited Ganesha testing so far, reads are better when exporting the
> kernel mount, writes are much better with the Ceph fsal. But that's
> expected for me as I'm using the CentOS kernel. I was hoping the
> aforementioned fix would make it into the rhel 7.4 kernel but doesn't look
> like it has.
When exporting the kernel-mounted CephFS via kernel nfsd, we see
similar speeds to serving the same set of files from a local bcache'd
RAID1 array on SAS disks. This is for a mix of reads and writes,
mostly small files (compile jobs, some packaging).
> From what I can see, it would have to be A/A/P, since MDS demands at least
> one stand-by.
> That's news to me.
From http://docs.ceph.com/docs/master/cephfs/multimds/ :
"Each CephFS filesystem has a max_mds setting, which controls how many
ranks will be created. The actual number of ranks in the filesystem
will only be increased if a spare daemon is available to take on the
new rank. For example, if there is only one MDS daemon running, and
max_mds is set to two, no second rank will be created."
Might well be I was mis-reading this... I had first read it to mean
that a spare daemon needs to be available *while running* A/A, but the
example sounds like the spare is required when *switching to* A/A.
> Is it possible you still had standby config in your ceph.conf?
Not sure what you're asking for, is this related to active/active or
to our Ganesha tests? We have not yet tried to switch to A/A, so our
config actually contains standby parameters.
Jens-U. Mozdzen voice : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG fax : +49-40-559 51 77
Postfach 61 03 15 mobile : +49-179-4 98 21 98
D-22423 Hamburg e-mail : jmozdzen at nde.ag
Vorsitzende des Aufsichtsrates: Angelika Torlée-Mozdzen
Sitz und Registergericht: Hamburg, HRB 90934
Vorstand: Jens-U. Mozdzen
USt-IdNr. DE 814 013 983
More information about the ceph-users