[ceph-users] auth error with ceph-deploy on jewel to luminous upgrade

John Spray jspray at redhat.com
Thu Oct 19 10:05:53 PDT 2017


On Wed, Oct 18, 2017 at 6:34 PM, Gary Molenkamp <molenkam at uwo.ca> wrote:
> Sorry to reply to my own question, but I noticed that the cephx key for
> client.bootstrap-mgr was inconsistent with the key in
> /var/lib/ceph/bootstrap-mgr/ceph.keyring.
>
> I deleted the entry in ceph:
>
>     ceph auth del client.bootstrap-mgr
>
> reran the ceph-deploy gather keys:
>
>     ceph-deploy gatherkeys cephtest-admin
>
> and used the key entry for client.bootstrap-mgr as the key value in
> /var/lib/ceph/bootstrap-mgr/ceph.keyring.
>
> I was then able to successfully run the ceph-deploy to create the mgr.
>
> I'm not sure where this inconsistent key came from as the
> /var/lib/ceph/bootstrap-mgr directory doesn't appear on a fresh install of
> jewel.

Thanks for the report - probably same issue as
http://tracker.ceph.com/issues/20950

Glad you could work around it, we also have a candidate fix for master
being tested now (you motivated me to take another look at it :-))

John

>
>
>
>
>
> On 18/10/17 11:04 AM, Gary molenkamp wrote:
>
> I'm running into a permission error when attempting to use ceph-deploy to
> create an mgr on a recently upgraded jewel->luminous ceph cluster.  I've
> attempted to track down the permission, but so far no success.   I'm doing
> this on a dev environment so I can replicate:
>
> Start with a sample jewel release (one admin node with mon, two storage
> nodes with one osd each).  ceph preflight complete (cephinst user with sudo,
> ntp, etc):
>
> On each storage node, the osd disk is formatted as xfs and mounted on
> /var/lib/ceph/osd/ceph-X, and "chown ceph:ceph /var/lib/ceph/osd/ceph-X"
>
> On admin server:
>
> su cephinst
> mkdir sci-cluster
> cd sci-cluster
> ceph-deploy new cephtest-admin
> ceph-deploy install cephtest-admin cephtest-stor1 cephtest-stor2
> ceph-deploy mon create-initial
> ceph-deploy osd prepare cephtest-stor1:/var/lib/ceph/osd/ceph-0
> ceph-deploy osd prepare cephtest-stor2:/var/lib/ceph/osd/ceph-1
> ceph-deploy osd activate cephtest-stor1:/var/lib/ceph/osd/ceph-0
> ceph-deploy osd activate cephtest-stor2:/var/lib/ceph/osd/ceph-1
> ceph-deploy admin cephtest-admin cephtest-stor1 cephtest-stor2
> ceph osd pool set rbd size 2
> sudo chmod a+r /etc/ceph/ceph.client.admin.keyring
>
> At this point I have a working, healthy jewel cluster as reported by "ceph
> -s".  I then upgrade the mon and try to deploy an mgr service:
>
> sudo sed -i 's/jewel/luminous/g' /etc/yum.repos.d/ceph-deploy.repo
> sudo sed -i 's/jewel/luminous/g' /etc/yum.repos.d/ceph.repo
> sudo systemctl stop ceph-mon at cephtest-admin.service
> sudo yum clean all
> sudo yum update
> sudo systemctl start ceph-mon at cephtest-admin.service
>
>
> "ceph -s" reports HEALTH_OK, but "mgr: no daemons active". Attempt to deploy
> a mgr service with ceph-deploy and it fails:
>
> ceph-deploy mgr create cephtest-admin
> [ceph_deploy.conf][DEBUG ] found configuration file at:
> /home/cephinst/.cephdeploy.conf
> [ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy mgr create
> cephtest-admin
> [ceph_deploy.cli][INFO  ] ceph-deploy options:
> [ceph_deploy.cli][INFO  ]  username                      : None
> [ceph_deploy.cli][INFO  ]  verbose                       : False
> [ceph_deploy.cli][INFO  ]  mgr                           :
> [('cephtest-admin', 'cephtest-admin')]
> [ceph_deploy.cli][INFO  ]  overwrite_conf                : False
> [ceph_deploy.cli][INFO  ]  subcommand                    : create
> [ceph_deploy.cli][INFO  ]  quiet                         : False
> [ceph_deploy.cli][INFO  ]  cd_conf                       :
> <ceph_deploy.conf.cephdeploy.Conf instance at 0x237b320>
> [ceph_deploy.cli][INFO  ]  cluster                       : ceph
> [ceph_deploy.cli][INFO  ]  func                          : <function mgr at
> 0x230ac08>
> [ceph_deploy.cli][INFO  ]  ceph_conf                     : None
> [ceph_deploy.cli][INFO  ]  default_release               : False
> [ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts
> cephtest-admin:cephtest-admin
> [cephtest-admin][DEBUG ] connection detected need for sudo
> [cephtest-admin][DEBUG ] connected to host: cephtest-admin
> [cephtest-admin][DEBUG ] detect platform information from remote host
> [cephtest-admin][DEBUG ] detect machine type
> [ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
> [ceph_deploy.mgr][DEBUG ] remote host will use systemd
> [ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to cephtest-admin
> [cephtest-admin][DEBUG ] write cluster configuration to
> /etc/ceph/{cluster}.conf
> [cephtest-admin][DEBUG ] create path if it doesn't exist
> [cephtest-admin][INFO  ] Running command: sudo ceph --cluster ceph --name
> client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth
> get-or-create mgr.cephtest-admin mon allow profile mgr osd allow * mds allow
> * -o /var/lib/ceph/mgr/ceph-cephtest-admin/keyring
> [cephtest-admin][ERROR ] 2017-10-18 10:42:00.460124 7f9325bcd700  0
> librados: client.bootstrap-mgr authentication error (1) Operation not
> permitted
> [cephtest-admin][ERROR ] [errno 1] error connecting to the cluster
> [cephtest-admin][ERROR ] exit code from command was: 1
> [ceph_deploy.mgr][ERROR ] could not create mgr
>
> It looks like cephx was set up properly:
>
> ceph auth list
> installed auth entries:
>
> osd.0
>     key: AQChOOZZe1PVGBAAGFRiziREnCm8TZL3QwuFnw==
>     caps: [mgr] allow profile osd
>     caps: [mon] allow profile osd
>     caps: [osd] allow *
> osd.1
>     key: AQDEOOZZQVqUORAALnPX6+tuhKM33+bveQxiDw==
>     caps: [mgr] allow profile osd
>     caps: [mon] allow profile osd
>     caps: [osd] allow *
> client.admin
>     key: AQD0N+ZZUSluEBAATIGq+KK3LdUYZHw4RWToXg==
>     caps: [mds] allow *
>     caps: [mgr] allow *
>     caps: [mon] allow *
>     caps: [osd] allow *
> client.bootstrap-mds
>     key: AQD1N+ZZZSnPABAANDqpSE9g6MqfmXglGzn6Nw==
>     caps: [mgr] allow r
>     caps: [mon] allow profile bootstrap-mds
> client.bootstrap-mgr
>     key: AQBnZ+dZzitjIxAArptc11qf8UuPq5QXFy+I9Q==
>     caps: [mon] allow profile bootstrap-mgr
> client.bootstrap-osd
>     key: AQD0N+ZZrLv5HRAAbeLByJJyplQEvjz1o1N8kg==
>     caps: [mgr] allow r
>     caps: [mon] allow profile bootstrap-osd
> client.bootstrap-rgw
>     key: AQD0N+ZZFL/hLxAAR9WbRd/ETL/GJ+NKXum2iA==
>     caps: [mgr] allow r
>     caps: [mon] allow profile bootstrap-rgw
>
>
> Any insight or hints would be appreciated.
> Thanks
> Gary
>
> PS. on a previous attempt I did upgrade the ceph versions on the osds as
> well before attempting to deploy the mgr; same result.
>
>
> --
> Gary Molenkamp Computer Science
> Systems Administrator University of Western Ontario
> molenkam at uwo.ca                 http://www.csd.uwo.ca
> (519) 661-2111 x86882 (519) 661-3566
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> --
> Gary Molenkamp Computer Science
> Systems Administrator University of Western Ontario
> molenkam at uwo.ca                 http://www.csd.uwo.ca
> (519) 661-2111 x86882 (519) 661-3566
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>


More information about the ceph-users mailing list