[ceph-users] auth error with ceph-deploy on jewel to luminous upgrade

Gary molenkamp molenkam at uwo.ca
Wed Oct 18 08:04:35 PDT 2017

I'm running into a permission error when attempting to use ceph-deploy 
to create an mgr on a recently upgraded jewel->luminous ceph cluster.  
I've attempted to track down the permission, but so far no success.   
I'm doing this on a dev environment so I can replicate:

Start with a sample jewel release (one admin node with mon, two storage 
nodes with one osd each).  ceph preflight complete (cephinst user with 
sudo, ntp, etc):

On each storage node, the osd disk is formatted as xfs and mounted on 
/var/lib/ceph/osd/ceph-X, and "chown ceph:ceph /var/lib/ceph/osd/ceph-X"

On admin server:

|su cephinst|
|mkdir sci-cluster|||
|cd sci-cluster|
|ceph-deploy ||new| |cephtest-admin|
|ceph-deploy install cephtest-admin cephtest-stor|||1 cephtest-stor|2||
ceph-deploy mon create-initial||
||ceph-deploy osd prepare |||cephtest-stor|1:/var/lib/ceph/osd/ceph-||0||
||ceph-deploy osd ||||prepare cephtest-stor||2:/var/lib/ceph/osd/ceph-||1||
|||ceph-deploy osd activate 
||ceph-deploy osd ||||activate cephtest-stor||2:/var/lib/ceph/osd/ceph-||1||
|ceph-deploy admin cephtest-admin ||||cephtest-stor|1 |||||cephtest-stor|2
|ceph osd pool set rbd size ||2
sudo chmod a+r /etc/ceph/ceph.client.admin.keyring

At this point I have a working, healthy jewel cluster as reported by 
"ceph -s".  I then upgrade the mon and try to deploy an mgr service:

sudo sed -i 's/jewel/luminous/g' /etc/yum.repos.d/ceph-deploy.repo
||||sudo sed -i 's/jewel/luminous/g' /etc/yum.repos.d/ceph.repo
sudo systemctl stop ceph-mon at cephtest-admin.service
sudo yum clean all
sudo yum update|
|sudo systemctl start ceph-mon at cephtest-admin.service

"ceph -s" reports HEALTH_OK, but "mgr: no daemons active". Attempt to 
deploy a mgr service with ceph-deploy and it fails:

ceph-deploy mgr create cephtest-admin
[ceph_deploy.conf][DEBUG ] found configuration file at: 
[ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy mgr 
create cephtest-admin
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  mgr                           : 
[('cephtest-admin', 'cephtest-admin')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : 
<ceph_deploy.conf.cephdeploy.Conf instance at 0x237b320>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mgr 
at 0x230ac08>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts 
[cephtest-admin][DEBUG ] connection detected need for sudo
[cephtest-admin][DEBUG ] connected to host: cephtest-admin
[cephtest-admin][DEBUG ] detect platform information from remote host
[cephtest-admin][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to cephtest-admin
[cephtest-admin][DEBUG ] write cluster configuration to 
[cephtest-admin][DEBUG ] create path if it doesn't exist
[cephtest-admin][INFO  ] Running command: sudo ceph --cluster ceph 
--name client.bootstrap-mgr --keyring 
/var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create 
mgr.cephtest-admin mon allow profile mgr osd allow * mds allow * -o 
[cephtest-admin][ERROR ] 2017-10-18 10:42:00.460124 7f9325bcd700  0 
librados: client.bootstrap-mgr authentication error (1) Operation not 
[cephtest-admin][ERROR ] [errno 1] error connecting to the cluster
[cephtest-admin][ERROR ] exit code from command was: 1
[ceph_deploy.mgr][ERROR ] could not create mgr

It looks like cephx was set up properly:

ceph auth list
installed auth entries:

     key: AQChOOZZe1PVGBAAGFRiziREnCm8TZL3QwuFnw==
     caps: [mgr] allow profile osd
     caps: [mon] allow profile osd
     caps: [osd] allow *
     key: AQDEOOZZQVqUORAALnPX6+tuhKM33+bveQxiDw==
     caps: [mgr] allow profile osd
     caps: [mon] allow profile osd
     caps: [osd] allow *
     caps: [mds] allow *
     caps: [mgr] allow *
     caps: [mon] allow *
     caps: [osd] allow *
     key: AQD1N+ZZZSnPABAANDqpSE9g6MqfmXglGzn6Nw==
     caps: [mgr] allow r
     caps: [mon] allow profile bootstrap-mds
     key: AQBnZ+dZzitjIxAArptc11qf8UuPq5QXFy+I9Q==
     caps: [mon] allow profile bootstrap-mgr
     key: AQD0N+ZZrLv5HRAAbeLByJJyplQEvjz1o1N8kg==
     caps: [mgr] allow r
     caps: [mon] allow profile bootstrap-osd
     key: AQD0N+ZZFL/hLxAAR9WbRd/ETL/GJ+NKXum2iA==
     caps: [mgr] allow r
     caps: [mon] allow profile bootstrap-rgw

Any insight or hints would be appreciated.

PS. on a previous attempt I did upgrade the ceph versions on the osds as 
well before attempting to deploy the mgr; same result.


Gary Molenkamp			Computer Science
Systems Administrator		University of Western Ontario
molenkam at uwo.ca                 http://www.csd.uwo.ca
(519) 661-2111 x86882		(519) 661-3566

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171018/3d2e4f59/attachment.html>

More information about the ceph-users mailing list