[ceph-users] auth error with ceph-deploy on jewel to luminous upgrade

Gary Molenkamp molenkam at uwo.ca
Wed Oct 18 10:34:39 PDT 2017


Sorry to reply to my own question, but I noticed that the cephx key for 
client.bootstrap-mgr was inconsistent with the key in 
/var/lib/ceph/bootstrap-mgr/ceph.keyring.

I deleted the entry in ceph:

     ceph auth del client.bootstrap-mgr

reran the ceph-deploy gather keys:

     ceph-deploy gatherkeys cephtest-admin

and used the key entry for client.bootstrap-mgr as the key value in 
/var/lib/ceph/bootstrap-mgr/ceph.keyring.

I was then able to successfully run the ceph-deploy to create the mgr.

I'm not sure where this inconsistent key came from as the 
/var/lib/ceph/bootstrap-mgr directory doesn't appear on a fresh install 
of jewel.




On 18/10/17 11:04 AM, Gary molenkamp wrote:
>
> I'm running into a permission error when attempting to use ceph-deploy 
> to create an mgr on a recently upgraded jewel->luminous ceph cluster.  
> I've attempted to track down the permission, but so far no success.   
> I'm doing this on a dev environment so I can replicate:
>
> Start with a sample jewel release (one admin node with mon, two 
> storage nodes with one osd each).  ceph preflight complete (cephinst 
> user with sudo, ntp, etc):
>
> On each storage node, the osd disk is formatted as xfs and mounted on 
> /var/lib/ceph/osd/ceph-X, and "chown ceph:ceph /var/lib/ceph/osd/ceph-X"
>
> On admin server:
>
> |su cephinst|
> |mkdir sci-cluster|||
> |cd sci-cluster|
> |ceph-deploy ||new| |cephtest-admin|
> |ceph-deploy install cephtest-admin cephtest-stor|||1 cephtest-stor|2||
> ceph-deploy mon create-initial||
> ||ceph-deploy osd prepare |||cephtest-stor|1:/var/lib/ceph/osd/ceph-||0||
> ||ceph-deploy osd ||||prepare 
> cephtest-stor||2:/var/lib/ceph/osd/ceph-||1||
> |||ceph-deploy osd activate 
> |||cephtest-stor|1||||||||||:/var/lib/ceph/osd/ceph-||0||
> ||ceph-deploy osd ||||activate 
> cephtest-stor||2:/var/lib/ceph/osd/ceph-||1||
> |ceph-deploy admin cephtest-admin ||||cephtest-stor|1 |||||cephtest-stor|2
> ||
> |ceph osd pool set rbd size ||2
> sudo chmod a+r /etc/ceph/ceph.client.admin.keyring
>
> At this point I have a working, healthy jewel cluster as reported by 
> "ceph -s".  I then upgrade the mon and try to deploy an mgr service:
>
> sudo sed -i 's/jewel/luminous/g' /etc/yum.repos.d/ceph-deploy.repo
> ||||sudo sed -i 's/jewel/luminous/g' /etc/yum.repos.d/ceph.repo
> sudo systemctl stop ceph-mon at cephtest-admin.service
> sudo yum clean all
> sudo yum update|
> |sudo systemctl start ceph-mon at cephtest-admin.service
>
>
> "ceph -s" reports HEALTH_OK, but "mgr: no daemons active". Attempt to 
> deploy a mgr service with ceph-deploy and it fails:
>
> ceph-deploy mgr create cephtest-admin
> [ceph_deploy.conf][DEBUG ] found configuration file at: 
> /home/cephinst/.cephdeploy.conf
> [ceph_deploy.cli][INFO  ] Invoked (1.5.39): /usr/bin/ceph-deploy mgr 
> create cephtest-admin
> [ceph_deploy.cli][INFO  ] ceph-deploy options:
> [ceph_deploy.cli][INFO  ]  username                      : None
> [ceph_deploy.cli][INFO  ]  verbose                       : False
> [ceph_deploy.cli][INFO  ]  mgr                           : 
> [('cephtest-admin', 'cephtest-admin')]
> [ceph_deploy.cli][INFO  ]  overwrite_conf                : False
> [ceph_deploy.cli][INFO  ]  subcommand                    : create
> [ceph_deploy.cli][INFO  ]  quiet                         : False
> [ceph_deploy.cli][INFO  ]  cd_conf                       : 
> <ceph_deploy.conf.cephdeploy.Conf instance at 0x237b320>
> [ceph_deploy.cli][INFO  ]  cluster                       : ceph
> [ceph_deploy.cli][INFO  ]  func                          : <function 
> mgr at 0x230ac08>
> [ceph_deploy.cli][INFO  ]  ceph_conf                     : None
> [ceph_deploy.cli][INFO  ]  default_release               : False
> [ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts 
> cephtest-admin:cephtest-admin
> [cephtest-admin][DEBUG ] connection detected need for sudo
> [cephtest-admin][DEBUG ] connected to host: cephtest-admin
> [cephtest-admin][DEBUG ] detect platform information from remote host
> [cephtest-admin][DEBUG ] detect machine type
> [ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
> [ceph_deploy.mgr][DEBUG ] remote host will use systemd
> [ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to cephtest-admin
> [cephtest-admin][DEBUG ] write cluster configuration to 
> /etc/ceph/{cluster}.conf
> [cephtest-admin][DEBUG ] create path if it doesn't exist
> [cephtest-admin][INFO  ] Running command: sudo ceph --cluster ceph 
> --name client.bootstrap-mgr --keyring 
> /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create 
> mgr.cephtest-admin mon allow profile mgr osd allow * mds allow * -o 
> /var/lib/ceph/mgr/ceph-cephtest-admin/keyring
> [cephtest-admin][ERROR ] 2017-10-18 10:42:00.460124 7f9325bcd700  0 
> librados: client.bootstrap-mgr authentication error (1) Operation not 
> permitted
> [cephtest-admin][ERROR ] [errno 1] error connecting to the cluster
> [cephtest-admin][ERROR ] exit code from command was: 1
> [ceph_deploy.mgr][ERROR ] could not create mgr
>
> It looks like cephx was set up properly:
>
> ceph auth list
> installed auth entries:
>
> osd.0
>     key: AQChOOZZe1PVGBAAGFRiziREnCm8TZL3QwuFnw==
>     caps: [mgr] allow profile osd
>     caps: [mon] allow profile osd
>     caps: [osd] allow *
> osd.1
>     key: AQDEOOZZQVqUORAALnPX6+tuhKM33+bveQxiDw==
>     caps: [mgr] allow profile osd
>     caps: [mon] allow profile osd
>     caps: [osd] allow *
> client.admin
>     key: AQD0N+ZZUSluEBAATIGq+KK3LdUYZHw4RWToXg==
>     caps: [mds] allow *
>     caps: [mgr] allow *
>     caps: [mon] allow *
>     caps: [osd] allow *
> client.bootstrap-mds
>     key: AQD1N+ZZZSnPABAANDqpSE9g6MqfmXglGzn6Nw==
>     caps: [mgr] allow r
>     caps: [mon] allow profile bootstrap-mds
> client.bootstrap-mgr
>     key: AQBnZ+dZzitjIxAArptc11qf8UuPq5QXFy+I9Q==
>     caps: [mon] allow profile bootstrap-mgr
> client.bootstrap-osd
>     key: AQD0N+ZZrLv5HRAAbeLByJJyplQEvjz1o1N8kg==
>     caps: [mgr] allow r
>     caps: [mon] allow profile bootstrap-osd
> client.bootstrap-rgw
>     key: AQD0N+ZZFL/hLxAAR9WbRd/ETL/GJ+NKXum2iA==
>     caps: [mgr] allow r
>     caps: [mon] allow profile bootstrap-rgw
>
>
> Any insight or hints would be appreciated.
> Thanks
> Gary
>
> PS. on a previous attempt I did upgrade the ceph versions on the osds 
> as well before attempting to deploy the mgr; same result.
>
> |||
> -- 
> Gary Molenkamp			Computer Science
> Systems Administrator		University of Western Ontario
> molenkam at uwo.ca                  http://www.csd.uwo.ca
> (519) 661-2111 x86882		(519) 661-3566
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Gary Molenkamp			Computer Science
Systems Administrator		University of Western Ontario
molenkam at uwo.ca                 http://www.csd.uwo.ca
(519) 661-2111 x86882		(519) 661-3566

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20171018/847f7804/attachment.html>


More information about the ceph-users mailing list