<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Sorry to reply to my own question, but I noticed that the cephx
key for client.bootstrap-mgr was inconsistent with the key in
/var/lib/ceph/bootstrap-mgr/ceph.keyring.</p>
<p>I deleted the entry in ceph:</p>
<p> ceph auth del client.bootstrap-mgr</p>
<p>reran the ceph-deploy gather keys:</p>
<p> ceph-deploy gatherkeys cephtest-admin</p>
<p>and used the key entry for client.bootstrap-mgr as the key value
in /var/lib/ceph/bootstrap-mgr/ceph.keyring.</p>
<p>I was then able to successfully run the ceph-deploy to create the
mgr.<br>
</p>
I'm not sure where this inconsistent key came from as the
/var/lib/ceph/bootstrap-mgr directory doesn't appear on a fresh
install of jewel.<br>
<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 18/10/17 11:04 AM, Gary molenkamp
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:7138daa2-9576-49bf-1247-f10e257412ca@uwo.ca">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<p>I'm running into a permission error when attempting to use
ceph-deploy to create an mgr on a recently upgraded
jewel->luminous ceph cluster. I've attempted to track down
the permission, but so far no success. I'm doing this on a dev
environment so I can replicate:</p>
<p>Start with a sample jewel release (one admin node with mon, two
storage nodes with one osd each). ceph preflight complete
(cephinst user with sudo, ntp, etc):</p>
<p>On each storage node, the osd disk is formatted as xfs and
mounted on /var/lib/ceph/osd/ceph-X, and "chown ceph:ceph
/var/lib/ceph/osd/ceph-X" </p>
<p>On admin server:</p>
<code class="java plain">su cephinst</code><br>
<code class="java plain">mkdir sci-cluster</code><code class="java
plain"></code><br>
<code class="java plain">cd sci-cluster</code>
<div class="line number3 index2 alt2"><code class="java plain">ceph-deploy
</code><code class="java keyword">new</code> <code class="java
plain">cephtest-admin</code></div>
<code class="java plain">ceph-deploy install cephtest-admin
cephtest-stor</code><code class="java plain"><code class="java
plain">1 cephtest-stor</code>2</code><code class="java plain"><br>
ceph-deploy mon create-initial</code><code class="java spaces"><br>
</code><code class="java plain">ceph-deploy osd prepare </code><code
class="java plain"><code class="java plain">cephtest-stor</code>1:/var/lib/ceph/osd/ceph-</code><code
class="java value">0</code><code class="java spaces"><br>
</code><code class="java plain">ceph-deploy osd </code><code
class="java plain"><code class="java plain"><code class="java
plain">prepare cephtest-stor</code></code>2:/var/lib/ceph/osd/ceph-</code><code
class="java value">1</code><code class="java plain"><br>
</code><code class="java plain"><code class="java plain">ceph-deploy
osd activate </code><code class="java plain"><code
class="java plain">cephtest-stor</code>1</code></code><code
class="java plain"><code class="java plain"><code class="java
plain"><code class="java plain"><code class="java plain"></code></code></code>:/var/lib/ceph/osd/ceph-</code><code
class="java value">0</code><code class="java spaces"><br>
</code><code class="java plain">ceph-deploy osd </code><code
class="java plain"><code class="java plain"><code class="java
plain">activate cephtest-stor</code></code>2:/var/lib/ceph/osd/ceph-</code><code
class="java value">1</code><code class="java plain"><br>
</code>ceph-deploy admin cephtest-admin </code><code
class="java plain"><code class="java plain"><code class="java
plain">cephtest-stor</code>1 </code></code><code
class="java plain"><code class="java plain"><code class="java
plain">cephtest-stor</code>2<br>
</code></code>
<div class="line number22 index21 alt1"><code class="java plain">ceph
osd pool set rbd size </code><code class="java value">2<br>
sudo chmod a+r /etc/ceph/ceph.client.admin.keyring<br>
<br>
At this point I have a working, healthy jewel cluster as
reported by "ceph -s". I then upgrade the mon and try to
deploy an mgr service:<br>
<br>
sudo sed -i 's/jewel/luminous/g'
/etc/yum.repos.d/ceph-deploy.repo <br>
</code><code class="java value"></code><code class="java value">sudo
sed -i 's/jewel/luminous/g' /etc/yum.repos.d/ceph.repo<br>
sudo systemctl stop <a class="moz-txt-link-abbreviated"
href="mailto:ceph-mon@cephtest-admin.service"
moz-do-not-send="true">ceph-mon@cephtest-admin.service</a><br>
sudo yum clean all<br>
sudo yum update</code><br>
<code class="java value">sudo systemctl start <a
class="moz-txt-link-abbreviated"
href="mailto:ceph-mon@cephtest-admin.service"
moz-do-not-send="true">ceph-mon@cephtest-admin.service</a><br>
<br>
<br>
"ceph -s" reports HEALTH_OK, but "mgr: no daemons active".
Attempt to deploy a mgr service with ceph-deploy and it fails:<br>
<br>
ceph-deploy mgr create cephtest-admin<br>
[ceph_deploy.conf][DEBUG ] found configuration file at:
/home/cephinst/.cephdeploy.conf<br>
[ceph_deploy.cli][INFO ] Invoked (1.5.39):
/usr/bin/ceph-deploy mgr create cephtest-admin<br>
[ceph_deploy.cli][INFO ] ceph-deploy options:<br>
[ceph_deploy.cli][INFO ] username :
None<br>
[ceph_deploy.cli][INFO ] verbose :
False<br>
[ceph_deploy.cli][INFO ] mgr :
[('cephtest-admin', 'cephtest-admin')]<br>
[ceph_deploy.cli][INFO ] overwrite_conf :
False<br>
[ceph_deploy.cli][INFO ] subcommand :
create<br>
[ceph_deploy.cli][INFO ] quiet :
False<br>
[ceph_deploy.cli][INFO ] cd_conf :
<ceph_deploy.conf.cephdeploy.Conf instance at 0x237b320><br>
[ceph_deploy.cli][INFO ] cluster :
ceph<br>
[ceph_deploy.cli][INFO ] func :
<function mgr at 0x230ac08><br>
[ceph_deploy.cli][INFO ] ceph_conf :
None<br>
[ceph_deploy.cli][INFO ] default_release :
False<br>
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts
cephtest-admin:cephtest-admin<br>
[cephtest-admin][DEBUG ] connection detected need for sudo<br>
[cephtest-admin][DEBUG ] connected to host: cephtest-admin <br>
[cephtest-admin][DEBUG ] detect platform information from
remote host<br>
[cephtest-admin][DEBUG ] detect machine type<br>
[ceph_deploy.mgr][INFO ] Distro info: CentOS Linux 7.4.1708
Core<br>
[ceph_deploy.mgr][DEBUG ] remote host will use systemd<br>
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to
cephtest-admin<br>
[cephtest-admin][DEBUG ] write cluster configuration to
/etc/ceph/{cluster}.conf<br>
[cephtest-admin][DEBUG ] create path if it doesn't exist<br>
[cephtest-admin][INFO ] Running command: sudo ceph --cluster
ceph --name client.bootstrap-mgr --keyring
/var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create
mgr.cephtest-admin mon allow profile mgr osd allow * mds allow
* -o /var/lib/ceph/mgr/ceph-cephtest-admin/keyring<br>
[cephtest-admin][ERROR ] 2017-10-18 10:42:00.460124
7f9325bcd700 0 librados: client.bootstrap-mgr authentication
error (1) Operation not permitted<br>
[cephtest-admin][ERROR ] [errno 1] error connecting to the
cluster<br>
[cephtest-admin][ERROR ] exit code from command was: 1<br>
[ceph_deploy.mgr][ERROR ] could not create mgr<br>
<br>
It looks like cephx was set up properly:<br>
<br>
ceph auth list<br>
installed auth entries:<br>
<br>
osd.0<br>
key: AQChOOZZe1PVGBAAGFRiziREnCm8TZL3QwuFnw==<br>
caps: [mgr] allow profile osd<br>
caps: [mon] allow profile osd<br>
caps: [osd] allow *<br>
osd.1<br>
key: AQDEOOZZQVqUORAALnPX6+tuhKM33+bveQxiDw==<br>
caps: [mgr] allow profile osd<br>
caps: [mon] allow profile osd<br>
caps: [osd] allow *<br>
client.admin<br>
key: AQD0N+ZZUSluEBAATIGq+KK3LdUYZHw4RWToXg==<br>
caps: [mds] allow *<br>
caps: [mgr] allow *<br>
caps: [mon] allow *<br>
caps: [osd] allow *<br>
client.bootstrap-mds<br>
key: AQD1N+ZZZSnPABAANDqpSE9g6MqfmXglGzn6Nw==<br>
caps: [mgr] allow r<br>
caps: [mon] allow profile bootstrap-mds<br>
client.bootstrap-mgr<br>
key: AQBnZ+dZzitjIxAArptc11qf8UuPq5QXFy+I9Q==<br>
caps: [mon] allow profile bootstrap-mgr<br>
client.bootstrap-osd<br>
key: AQD0N+ZZrLv5HRAAbeLByJJyplQEvjz1o1N8kg==<br>
caps: [mgr] allow r<br>
caps: [mon] allow profile bootstrap-osd<br>
client.bootstrap-rgw<br>
key: AQD0N+ZZFL/hLxAAR9WbRd/ETL/GJ+NKXum2iA==<br>
caps: [mgr] allow r<br>
caps: [mon] allow profile bootstrap-rgw<br>
<br>
<br>
Any insight or hints would be appreciated. <br>
Thanks<br>
Gary<br>
<br>
PS. on a previous attempt I did upgrade the ceph versions on
the osds as well before attempting to deploy the mgr; same
result.<br>
<br>
</code><code class="java value"></code></div>
<pre class="moz-signature" cols="72">--
Gary Molenkamp Computer Science
Systems Administrator University of Western Ontario
<a class="moz-txt-link-abbreviated" href="mailto:molenkam@uwo.ca" moz-do-not-send="true">molenkam@uwo.ca</a> <a class="moz-txt-link-freetext" href="http://www.csd.uwo.ca" moz-do-not-send="true">http://www.csd.uwo.ca</a>
(519) 661-2111 x86882 (519) 661-3566
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
ceph-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:ceph-users@lists.ceph.com">ceph-users@lists.ceph.com</a>
<a class="moz-txt-link-freetext" href="http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com">http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Gary Molenkamp Computer Science
Systems Administrator University of Western Ontario
<a class="moz-txt-link-abbreviated" href="mailto:molenkam@uwo.ca">molenkam@uwo.ca</a> <a class="moz-txt-link-freetext" href="http://www.csd.uwo.ca">http://www.csd.uwo.ca</a>
(519) 661-2111 x86882 (519) 661-3566
</pre>
</body>
</html>