[ceph-users] Customized Crush location hooks in Mimic

Oliver Freyermuth freyermuth at physik.uni-bonn.de
Fri Nov 30 06:46:03 PST 2018


Dear Cephalopodians,

sorry for the spam, but I found the following in mon logs just now and am finally out of ideas:
------------------------------------------------------------------------------------------
2018-11-30 15:43:05.207 7f9d64aac700  0 mon.mon001 at 0(leader) e3 handle_command mon_command({"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["1"]} v 0) v1
2018-11-30 15:43:05.207 7f9d64aac700  0 log_channel(audit) log [INF] : from='osd.1 10.160.12.101:6816/90528' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "hdd", "ids": ["1"]}]: dispatch
2018-11-30 15:43:05.208 7f9d64aac700  0 mon.mon001 at 0(leader) e3 handle_command mon_command({"prefix": "osd crush create-or-move", "id": 1, "weight":3.6824, "args": ["datacenter=FTD", "host=osd001", "root=default"]} v 0) v1
2018-11-30 15:43:05.208 7f9d64aac700  0 log_channel(audit) log [INF] : from='osd.1 10.160.12.101:6816/90528' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":3.6824, "args": ["datacenter=FTD", "host=osd001", "root=default"]}]: dispatch
2018-11-30 15:43:05.208 7f9d64aac700  0 mon.mon001 at 0(leader).osd e2464 create-or-move crush item name 'osd.1' initial_weight 3.6824 at location {datacenter=FTD,host=osd001,root=default}
------------------------------------------------------------------------------------------
So the request to move to datacenter=FTD arrives at the mon, but no action is taken, and the OSD is left in FTD_1.

Cheers,
	Oliver

Am 30.11.18 um 15:25 schrieb Oliver Freyermuth:
> Dear Cephalopodians,
> 
> further experiments revealed that the crush-location-hook is indeed called!
> It's just my check (writing to a file in tmp from inside the hook) which somehow failed. Using "logger" works for debugging.
> 
> So now, my hook outputs:
> host=osd001 datacenter=FTD root=default
> as explained before. I have also explicitly created the buckets beforehand in case that is needed.
> 
> Tree looks like that:
> # ceph osd tree
> ID  CLASS WEIGHT   TYPE NAME            STATUS REWEIGHT PRI-AFF
>   -1       55.23582 root default
>   -9              0     datacenter FTD
> -12       18.41194     datacenter FTD_1
>   -3       18.41194         host osd001
>    0   hdd  3.68239             osd.0        up  1.00000 1.00000
>    1   hdd  3.68239             osd.1        up  1.00000 1.00000
>    2   hdd  3.68239             osd.2        up  1.00000 1.00000
>    3   hdd  3.68239             osd.3        up  1.00000 1.00000
>    4   hdd  3.68239             osd.4        up  1.00000 1.00000
> -11              0     datacenter FTD_2
>   -5       18.41194     host osd002
>    5   hdd  3.68239         osd.5            up  1.00000 1.00000
>    6   hdd  3.68239         osd.6            up  1.00000 1.00000
>    7   hdd  3.68239         osd.7            up  1.00000 1.00000
>    8   hdd  3.68239         osd.8            up  1.00000 1.00000
>    9   hdd  3.68239         osd.9            up  1.00000 1.00000
>   -7       18.41194     host osd003
>   10   hdd  3.68239         osd.10           up  1.00000 1.00000
>   11   hdd  3.68239         osd.11           up  1.00000 1.00000
>   12   hdd  3.68239         osd.12           up  1.00000 1.00000
>   13   hdd  3.68239         osd.13           up  1.00000 1.00000
>   14   hdd  3.68239         osd.14           up  1.00000 1.00000
> 
> So naively, I would expect that when I restart osd.0, it should move itself into datacenter=FTD.
> But that does not happen...
> 
> Any idea what I am missing?
> 
> Cheers,
>      Oliver
> 
> 
> 
> Am 30.11.18 um 11:44 schrieb Oliver Freyermuth:
>> Dear Cephalopodians,
>>
>> I'm probably missing something obvious, but I am at a loss here on how to actually make use of a customized crush location hook.
>>
>> I'm currently on "ceph version 13.2.1" on CentOS 7 (i.e. the last version before the upgrade-preventing bugs). Here's what I did:
>>
>> 1. Write a script /usr/local/bin/customized-ceph-crush-location. The script can be executed by user "ceph":
>>    # sudo -u ceph /usr/local/bin/customized-ceph-crush-location
>>    host=osd001 datacenter=FTD root=default
>>
>> 2. Add the following to ceph.conf:
>>   [osd]
>>   crush_location_hook = /usr/local/bin/customized-ceph-crush-location
>>
>> 3. Restart an OSD and confirm that is picked up:
>>   # systemctl restart ceph-osd at 0
>>   # ceph config show-with-defaults osd.0
>>    ...
>>    crush_location_hook        /usr/local/bin/customized-ceph-crush-location  file
>>    ...
>>    osd_crush_update_on_start  true                                           default
>>    ...
>>
>> However, the script is not executed, and I can ensure that since the script should also write a log to /tmp, which is not created.
>> Also, the "datacenter" type does not show up in the crush tree.
>>
>> I have already disabled SELinux just to make sure.
>>
>> Any ideas what I am missing here?
>>
>> Cheers and thanks in advance,
>>      Oliver
>>
> 
> 
> 
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5432 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20181130/7f7bf86e/attachment.bin>


More information about the ceph-users mailing list