[sepia] Pets and Cattle

Loic Dachary loic at dachary.org
Fri Sep 4 03:55:34 PDT 2015


Now that we have the OpenStack backend for teuthology (thanks Zack for your patience with that)[1] and an OpenStack tenant with large quotas to add to the sepia lab, we're going to start using it in various ways. To start on the right foot, I propose we separate cattle from pets so the pets are not trampled by the cattle. 

When the OpenStack backend launches hundreds of virtual machines and creates hundreds of disk volumes, it also deletes all that eventually. And when something bad happens, a script collects the left over and delete them. There are safeguards : multiple teuthology clusters can co-exist within the same OpenStack tenant and will not destroy each other VM. In the typical use case, when the teuthology suite completes, its results are uploaded to public location [2] and there is nothing left in the OpenStack tenant.

The OpenStack cluster can also be used for hosting gitbuilder machines, bots, secondary DNS servers and what not. The cheapest instance costs 3 euros per month with 2GB RAM, 1 vCPU, 10 GB disk: there is plenty of uses cases for that. These are long lived machines which are not expected to be destroyed.

I would be extremely nervous to know that teuthology operates in the same OpenStack tenant as long lived instances. It's not a matter of knowing if a pet will be slaughtered by teuthology, it's a matter of when. Bugs are bound to happen and transform teuthology into an instrument of mass destruction. 

To avoid that unfortunate scenario, I propose that long lived machines are hosted in a separate tenant. The OpenStack provider allows for more than one tenant to be created under the same account. I created this tenant (called Pets), renamed the current tenant (Cattle) and asked the provider that the quotas are set to limits higher than the defaults and (hopefully) suitable to the use case of long lived machines. That is:

  100 instances
  400 vCPUS
  100 volumes
  10TB disk

It's not set in stone: the provider can increase the quotas with one week delay. And if this turns out to be a bad idea, the tenant can be left unused and won't cost anything.

What do you think ?


[1] https://github.com/ceph/teuthology/#openstack-backend
[2] http://teuthology-logs.public.ceph.com/

Loïc Dachary, Artisan Logiciel Libre

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ceph.com/pipermail/sepia-ceph.com/attachments/20150904/dcd9d0c0/attachment.pgp>

More information about the Sepia mailing list