GitLab

Reboots of the ctl node for the Liberty version would result in
failures to startup mysql, and this renders all openstack services
inoperable.

Recall that in the common case (because we have many testbeds whose
nodes only have one expt interface), we setup the openstack mgmt lan as
a VPN over the control net between all the nodes, served from the nm
node.

Well, mysql binds to and listens on the ip addr of the mgmt net device,
and when the ctl node is rebooted, mysql starts long before openvpn can
bring up the vpn client net device.  Moreover, rabbitmq would fail to
start for the same reason, and rabbitmq is the AMQP messaging service
that underlies all openstack RPC.

For various reasons, it's not sufficient to just make the mysql
initscript (which on 15.10 is still legacy LSB!) depend on the openvpn
legacy LSB initscript.

So I wrote a little initcript (embedded in setup-controller.sh) that
spins in a sleep 1; loop, looking for the mgmt net to get its known IP
from the openvpn client.  It has reverse dependency on mysql, so it runs
to completion before mysql starts.

Then, we had to handle the rabbitmq case... but rabbitmq has a modern
systemd unit file, not an LSB initscript.  So I wrote a systemd unit
file that invokes my mgmt net LSB initscript to wait for the mgmt net
IP... and that has a reverse dep on rabbitmq-server.service.

Now all is good.  mysql and rabbitmq-server are certainly blocked for a
few extra seconds, while the VPN comes up, but all the openstack
services themselves are written defensively to handle RPC server
disconnects, or database disconnects (doh).

User has to include a tarball containing a single dir which in turn
contains a script called setup.sh .  This tarball must be installed into
/tmp/setup/ext .  Minimal, but who cares, it works for now.

This was to try to solve the performance problem; no luck here.

Apparently oslo.service handles stop signals (like those from systemd)
in a bad way.  When systemd stops a service, it can drive the service
into an exception (in one of its threads, I think) and I assume systemd
waits until its timeout and then kills it more firmly.  This patch went
in to oslo.service 1.2.0 or so, but of course we're stuck back in the
stone age still, around 0.9 or whatever.  So apply the patch, stupid.
We start and stop services a lot as we're setting up.

Memcache + Keystone + WSGI/Apache seems to cause a problem where
Keystone is effectively unavailable (internal errors) for about a
minute... then it comes back by itself.

So we disable it by default.  The docs default to using it, but this is
far from the first time the doc defaults trigger bugs or are simply bad
configuration!

Add Liberty support.

Add keystone v3 support.  Now you can choose which version of keystone
to run... all combinations tested exception Juno with v3.

Make node type and link speed configurable.

Make token and session timeouts much longer by default (so people don't
get logged out so quickly), but also configurable.

Keystone is now served by WSGI through Apache on Kilo and Liberty.

Memcached keystone token caching is disabled for now; it causes
intermittent problems; so using SQL for now.

Add localhost to /etc/hosts file.  This doesn't cause problems anymore,
if it ever did.

We now use the `openstack' CLI command for >= Kilo, instead of the
per-service client CLI tools.

Stick with ovs agent even in Liberty -- even though the default is now
linuxbridge, it seems.

In general, get rid of nearly all the rest of the cat <<EOF ... EOF
stuff and replace it with crudini --set/--del.  A touch slower, but much
cleaner.

Also in general, improve the Kilo support so that it more closely
matches the docs.

Also, adds a geni-lib script that generates an rspec instead of printing
it (although print still works at portal) and generates input for
CM::AddNodes() when requested.  This generator is stateful; it tries
to avoid generating new nodes with previously-used IPs or client_ids;
thus it is a separate object.  It is designed so that it can be imported
into a script, and the importing script can look for special
DYNSLICE_GENERATOR variables to use its rspec foo to create a slice and
add nodes in some semantic way.

This is the fallback if we don't get an encrypted passwd from the
manifest.

Quit trying to apt-get packages if they're installed, unless the
user selects the new DO_APT_UPGRADE option.  Always install was nice
in the beginning, but it is no longer the best use case, and it can
cause uncertainty when failures happen (i.e., if new versions of
packages get installed that the scripts can't handle).  So now there
are three apt options in the scripts and in the geni-lib script:

DO_APT_UPDATE -- updates the apt cache (often hard to do pkg
  install/upgrade if the cache is out of date); defaults to 1
DO_APT_INSTALL -- if this is set 0, we don't install *anything*
  other than critical deps (think python-m2crypto); defaults to 1
DO_APT_UPGRADE -- if this is set 1, we always run apt-get install
  to either install and/or upgrade OpenStack packages and deps.
  The big change is that this now defaults to 0 -- so packages are
  not upgraded from their current versions if they exist.

Mostly, we keep trying to start the app until it succeeds.  It seems
that the server needs some time to get started before it's safe to
start up the app.  Could be as long as 5-6 seconds.  We (and openstack)
can't do anything if the app isn't running in the server.

There's a fallback to this too -- if for whatever reason, we don't get
a password, we generate a random one and email it to the user.  Not
perfect, but still better than letting a null passwd through.

This allows us to operate on images with the openstack packages
installed, but services disabled to speed up boot.

This is really, really ugly.  So, keystone and nova don't offer us a way to
upload keypairs on behalf of another user.  Recall, we're using the adminapi
account to do all this setup, because we don't know the admin password.  So,
the below code adds all the keys as 'adminapi', then we dump the sql, do some
sed magic on it to get rid of the primary key and change the user_id to the
real admin user_id, then we insert those rows, then we cleanup the lint.  By
doing it this way, we eliminate our dependency on the SQL format and column
names and semantics.  We make two assumptions: 1) that there is only one field
that has integer values, and 2) the only field that is an exception to #1 is
called 'deleted' and we just set those all to 0 after we jack the whacked sql
in.  Ugh, it's worse than I hoped, but whatever.

This is one of the sickest one-liners I've ever come up with, but it
works like a charm!

Can't suck up all the disk with these...

The lack of this in the distro packages (bugfix) triggers another bug
and crashes part of ceilometer, so just write this file if it's not
there.

Switch to using the newer messagingv2 notification_driver (I note
they've updated the Juno docs to use this too, so maybe it was available
then and the docs just weren't updated).

Handle a few Kilo-new config file things.  They didn't seem to be
hurting anything to do it the Juno way, but now it's "correct".

Make sure to add the glance registry, not just the api, for images.

We download the latest versions, and edit them.  However, the online
versions have bugs in the pipeline configuration, and the default
pipeline config no longer supports keystone auth through the proxy
server.  I don't know why, it means their online setup docs result in
non-working configurations, but whatever.

So, always set the pipeline now, and make sure it has keystoneauth.

Openstack doesn't yet have 1) serial console client builtin to
dashboard, nor 2) serial console log + r/w serial console access.

So right now, you have to choose if you want to enable r/w console
access via CLI client, or if you want to be able to view serial logs in
the dashboard web UI; default is the latter (logs).

If you enable r/w consoles, we download a simple websocket console
client, setup a little frontend script, so users can type

  $ /root/setup/novaconsole.sh <instancename>

and get to the console.  Escape is ~.  In addition to the console CLI
tool, we have to grab the latest version of the python websocket
library, cause the one in Ubuntu 15.04 is horribly out of date and
doesn't seem to support binary/base64 websockets, which the console CLI
tool requires (as does the server).

Setup several kinds of networks: tunnels, flat networks, flat networks
multiplexed via vlans over physical networks (where openstack doesn't
manage the vlan ids), and real vlan networks (where openstack *does*
manage the vlan ids).  Tunnels always go over the first flat data net.

Be very flexible in terms of assigning IPs; generate them ourselves
if they dind't come to us, or if user wants to use our generated ones.
I tried to be smart (enough) with this.

Setup VNC-based consoles on x86-64; working in dashboard.

Don't put plaintext admin password in profile anymore; instead, expect
a hash of the admin password.  Replace the temp admin password in the
keystone database with the hash we get.  But, since the CLI tools
all require real user auth, setup a secondary 'adminapi' account
that is a real admin, and use that to see admin-openrc.sh for CLI
tools, and for all our configuration, and places where the services
use a real admin account to auth.  Also, push the admin password
hash all the way into our instance images.

Also add some params, like setting the admin password and opting
not to install or update any of the installed openstack packages...

Also, use apt package if available, else use pip like we used to do.