GitLab

Docker containers may be (and default to, and in the shared host case,
must be) deprivileged; thus, they cannot mount devices, much less tell
the kernel (via iscsi userspace tools, etc) to make devices.

Therefore, we must setup any storage backing devices (temp LVs, iscsi
attachments) outside the container.  This commit makes that possible for
rc.storage and linux liblocstorage.  Basically, rc.storage now supports
(for the Linux liblocstorage and Docker) the -j vnodeid calling
convention; and if it's being called on behalf of a vnodeid, it uses
per-vnodeid fstab for any mounts, storage.conf for its state; etc.

I modified libvnode_docker to *not* create virtual networks for
remote blockstore links, because those are pinned to /30s, and thus I
have no client blockstore link address to place on a device in the root
context.  However, I (ab)used the existing Docker network setup for the
blockstore links, and that all happens the same as it used to; we just
no longer create the Docker virtual network nor attach the container to
it.

Finally, I modified tmcd dostorageconfig slightly to return
HOSTIP/HOSTMASK for remote blockstores; and now
libsetup::getstorageconfig will use HOSTIP in preference to its own
HOSTID->HOSTIP translation.  I had to do this so that libvnode_docker in
the root context would not have to go through the mess of translating
HOSTID on behalf of a vnode.

* I started out to add just deferred aggregates; those that are offline
  when starting an experiment (and marked in the apt_aggregates table as
  being deferable). When an aggregate is offline, we add an entry to the
  new apt_deferred_aggregates table, and periodically retry to start the
  missing slivers. In order to accomplish this, I split create_instance
  into two scripts, first part to create the instance in the DB, and the
  second (create_slivers) to create slivers for the instance. The daemon
  calls create_slivers for any instances in the deferred table, until
  all deferred aggregates are resolved.

  On the UI side, there are various changes to deal with allowing
  experiments to be partially create. For example used to wait till we
  have all the manifests until showing the topology. Now we show the
  topo on the first manifest, and then add them as they come in. Various
  parts of the UI had to change to deal with missing aggregates, I am
  sure I did not get them all.

* And then once I had that, I realized that "scheduled" experiments was
  an "easy" addition, its just a degenerate case of deferred. For this I
  added some new slots to the tables to hold the scheduled start time,
  and added a started stamp so we can distinguish between the time it
  was created and the time it was actually started. Lots of data.

  On the UI side, there is a new fourth step on the instantiate page to
  give the user a choice of immediate or scheduled start. I moved the
  experiment duration to this step. I was originally going to add a
  calendar choice for termination, but I did not want to change the
  existing 16 hour max duration policy, yet.

We now try to emulate any simple COPY <src> <dst> instructions via rsync
prior to image build.

This *does* mean that artifact builder scripts must be careful to create
all necessary dirs according to the base image semantics, because the
base image content is not there when we emulate the COPY instructions.
For instance, many of the modified Dockerfile-runit and
runit-artifacts.sh files depended on built runit packages being
installed into /tmp in the final image -- but they didn't create the
/tmp dir because the COPY instruction they used was running atop a
fully-populated base image that already had /tmp.  Thus, the
runit-artifacts.sh scripts had to be changed to create /tmp with the
proper permissions.

process that hands off).

email, so that it has the correct subject tag.

case we have to ask the clearinghouse.

so that the SCS will accept it. Oh, and I learned that the SCS is
sensitive to the order of elements in the XML file! The link *must* be
after the nodes, or the SCS refuses to do anything with it. Sheesh.

when we have a candidate powder aggregate.

as per issue #440.

locally on boss.

We can probably not start ops mysqld anywhere.