GitLab

This is probably true for Xen too, but in some cases, the
vnodesetup early-release hackwaitandexit timeout of 30 seconds
causes a race condition.  Normally, the first node sets up
significant network state, and sometimes flips MAC addresses
around from interface to interface -- OR puts a physical interface
into a bridge, then changes the bridge's MAC address.  There is a
short window of time where both the bridge and the new member
interface share a MAC address -- and if the tmcc ifconfig assembly
process for vnodes following the first vnode resolves
the wrong device's MAC address and uses that to flesh out the
ifconfig info, the vnodesetup will be in a world of hurt (i.e., you
might see an attempt to make a vlan device out of a vlan device).
The chance of this happening is miniscule, but I've seen it.

So, at least for docker for now, we protect the first vnode against
the 30-second timeout in vnodesetup hackwaitandexit, and we wait for the
actual running file to be written, or error.

This is probably applicable to any linux mkvnode.pl path, but I suppose
it would have been another hundred thousand vnode creates before I saw
it again.

Under high load, of course we can have DNS problems.  However, perl
seems to get stuck on retry; it's like the nak gets cached (which would
be extremely odd, but can't argue with the evidence).

Anyway, if resolution continues to fail, give up and feed the name to
iptables, and let it try :).

(All we need to do while holding the global lock is allocated IFBs; the
generation of routing scripts and traffic shaping scripts is both
unlikely to fail and potentially slow due to running djikstra.  So, also
let the vnode early release prior to those things, immediately after IFB
allocation.)

Docker vnodes require the full image path anytime the vnode is created,
even if the image in question already exists on the vhost.  This is
because emulab custom docker images are fully-qualified with their
hosting private registry, so we need that detail even if we're not
re-pulling the image; i.e., if the vnode gets destroyed and recreated
after its initial reload.

Prior to this commit, I had reused all the libvnode:: network functions,
but that requires these system-wide indexes to be built (i.e. bridges,
ifaces) before using the helpers -- and on any network state change.
Those indexing functions take a long time on heavily-loaded systems
(i.e., 5k processes, hundreds of ifaces).  The helpers become very fast;
but for the case of large numbers of vnodes on one vhost, it is better
to hold the global lock for less time, and pay a higher per-helper cost,
instead of making use of a nicely-indexed cache.

So now we cache nothing and (hopefully) use /sys intelligently to avoid
forking unnecessarily.

and so we have to look through the image versions for each image
in the alias.

When running hundreds of containers per vhost, need a bit more
time for these.  I really need to reassess which of these is really
necessary any longer.

The docker engine API may not be available by the time bootvnodes runs;
this can happen, for instance, if many containers on the vhost and
if some had stale sandboxes.

(Also, pass vhost node attributes to rootPreConfig.)

Seems that when the host is heavily loaded, docker will mix in control
chars, etc, to the output of a container.  We try to filter as much as
possible, but sadly it's not just control chars; printable chars can
sneak through.

(linkmap, lt(p)map can all grow large on very large topos; so make
space for them.

reservations that have started (start time in the past), the date picker
was actually showing the current day instead, need an option to make it
show a date in the past.

1) When selecting the day from the calendar, init the start time to 7am
   instead of leaving it unset. But do not do this if the hour is
   already set.

2) When using the search button, see if the reservation can be shifted
   to start at 7am. But do not pick a new slot if it can't, we want to
   give the user the earliest slot available.

only people would take the time to report things that do not work as
expected!

show the user, there was a path in which this was not happening.

1. When editing a reservation, request the forecast info specifically
   for the project the reservation is attached to. This is really
   important when looking at it as an admin, since we want to edit in
   the user's project context.

2. But to do that, we have to wait for reservation info to come back,
   and then ask for the forecast data.

3. And if the user clicks on the "Search" button before the res data
   comes back, we have to wait for it before we can search.

4. Fix the Search function; I was handling duplicates in the forecast
   data incorrectly.

the generic error code for all rspec errors, so that the Portal does
the right thing when displaying the error.