GitLab

trying to copy it via NFS.

Add code to restart the tmcc proxy, which appears to die off for no
apparent reason. I now waitpid for it while the jail is running, and
restart if it exits.

At Mike's suggestion temporarily wrap tmcc proxy in a ktrace to see if we
can debug this problem. The ktrace file (using -a option) is written as
/var/emulab/jails/nodeid/tmcc.ktrace.

See initial revision for tbsetup/gentopfile for more details. Note
that a gzipped version of the file is also created by gentopofile, and
we try to get that first to cut down on the amount of data copied over
(the topomap is highly compressable).

the pnode rebooted, the file got multiple entries causing the generated
gated config to have syntax errors.

in the timestamps, and seeing what was taking so much time.

* Increase the overlap from two to four seconds (have vnodesetup exit
  earlier) so that the next jail can start sooner. Using getrusage it
  seemed pretty clear that there were plenty of cycles available to
  start the next jail going a little earlier. Instead of looking for
  the watchdog pid file to appear, I touch (in mkjail) a specific file
  in the jail directory , and have vnodesetup look for that file
  instead of the pid.

* Cut back the number of inodes in the filesystem! Sheesh, we ere
  creating 137,000 inodes for a 50MB vnode filesystem. Takes 1
  second to do that. Reduced to 30,000 and the time when to .1
  seconds. Could probably go even lower.

* Add -p option to fsck when restoring a jail from an existing
  filesystem. No need to force check the filesystem if its clean.

* Instead of running MAKDEV inside each jail filesystem (which was
  also taking .8 seconds to do), do it just once and stash a copy in
  /var/emulab/jails. Each jail does a "hier cp" of that master copy
  into the jail filesystem. Reduced the time to .05 seconds.

So, for the results:

Original time to boot a jail: 8.2 seconds, with the last two seconds
overlapped with the next one.

Original time to reinstate a jail after halting it (filesystem left
behind): 6.5 seconds, with the last 1.5 seconds overlapped with the next
one.

Original time to boot 16 jails on a pc850: 2 minutes and 12 seconds.

---------------------------------------------------------------------------

New time to boot a jail: 6.2 seconds, with the last 4 seconds overlapped
with the next one.

New time to reinstate a jail after halting it: 6 seconds, with the last 4
seconds overlapped with the next onw.

New time to boot 16 jails on a pc850: 88 seconds. Does this number ring any
bells? :-)

It didn't fix the problem and it makes nodes fail on startup sometimes.

filesystems.  Also add sysctl call to increase socketbuf size for NFS
sockets.

this was to add soft reconfig support so that nodes could be
reconfigured without having to reboot them. This appears to work, and
has been tested with jails getting moved around. I've also tested the
new code on the MFS, but still no testing has been done on PLAB nodes.

The main change is that most of the code moved out of libsetup.pm, and
was split into constituent rc scripts, each of which does its own
thing, including cleaning up and preparing for making an image. Most
of that central knowledge has been moved out into the scripts. Still more
to do but this was a good start.

From the code:
        # N.B. privnet is setup first even before the default route.
        # This is because the gateway for the default route is on the
        # privnet, but our control net interface is not (because it has
        # a 255.255.255.255 netmask by virtue of being an alias on the
        # real control interface).  Thus there is no way to use the
        # default route until the privnet route is up (ARP complains
        # about "host is not on local network").

number of connections to tmcd, and the resulting number of DB queries.
Currently thats about 24 per node when it boots. Each vnode adds
another 24 or so. The new approach is to use the "fullconfig" command,
which dumps the entire config in one shot, saving about 20 of those
connections. We still need to do the status/state commands for real of
course. When a node boots, it requests the fullconfig; the client side
takes this fullconfig, and dumps the individual sections to
/var/emulab/boot/tmcc/section_name. Subsequent requests first look for
it locally in the above named files, falling back to real tmcc if none
exists. The update command also refreshes the cache.

Tested for jails and plab node vservers as well.

All to try to avoid loopback lockups.

Previously, if you accessed a phys node by its real IP address (155.101.132.N)
from inside a jail, it used the default route going to the router and back.
Now it will put it out directly on the control net.

The main reason for this is so that minibed vnodes can reach their boss and
ops nodes.  Vnodes in mini don't have a default route that works, so we needed
a way to get to boss/ops.

Note that gated will remove this route when it starts up.  Thus, vnodes will
revert to using the default route to get to the real control net.  On mainbed,
this will still work (hop through the router as before).  On mini, it won't
work at all.

Moral: don't use session routing and vnodes on mini.

batch jobs will end! There were two problems; 1) The socket was
744, so mere users were not able to run tmcc. 2) libsetup was not
even using the proxy socket cause it was never looking for it.

We have some evidence that some application is "chmod 600"-ing
/dev/console and, in the process, /dev/null.

Remove the code to create vn devices, bootvnodes does this now.

Add a -V option to mkjail.pl telling it to use the virtual control net
IP (172.16.0.1) for the default route instead of the real control net
IP (155.101.132.1).  In this case, it also assigns a virtual control net
IP alias (172.17.<pnode>.0) to the physical interface so that the
aforementioned routes can actually be installed.

Add the same -V option (only valid with -j) in vnodesetup so that we
can get it through from bootvnodes.  (Also cleaned up the usage message
some)

The virtual control net address/mask are currently constants in mkjail.pl.
At some point they can/should be changed to config time options so that we
can easily use this on minibed too.

In theory, this (-V) shouldn't be an option and we should just use
virtual control net routing all the time.  But I want to get some more
testing and we have to resolve the minibed conflicts first.  So for now,
this is an option and it is off by default.

Haven't tied this option in with gated yet, so creating a vnode experiment
with Session routing still won't work.  If you want to do that, create the
experiment, watch gated blow up right and left, login to all the physical
nodes and change bootvnodes to add -V to the invocation of vnodesetup,
and then reboot everything.  Now it should be working!

Symlink /opt to /local/pid/eid/vnodeid so that its easier to install
per-jail software.

configs

generation code

Also a couple perf hacks:

* Local vnodes can start with the password/group file from the
  physnode, since locally they will be the same anyway. This avoids a
  blizzard of accounts requests at startup, which is by far the
  biggest chunk of data returned (well, except for host tables).

* To help serialize boot startup, vnodesetup now waits for the jail to
  finish starting up before it exits. It does this via a "goofy"
  mechanism I will not bother to describe.

recreated at jail reboot. Note this file is *inside* the jail and
consists of the common routes (gateway, loopback, etc).

course; its an optional argument.

Add group stuff; we get a group (csh, tsch) and a routine maps that
into a valid path to use in the pw commands.

Remove jaildog, and use the common watchdog instead.

like we set hostnames of local nodes (v0.wa-jail-link.testbed.emulab.net).
Also build a proper /etc/hosts file just like local nodes inside the
jail. Minor change to dohostnames(); do not munge /etc/hosts in place!
Generate a new copy and then atomically move into place. Phew, this
was a close one! Mike has me really worried about race conditions now
that I screwed the pooch in the frisbee client.

Add new crontab file install to run a stub crontab inside jails;
basically, runs the atq and the newsyslog stuff. Other stuff is
turned off including periodic.