GitLab

In checknode code for FreeBSD don't check the /dev/ad* device if it is a symlink.
[I think the a error in the test command for -c]

Provide automated setup of an ssh keypair enabling root to login without
a password between nodes. The biggest challenge here is to get the private
key onto nodes in such a way that a non-root user on those nodes cannot
obtain it. Otherwise that user would be able to ssh as root to any node.
This precludes simple distribution of the private key using tmcd/tmcc as
any user can do a tmcc (tmcd authentication is based on the node, not the
user).

This version does a post-imaging "push" of the private key from boss using
ssh. The key is pushed from tbswap after nodes are imaged but before the
event system, and thus any user startup scripts, are started. We actually
use "pssh" (really "pscp") to scale a bit better, so YOU MUST HAVE THE
PSSH PACKAGE INSTALLED. So be sure to do a:

    pkg install -r Emulab pssh

on your boss node. See the new utils/pushrootkeys.in script for more.

The public key is distributed via the "tmcc localization" command which
was already designed to handle adding multiple public keys to root's
authorized_keys file on a node.

This approach should be backward compatible with old images. I BUMPED THE
VERSION NUMBER OF TMCD so that newer clients can also get back (via
rc.localize) a list of keys and the names of the files they should be stashed
in. This is used to allow us to pass along the SSL and SSH versions of the
public key so that they can be placed in /root/.ssl/<node>.pub and
/root/.ssh/id_rsa.pub respectively. Note that this step is not necessary for
inter-node ssh to work.

Also passed along is an indication of whether the returned key is encrypted.
This might be used in Round 2 if we securely implant a shared secret on every
node at imaging time and then use that to encrypt the ssh private key such
that we can return it via rc.localize. But the client side script currently
does not implement any decryption, so the client side would need to be changed
again in this future.

The per experiment root keypair mechanism has been exposed to the user via
old school NS experiments right now by adding a node "rootkey" method. To
export the private key to "nodeA" and the public key to "nodeB" do:

    $nodeA rootkey private 1
    $nodeB rootkey public 1

This enables an asymmetric relationship such that "nodeA" can ssh into
"nodeB" as root but not vice-versa. For a symmetric relationship you would do:

    $nodeA rootkey private 1
    $nodeB rootkey private 1
    $nodeA rootkey public 1
    $nodeB rootkey public 1

These user specifications will be overridden by hardwired Emulab restrictions.
The current restrictions are that we do *not* distribute a root pubkey to
tainted nodes (as it opens a path to root on a node where no one should be
root) or any keys to firewall nodes, virtnode hosts, delay nodes, subbosses,
storagehosts, etc. which are not really part of the user topology.

For more on how we got here and what might happen in Round 2, see:

    #302

operating in standalone mode (not part of a federation), which would be
the case for everyone that is not us. Further exercise would be to
automate portal setup when part of a federation. Not a big deal to add,
but lets checkpoint what I have done so far.

camcontrol cannot change the cache settings on "ada" devices.

FreeBSD 8.x smartctl binary does work with 10.x kernel.

See emulab/emulab-devel issue #303. Ensure we have a controlled set of
pubkeys in root's .ssh/authorized_keys file when we create and load new
images. But allow for a user added key to survive node reboots if they
customize it within an experiment.

We want both to wind up in authorized_keys.

safeLibOp blocks all our vnodesetup-related signals from interrupting
libvnode ops to ensure at least op-level consistency.  However, there
was an opportunity for signals to sneak in, in between a successful
vnodeCreate and the writing of the vnode.info file (that mkvnode.pl uses
to know if the vnode was created or not).

So I redid safeLibOp to make blocking signals optional (of course it's
on for nearly all calls, except now vnodeCreate, and formerly
vnodePoll).  Now there's a signal-safe zone all the way around
vnodeCreate, including a StoreState() before we unblock.  This should
ensure consistency in that particular spot.  I didn't think about
whether this affects anything else.

(And fix it up for Docker...)

See clientside/tmcc/linux/docker/README.md for design notes.
See clientside/tmcc/linux/docker/dockerfiles/README.md for a description
of how we automatically Emulabize existing Docker images.

Also, this mostly fits within the existing vnodesetup path, but I did modify
mkvnode.pl to allow the libvnode backend to provide a vnodePoll wait
loop instead of the builtin vnodeState loop.

This allows other callers than rc.hostnames (i.e. docker clientside) to
generate the hostname list for an experiment.

Also remove use of local5.* in syslog.conf so we can turn around
and re-add it!

in blockstore-related VGs.

Right now, you have to decide globally and in advance, what disk types
are going to be included in blockstore pools. Then you set the sitevar
accordingly and then set the DB sysvol/nonsysvol/any node_type_features
to reflect the amount of storage available on just drives of that type.

This value is passed to clients via the otherwise unused PROTO field
of the blockstore line (when CMD=SLICE and CLASS=local), so this change
is backward compatible (OS images with older client code will ignore it
and just give you blockstores including all the devices).

So at Wisconsin, I set storage/local/disktype to "HDD-only" and tweak
the node_type_attributes '?+disk_any' and '?+disk_nonsysvol' to not
include the space for the 1 or 2 SSD drives in each machine. tmcd passes
the PROTO=HDD-only value and the client sees that and does not include
any SSD devices among the eligible devices from which to create the VG.

The hope is that ultimately, we could get rid of the sitevar and use the
PROTO field to select, per-blockstore, its type (only HDD, only SSD).
But that will require additional per node (type) assign features
differentiating the amount of each type available.

I'm talking to you Ubuntu16!

See emulab-devel issue 227 for details.

Also, on a "reset" clean out the correct BDB files. It has been
a long time since they used ".db" as the suffix. Now there are
".pag" and ".dir" files. We haven't noticed because we don't really
use the "reset" operation. The prepare script just removes
everything in /var/emulab/db.

Get rid of ELVIN_COMPAT and CONFIG_OPSVM from elabinelab land.
These options still exist throughout the install code, didn't touch that.

Normally, "pkg" will update itself to the latest version from FreeBSD
Central, but that version is now built with FreeBSD 10.3 and the binary
is incompatible with earlier 10.x versions.

So we go to heroic efforts to install the version from the pre-built
Emulab packages.

There are three pieces here, a change to the frisbee protocol itself, an
Emulab event component to get status back to the portal, and the surrounding
infrastructure to make it all work.

Frisbee heartbeat messages:

Added a new message type to the frisbee protocol, "Progress". In theory it
operates by having the server send a multicast progress request to its clients
which includes an interval at which to report (or "just once") and an
indication of what to report (nothing, progress summary, or full stats). The
client then sends unicast "fire and forget" UDP replies according to that
schedule. However, I took a shortcut for the moment and just added a command
line option to the client to tell it to report a summary at the indicated
interval (-H <interval>).  So the server never sends requests.

This is implemented in the client by a fourth thread since I wanted it to
operate independent of packet reception (which would cause clients to report
in a highly synchronized fashion due to multicast). The server instance just
logs progress reports into its log.

This protocol addition should be fully backward compatible as both client and
server ignore (but log) unknown messages.

Emulab progress report events:

When this is compiled in (-DEMULAB_EVENTS) and turned on (-E <server>), the
frisbee server instances will send a FRISBEEPROGRESS event to the indicated
event server for every progress report it receives (in addition to logging the
events to its own log). Right now it will create an event with key/value pairs
for the information in a client summary reply:

TSTAMP is the client's time at which it sends the event. Could be used by the
received to determine latency of the report if it cared (and if it assumed
that the clocks are in sync). We don't care about this.

SEQUENCE is the report number. Again, could be used by the receiver, in this
case to detect loss, if it cared. We don't.

CHUNKS_RECV is complete chunks that the client has received from the network.
CHUNKS_DECOMP is chunks decompressed by the client.  BYTES_WRITTEN is bytes
written to disk by the client.

Any of the three can be used by the event receiver as an indication of life
and/or progress. However, only the last would be a reasonable indicator of
time remaining since it is the last (and slowest) phase of imaging. To
estimate time remaining we could compare that value to the amount of
uncompressed data that is in the image. This makes the sketchy assumptions
that time for writes to the disk are uniform and that the number and distance
of seeks is uniform, but it is better than a sharp stick in the eye.

Emulab infrastructure:

There is a new sitevar "images/frisbee/heartbeat" which can be set to a
non-zero value to tell the frisbee MFS to fire off frisbee with -H <value>
and thus make reports. The default value of zero means to not make reports.
The tmcd "loadinfo" command sends this through via the HEARTBEAT=<value>
param.

REQUIRED A TMCD VERSION BUMP TO 41.

Support FreeBSD 10.3. We will need to be moving to this before long
as 10.2 EOLs in two days.

Support setup of "Emulab-aware" ZFS use in install scripts. Note that
the core support code was already done (WITHZFS, WITHAMD). Mostly this
involves changes to setup either amd (WITHAMD==1) or autofs (WITHAMD==0)
on the boss node and to NOT add mounts of /{users,groups,proj} to
/etc/fstab. We still need to add a section to the install documentation
about setting up a zpool for Emulab to use. There was also a fix to the
firstuser script which did not do the account setup correctly.

Support setup of ZFS in elabinelab. The elabinelab attributes CONFIG_ZFS
and CONFIG_AUTOFS are used to convey intent here. Currently they can only
be used in an "ops+fs" config (e.g., the standard boss and ops config,
NOT the seperate fs node config). It should work with either the physical
or virtual node setups:

* For the physical node setup, we actually use local blockstores ...

For the case in which mkextrafs is used to create local homedirs/projdirs:

Look for the desired mount point (/local) in /etc/fstab and use that if
it exists (i.e., that FS was already setup by the blockstore system or a
previous mkextrafs).

Otherwise, look for /var/emulab/boot/extrafs which should contain info
left behind by the local blockstore setup code indicating a FS or unused
device to use. For an unused device, rc.storage will identify the largest
available device that is at least 10MB.

from boss libtestbed,pm into clientside version of same file.

The prepare script now supports pre and post hooks.  It runs all hooks
in rc order, from the DYNRUNDIR/prepare.pre.d and BINDIR/prepare.pre.d
dirs (rc order in this case is the BSD order, or my version of it ---
any file prefixed with a number is run in numeric order; other files are
run sorted alphabetically following numeric files).  Post hooks are in
prepare.post.d, and are run at the end of prepare.

(DYNRUNDIR is always /var/run/emulab .  STATICRUNDIR is usually
/etc/emulab/run but could be /etc/testbed/run, depending on the
clientside installation.)

We now allow users to override our default interface configuration --
and if they do, and tell us about it by writing a file in either
$DYNRUNDIR or $STATICRUNDIR named interface-done-$mac , we will not
attempt to configure it, and will assume they have done it!  If they are
nice to us and write
  $iface $ipaddr $mac
into the file, we will parse that and put it into the @ifacemap and
%mac2iface structures in doboot().  We do *not* attempt to provide them
the ifconfig info in env vars or anything; they have to grok our
ifconfig file format, in all its potential glory.

We read the hosts.head file(s) from /etc, DYNRUNDIR, and STATICRUNDIR,
and prepend them to our Emulab hosts content.  Then, we append the
content of the hosts.tail file(s) from /etc, DYNRUNDIR, and STATICDIR
--- and that file becomes the new /etc/hosts file.

getmanifest() has become getrcmanifest() to avoid confusion with the
GENI manifest.  Also, it now supports local manifests embedded in the
filesystem from $DYNRUNDIR and $STATICRUNDIR (priority is manifest from
exp, then DYNRUNDIR, then STATICRUNDIR).  All manifests read and
applied.  Local manifests may also reference local files instead of blob
ids, of course.  It is important to support local manifests so that
experimenters can hook our services by default in the disk image.

The bigest improvement happened on day one when I took out the 20 second sleep
between vnode starts in bootvnodes. That appears to have been an artifact of
an older time and an older Xen. Or, someone smarter than me saw the potential
of getting bogged down for, oh say three weeks, trying to micro-optimize the
process and instead just went for the conservative fix!

Following day one, the ensuing couple of weeks was a long strange trip to
find the maximum number of simultaneous vnode creations that could be done
without failure. In that time I tried a lot of things, generated a lot of
graphs, produced and tweaked a lot of new constants, and in the end, wound
up with the same two magic numbers (3 and 5) that were in the original code!
To distinguish myself, I added a third magic number (1, the loneliest of
them all).

All I can say is that now, the choice of 3 or 5 (or 1), is based on more
solid evidence than before. Previously it was 5 if you had a thin-provisioning
LVM, 3 otherwise. Now it is based more directly on host resources, as
described in a long comment in the code, the important part of which is:

 #
 # if (dom0 physical RAM < 1GB) MAX = 1;
 # if (any swap activity) MAX = 1;
 #
 #    This captures pc3000s/other old machines and overloaded (RAM) machines.
 #
 # if (# physical CPUs <= 2) MAX = 3;
 # if (# physical spindles == 1) MAX = 3;
 # if (dom0 physical RAM <= 2GB) MAX = 3;
 #
 #    This captures d710s, Apt r320, and Cloudlab m510s. We may need to
 #    reconsider the latter since its single drive is an NVMe device.
 #    But first we have to get Xen working with them (UEFI issues)...
 #
 # else MAX = 5;

In my defense, I did fix some bugs and stuff too (and did I mention
the cool graphs?) See comments in the code and gitlab emulab/emulab-devel
issue #148.

Using "set-rwclone" ala:

    set $bsobj [$ns blockstore]
    $bsobj set-lease "emulab-ops/bar"
    $bsobj set-node $node
    $bsobj set-rwclone 1
    ...

in your NS file will create a clone of the indicated persistent blockstore.

Somewhat limited in utility since you can only have one clone of a
particular blockstore per experiment.