GitLab

Also add partial support for 11.2 MFS (just kernel right now,
binaries are still 10.3).

hardware address.
Fixed a egrep expression which was preventing the matching of said list.

Docker containers may be (and default to, and in the shared host case,
must be) deprivileged; thus, they cannot mount devices, much less tell
the kernel (via iscsi userspace tools, etc) to make devices.

Therefore, we must setup any storage backing devices (temp LVs, iscsi
attachments) outside the container.  This commit makes that possible for
rc.storage and linux liblocstorage.  Basically, rc.storage now supports
(for the Linux liblocstorage and Docker) the -j vnodeid calling
convention; and if it's being called on behalf of a vnodeid, it uses
per-vnodeid fstab for any mounts, storage.conf for its state; etc.

I modified libvnode_docker to *not* create virtual networks for
remote blockstore links, because those are pinned to /30s, and thus I
have no client blockstore link address to place on a device in the root
context.  However, I (ab)used the existing Docker network setup for the
blockstore links, and that all happens the same as it used to; we just
no longer create the Docker virtual network nor attach the container to
it.

Finally, I modified tmcd dostorageconfig slightly to return
HOSTIP/HOSTMASK for remote blockstores; and now
libsetup::getstorageconfig will use HOSTIP in preference to its own
HOSTID->HOSTIP translation.  I had to do this so that libvnode_docker in
the root context would not have to go through the mess of translating
HOSTID on behalf of a vnode.

as per issue #440.

This is probably true for Xen too, but in some cases, the
vnodesetup early-release hackwaitandexit timeout of 30 seconds
causes a race condition.  Normally, the first node sets up
significant network state, and sometimes flips MAC addresses
around from interface to interface -- OR puts a physical interface
into a bridge, then changes the bridge's MAC address.  There is a
short window of time where both the bridge and the new member
interface share a MAC address -- and if the tmcc ifconfig assembly
process for vnodes following the first vnode resolves
the wrong device's MAC address and uses that to flesh out the
ifconfig info, the vnodesetup will be in a world of hurt (i.e., you
might see an attempt to make a vlan device out of a vlan device).
The chance of this happening is miniscule, but I've seen it.

So, at least for docker for now, we protect the first vnode against
the 30-second timeout in vnodesetup hackwaitandexit, and we wait for the
actual running file to be written, or error.

This is probably applicable to any linux mkvnode.pl path, but I suppose
it would have been another hundred thousand vnode creates before I saw
it again.

(Also, pass vhost node attributes to rootPreConfig.)

pubsubd wasn't restarting, surely because the existing pubsubd was still
running and/or socket state was still live in the kernel even after
putative death.  This took a long time to manifest, and it's not clear
exactly what the problem was, but making sure pubsubd is dead (and is no
longer holding its specific port) is appropriate even if we assume
REUSEADDR is working, and fixes the current problem.  This was only
observable on the pc3000s and c220g2s, as far as I saw.

If $ETCDIR/ltmap-gzip exists on a clientside node, rc.topomap will only
download the gzip'd versions of ltmap/ltpmap into $BOOTDIR, and
linktest.pl will use them.  Those files are TMLTMAPGZ() and TMLTPMAPGZ().

This is important for multi-thousand node exps, where the lt*map files
easily grow to 250MB or more (and are compressible to 25:1 or so!);
saves CoW virtual disk blocks and raw disk space.  And now that commit
67cd8518 means nodetype no longer uses ltpmap, linktest is the only
consumer of lt*map files.

(I had a disk image containing unmodifiable binary software that would
overwrite dhcpcd's sane copy of /etc/resolv.conf, at a nondeterministic
point in time, with something completely bogus.  That screwed up
startcmdstatus reports; this helps out with that case (in combination
with other custom scripting that returns /etc/resolv.conf to sanity).

Note though that we only retry infinitely once runstartup has
successfully gone to the background; up til then, we're limited to about
a minute's worth of retries.  Likewise, we don't retry forever if
runstartup itself experiences an error.  We only retry forever if we
actually have a status to send.

If the TBScriptLock caller provides a debug message, it will be stored
in a file, and other blocked TBScriptLock callers will get (possibly
slightly racy) info about who holds the lock.

Then, use this in libvnode_xen to get some info about long calls to xl
(create|halt|reboot|etc).

Also enable lockdebug in libvnode_xen for now.

A new tmcd command, publicaddrinfo, just dumps the relevant bits of
virt_node_public_addr to any node in an experiment that has addrs
allocated (we don't want to restrict based on calling node_id or
pool_id).

Then the generic getfwconfig() function calls that, and sets some bits.
I also extended this function to add some dynamic clientside vars
(EMULAB_DOMAIN, EMULAB_EXPDOMAIN, EMULAB_PUBLICADDRS) so that user
firewall rule writers can use them to refer to the control net IPs of
nodes in their experiment (i.e., node-0.EMULAB_EXPDOMAIN); and so that
rules can be written over EMULAB_PUBLICADDRS -- a command-delineated
list of IP addrs).

Finally, I extended the Linux firewalling code to allow any experiment
node to answer ARPs for the public IP addresses; we can't know a priori
which node should answer -- and it could change.

This closes #353 .

This means that users can still use our ssh urls to reach containers
that don't run sshd.  We run a private sshd that has Ports to listen
on, and Match blocks containing ForceCommand directives, which run
docker exec $vnode_id <shell>.  User can configure which shell.

In checknode code for FreeBSD don't check the /dev/ad* device if it is a symlink.
[I think the a error in the test command for -c]

Provide automated setup of an ssh keypair enabling root to login without
a password between nodes. The biggest challenge here is to get the private
key onto nodes in such a way that a non-root user on those nodes cannot
obtain it. Otherwise that user would be able to ssh as root to any node.
This precludes simple distribution of the private key using tmcd/tmcc as
any user can do a tmcc (tmcd authentication is based on the node, not the
user).

This version does a post-imaging "push" of the private key from boss using
ssh. The key is pushed from tbswap after nodes are imaged but before the
event system, and thus any user startup scripts, are started. We actually
use "pssh" (really "pscp") to scale a bit better, so YOU MUST HAVE THE
PSSH PACKAGE INSTALLED. So be sure to do a:

    pkg install -r Emulab pssh

on your boss node. See the new utils/pushrootkeys.in script for more.

The public key is distributed via the "tmcc localization" command which
was already designed to handle adding multiple public keys to root's
authorized_keys file on a node.

This approach should be backward compatible with old images. I BUMPED THE
VERSION NUMBER OF TMCD so that newer clients can also get back (via
rc.localize) a list of keys and the names of the files they should be stashed
in. This is used to allow us to pass along the SSL and SSH versions of the
public key so that they can be placed in /root/.ssl/<node>.pub and
/root/.ssh/id_rsa.pub respectively. Note that this step is not necessary for
inter-node ssh to work.

Also passed along is an indication of whether the returned key is encrypted.
This might be used in Round 2 if we securely implant a shared secret on every
node at imaging time and then use that to encrypt the ssh private key such
that we can return it via rc.localize. But the client side script currently
does not implement any decryption, so the client side would need to be changed
again in this future.

The per experiment root keypair mechanism has been exposed to the user via
old school NS experiments right now by adding a node "rootkey" method. To
export the private key to "nodeA" and the public key to "nodeB" do:

    $nodeA rootkey private 1
    $nodeB rootkey public 1

This enables an asymmetric relationship such that "nodeA" can ssh into
"nodeB" as root but not vice-versa. For a symmetric relationship you would do:

    $nodeA rootkey private 1
    $nodeB rootkey private 1
    $nodeA rootkey public 1
    $nodeB rootkey public 1

These user specifications will be overridden by hardwired Emulab restrictions.
The current restrictions are that we do *not* distribute a root pubkey to
tainted nodes (as it opens a path to root on a node where no one should be
root) or any keys to firewall nodes, virtnode hosts, delay nodes, subbosses,
storagehosts, etc. which are not really part of the user topology.

For more on how we got here and what might happen in Round 2, see:

    #302

operating in standalone mode (not part of a federation), which would be
the case for everyone that is not us. Further exercise would be to
automate portal setup when part of a federation. Not a big deal to add,
but lets checkpoint what I have done so far.

camcontrol cannot change the cache settings on "ada" devices.

FreeBSD 8.x smartctl binary does work with 10.x kernel.

See emulab/emulab-devel issue #303. Ensure we have a controlled set of
pubkeys in root's .ssh/authorized_keys file when we create and load new
images. But allow for a user added key to survive node reboots if they
customize it within an experiment.

We want both to wind up in authorized_keys.

safeLibOp blocks all our vnodesetup-related signals from interrupting
libvnode ops to ensure at least op-level consistency.  However, there
was an opportunity for signals to sneak in, in between a successful
vnodeCreate and the writing of the vnode.info file (that mkvnode.pl uses
to know if the vnode was created or not).

So I redid safeLibOp to make blocking signals optional (of course it's
on for nearly all calls, except now vnodeCreate, and formerly
vnodePoll).  Now there's a signal-safe zone all the way around
vnodeCreate, including a StoreState() before we unblock.  This should
ensure consistency in that particular spot.  I didn't think about
whether this affects anything else.

(And fix it up for Docker...)

See clientside/tmcc/linux/docker/README.md for design notes.
See clientside/tmcc/linux/docker/dockerfiles/README.md for a description
of how we automatically Emulabize existing Docker images.

Also, this mostly fits within the existing vnodesetup path, but I did modify
mkvnode.pl to allow the libvnode backend to provide a vnodePoll wait
loop instead of the builtin vnodeState loop.