GitLab

In checknode code for FreeBSD don't check the /dev/ad* device if it is a symlink.
[I think the a error in the test command for -c]

Provide automated setup of an ssh keypair enabling root to login without
a password between nodes. The biggest challenge here is to get the private
key onto nodes in such a way that a non-root user on those nodes cannot
obtain it. Otherwise that user would be able to ssh as root to any node.
This precludes simple distribution of the private key using tmcd/tmcc as
any user can do a tmcc (tmcd authentication is based on the node, not the
user).

This version does a post-imaging "push" of the private key from boss using
ssh. The key is pushed from tbswap after nodes are imaged but before the
event system, and thus any user startup scripts, are started. We actually
use "pssh" (really "pscp") to scale a bit better, so YOU MUST HAVE THE
PSSH PACKAGE INSTALLED. So be sure to do a:

    pkg install -r Emulab pssh

on your boss node. See the new utils/pushrootkeys.in script for more.

The public key is distributed via the "tmcc localization" command which
was already designed to handle adding multiple public keys to root's
authorized_keys file on a node.

This approach should be backward compatible with old images. I BUMPED THE
VERSION NUMBER OF TMCD so that newer clients can also get back (via
rc.localize) a list of keys and the names of the files they should be stashed
in. This is used to allow us to pass along the SSL and SSH versions of the
public key so that they can be placed in /root/.ssl/<node>.pub and
/root/.ssh/id_rsa.pub respectively. Note that this step is not necessary for
inter-node ssh to work.

Also passed along is an indication of whether the returned key is encrypted.
This might be used in Round 2 if we securely implant a shared secret on every
node at imaging time and then use that to encrypt the ssh private key such
that we can return it via rc.localize. But the client side script currently
does not implement any decryption, so the client side would need to be changed
again in this future.

The per experiment root keypair mechanism has been exposed to the user via
old school NS experiments right now by adding a node "rootkey" method. To
export the private key to "nodeA" and the public key to "nodeB" do:

    $nodeA rootkey private 1
    $nodeB rootkey public 1

This enables an asymmetric relationship such that "nodeA" can ssh into
"nodeB" as root but not vice-versa. For a symmetric relationship you would do:

    $nodeA rootkey private 1
    $nodeB rootkey private 1
    $nodeA rootkey public 1
    $nodeB rootkey public 1

These user specifications will be overridden by hardwired Emulab restrictions.
The current restrictions are that we do *not* distribute a root pubkey to
tainted nodes (as it opens a path to root on a node where no one should be
root) or any keys to firewall nodes, virtnode hosts, delay nodes, subbosses,
storagehosts, etc. which are not really part of the user topology.

For more on how we got here and what might happen in Round 2, see:

    #302

operating in standalone mode (not part of a federation), which would be
the case for everyone that is not us. Further exercise would be to
automate portal setup when part of a federation. Not a big deal to add,
but lets checkpoint what I have done so far.

Apparently there are some issues with UFS2 support in Linux.
Fsck mostly fixes incorrect block counts:
    INCORRECT BLOCK COUNT I=1043374 (0 should be 8)
for inodes that get created by Linux (e.g., /etc/ssh host keys,
/var/emulab/boot stuff). Everything seems to be fine after the fsck.

Also: specify "-Zy" when creating LVMs so that old GPTs, superblocks, etc.
don't leak through. LVM seems to be frightfully deterministic in its
allocation strategies to the extent that virtual disks created in previous
experiments have their metadata show up in newer experiment LVMs.
All the things that are changed

Previously, we were still putting the default utah boss key there.

camcontrol cannot change the cache settings on "ada" devices.

Try to catch the occasional corrupt filesystem before we make it worse.

FreeBSD 8.x smartctl binary does work with 10.x kernel.

This will close issue #310.

And a sleep as well, to make sure we see the message before getting rebooted!

See emulab/emulab-devel issue #303. Ensure we have a controlled set of
pubkeys in root's .ssh/authorized_keys file when we create and load new
images. But allow for a user added key to survive node reboots if they
customize it within an experiment.

We want both to wind up in authorized_keys.

We were expecting it to always contain "ttyS0", but it could be
"ttyS1" as well depending on where we last made the image.

The bug was that, even when the console pty device does not change,
we still need to re-open the device since our caller has closed it.

The race is that, on starting a domU, xenstore may briefly report the
wrong device before finally reporting the correct one.

On first call to xenmode, we recover from the race because we catch the
attempt to open the non-existent device, but then when we retry the
xenmode and we get back the same pty name, we would not do the open
and the caller would then do a select/read on a closed fd. That is a
fatal error. Now xenmode will report an error when it trys to reopen
the bad pty and we will just keep calling xenmode until we finally get
the right device.

In theory.

It seems that some systemds (i.e. 219) cannot handle by-uuid unitfile
names (although they are happy to print them out via systemctl
list-units).  So, in one place in the fstab-generator, just use the raw
device naming convention.  systemd doesn't care what we use.  From the
comments:

  Ok, don't use the by-uuid method (dev-disk-by\\x2duuid-${transuuid}.swap).
  It seems to me that the vintage of systemd on Centos7
  (i.e. 219) doesn't correctly process dev-by-uuid filenames nor
  unitnames (even systemctl status <blah>, where <blah> is a
  by-uuid unit name reported by systemctl list-units, does not
  work!).  systemd 229 on Ubuntu seems happy to use the by-uuid
  unitfilename we generated above.

(Perhaps I should have done this in all places in this script, but I
didn't for now :(.  I believe my by-uuid encoding is correct, and I
really don't want to rock a mostly-working boat.  This fix is enough for
all the cases we have, I hope.)

From the comments:

  Work around dhclient-scripts that forcibly set preferred_lft and
  valid_lft.  We cannot override the lease time sent from the server
  with a real infinite value (our best bet would be UINT32_MAX, and that
  sucks), so we intercept dhclient's name for the new lease time it's
  about to feed to the ip command.  Does this suck any less?  We cannot
  ex post facto run `ip addr change ...` just to reset the preferred_lft
  and valid_lft fields to "forever"; that seems to be tightly coupled
  with assigning an address to an interface (and we don't want to re-add
  the address; that is the whole point of dhclient-script!).  (Some
  dhclients also do not process "expire never" in dhclient.conf
  correctly, so this is what we are left with!)

So we can see where things might be getting hung.

building the new files, since this will cause user additions to get
lost, and that is the whole point of this code.

(right now, specifically only for Docker, but could potentially be used
for any crazy container impl where the local filesystem doesn't appear
local to df -l.  From the commit:)

  Some environments do not give us a valid / mount (like Docker); thus,
  df -l does not work.  Thus we must rely on /proc/mounts to tell us if
  a dir is local or not.  Well, ok, /proc/mounts is hard (and potential
  bind mount chains would make it harder).  So instead, we assume that /
  is local, (well, we check to ensure it is not NFS in /proc/mounts),
  and use os_samefs above to ensure that $dir is on the same device as
  /.  If that is true, that is good enough to call it a local dir.

This really seems to be a worse problem in the community edition; hmm.