A bit overdue, but here they are.

* Disallow image creation for any taint state on node/image
* Disallow console access for "blackbox" and "useronly" states
* Disallow node_admin for "blackbox" and "useronly" states

TB Admins are exempt from these restrictions.

Change to use the realpath function in the 'Cwd' module instead of
calling realpath via the shell.  The shell command varies in its
reaction to a missing final path component.  On some platforms (Linux,
FBSD10+) realpath reports an error if the final component doesn't exist
on the filesystem.  On others (FBSD < 10), it does not report an error.

The perl function from 'Cwd' emulates the same behavior as FBSD prior to
version 10, which is the behavior the scripts expect.

From here on out, instead of using `realpath`, do the following:

use Cwd qw(realpath);
..
..
my $realpath = realpath($somepath);

VM. Note that the target script uses sudo, but not in all the places
it needs to, so lets just run the script as root, and yank the sudo
out later.

descriptor updated with the progress of the image capture.

This should be run whenever an image is created or updated and possibly
periodically over existing images. It makes sure that various image
metadata fields are up to date:

 * hash: the SHA1 hash of the image. This field has been around for
   awhile and was previously maintained by "imagehash".

 * size: the size of the image file.

 * range: the sector range covered by the uncompressed image data.

 * mtime: modification time of the image. This is the "updated"
   datetime field in the DB. Its intent was always to track the update
   time of the image, but it wasn't always exact (create-image would
   update this with the current time at the start of the image capture
   process).

Documentation? Umm...the usage message is comprehensive!
It sports a variety of useful options, but the basics are:

 * imagevalidate -p <image> ...
    Print current DB metadata for indicated images. <image> can either
    be a <pid>/<imagename> string or the numeric imageid.

 * imagevalidate <image> ...
    Check the mtime, size, hash, and image range of the image file and
    compare them to the values in the DB. Whine for ones which are out
    of date.

 * imagevalidate -u <image> ...
    Compare and then update DB metadata fields that are out of date.

Fixed a variety of scripts that either used imagehash or computed the
SHA1 hash directly to now use imagevalidate.

Prior to this commit, XEN guests disks were single partition, no MBR,
the bits dumped into the lvm. This makes a snapshot of a XEN node,
look completely different then a physical disk image, especially if
users want more disk space (mkextrafs) inside the guest, and then want
to take a snapshot of that, and then run it on a physical node (which
was not possible).

With these changes, guests now use the same MBR layout as our version
two MBR, which makes them interchangeable with physical disk images.
In fact, the goal is to be able to switch back and forth as needed,
based on physical resource availability.

a single slice image, since we can now pull the kernel (ramdisk) out
from the guest filesystem (using pygrub for linux, or just mounting
BSD filesystems). This is a lot faster and easier to deal with. I
added an option to the newimage page so that people can set this, but
in general we need a better way to guess that we need it. Always set
for EC2 images.

To create a new descriptor that will be an import from EC2 (and thus
run under XEN), add ?ec2=1 to newimage_ez.php3. Eventually will link
it in someplace. The form will create a XEN based VM, but instead of
node to snapshot from, provide user@host for the EC2 instance.

On the image snapshot page, instead of node use user@host for the EC2
instance.

The backend script (create_image) will call over to ops and invoke
Srikanth's code. I have called that script ec2import-image.pl. See
create_image for how arguments are passed to the script.

and do not send it from create_image

when created via the ProtoGeni interface; we run everything as
GeniUser and so lose that info. We want to know this for history
information and so we can manage permissions on image deletion
effectively.

Also add updater, updater_idx, and updater_urn to track last change
to the image.

CAVEAT: For system images, we do not actually overwrite the image, but
store it out in /proj/$pid/images and then sometime later maybe we
copy it back to boss. This wrecks the tracking info quite effectively
since the image may never be copied back. Needs more thought.

This commit is intended to makes the license status of Emulab and
ProtoGENI source files more clear.  It replaces license symbols like
"EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
blocks that contain actual license statements.

This change was driven by the fact that today, most people acquire and
track Emulab and ProtoGENI sources via git.

Before the Emulab source code was kept in git, the Flux Research Group
at the University of Utah would roll distributions by making tar
files.  As part of that process, the Flux Group would replace the
license symbols in the source files with actual license statements.

When the Flux Group moved to git, people outside of the group started
to see the source files with the "unexpanded" symbols.  This meant
that people acquired source files without actual license statements in
them.  All the relevant files had Utah *copyright* statements in them,
but without the expanded *license* statements, the licensing status of
the source files was unclear.

This commit is intended to clear up that confusion.

Most Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the Affero GNU General Public License, version 3
(AGPLv3).

Most Utah-copyrighted files related to ProtoGENI are distributed under
the terms of the GENI Public License, which is a BSD-like open-source
license.

Some Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the GNU Lesser General Public License, version 2.1
(LGPL).

shared nodes (where the user does not have an account and so
must ssh in as root).

web interface.

Most of this commit is about making it easier to create the
descriptors (of any kind) The newimageid_ez page now takes an optional
node_id on which to base the new descriptor; most of the fields are
initialized.

You can also provide optional imageid and osinfo arguments, which
allows for slightly less info to be initialized.

Then on the ShowNode page, the link to create an image will take you
to the snapshot image page if the user has write access to the image,
otherwise to the newimageid_iz page to create and snapshot a new
image descriptor.

clone_image is a wrapper around newimageid_ez and create_image, that
simplifies the most common operation; creating a new imageid derived
from the image/os that is currently running in the node, and then
taking a snapshot of the node. So for example, if node pcXXX is
running image FREEBSD, and you want to create a custom image from that
node, all you need to do:

	boss> clone_image myfreebsd pcXXX

which will create the new descriptor, deriving everything from the
FREEBSD image on the node, and then take a snapshot from pcXXX. If
the descriptor already exists, just take the snapshot.

So what if you do:

	boss> clone_image FREEBSD pcXXX

well, the image is always looked up in the project the node is
currently attached to, so in fact a new descriptor is created in that
project, and you do not actually overwrite an image from some other
project. 

I've added some locking to images to prevent concurrent snapshots.
This seemed like a good idea since this script is going to be used
from the ProtoGeni interface. More on this in another commit.

Supporting the uploader through the cnet firewall right now would require
opening up every possible unprivileged TCP port. Don't want to go there.

This reverts commit fc89eb38.

Checked in a bunch of crap that was unrelated.

When downloading an image, start the frisbeed process with the minimum set of
gids necessary to access the image. This includes the unix gid of the
project that the image is in and, optionally, the unix gid of the project
subgroup if the image is part of one.

Previously, we just use the gid set of the uid of the swapper of the
experiment. Not only was this excessive, but it might also not include the
gids needed in the case of a "global" image that is not in the world-readable
/usr/testbed/images directory.

frisuploadd will move <foo>.tmp to <foo> when it completes. Since create_image
monitors the size of the former, it might get an undefined value if it checks
the size right after it have been renamed. Compensate.

Recall that exit codes are 8-bits and in the second byte.
I've only known about this for...oh, 30 years now.

Backward compat fall back when no frisbee upload...FAIL.
Upgrade MFS so no backward compat needed...FAIL.

1. Support for PUT.

The big change is support for uploading via the master server, based heavily
on the prototype that Grant did. Currently only host-based (IP-based)
authentication is done as is the case with download. Grant's SSL-based
authentication code is "integrated" but has not even been compiled in.

The PUT protocol allows for assorted gewgaws, like specifying a maximum size,
setting a timeout value, returning size and signature info, etc.

There is a new, awkwardly-named client utility "frisupload" which, like the
download client, takes an "image ID" as an argument and requests to upload
(PUT) that image via the master server. As with download, the image ID can
be either of the form "<pid>/<emulab-image-name>", to upload/update an actual
Emulab image or it can start with a "/" in which case it is considered to
be a pathname on the server.

On the server side, the master server takes PUT requests, verifies permission
to upload the image, fires up a separate instance of an upload daemon (with
the even catchier moniker "frisuploadd"), and returns the unicast addr/port
info to the client which then begins the upload. The master server also acts
as a traffic cop to make sure that downloads and uploads (or uploads and
uploads) don't overlap.

This has been integrated into the Emulab "create image" process in a
backward-compatible way (i.e., so old admin MFSes will continue to work).
Boy, was that fun. One not-so-desirable effect of this integration is that
images now traverse our network twice, once to upload from node to boss and
once for boss to write out the image file across NFS to ops. This is not
really something that should be "fixed" in frisbee, it is only "undesirable"
because we have a crappy NFS server.

What has NOT been done includes: support of hierarchical PUT operations
(we don't need it for either the elabinelab or subboss case), support for
uploading standard images stored on boss (we really want something better
than host-based authentication here), and the aforementioned support of
SSL-based authentication.

2. Other tidbits that got mixed in with PUT support:

Added two new site variables:
    images/frisbee/maxrate_std
    images/frisbee/maxrate_usr
which replace the hardwired (in mfrisbeed and frisbeelauncher before that)
bandwidth limits for image download. mfrisbeed reads these (and the
images/create/* variables) when it starts up or receives a HUP signal.
These could be read from the DB on every GET/PUT, but they really don't change
much and I needed something to test the reread-the-config-on-a-HUP code!

Fixed avoidance of "problematic multicast addresses" so it would actually
work as intended.

Lots of internal "refactoring" to make up for things I did wrong the first
time and to give the general impression that "Wow, Mike did a LOT!"

No more frisbeelauncher or assorted subboss frisbee stuff.

(eventual) support for NFS servers without race conditions!

This means no NFS between nodes and ops/fs. There are still NFS mounts of
ops on boss however.

Added new defs-* variable NOSHAREDFS, which when set non-zero will disable
the export of NFS filesystems to nodes.  Involved lots of little changes:

 * /users, /proj, and /share filesystems are not exported to nodes.

 * Returned mount info now includes an FSTYPE key which will be set to "LOCAL"
   if NOSHAREDFS is in effect (by default it is set to "NFS-RACY"; more on
   this later).  In the case where it is set to LOCAL, the other mount lines
   no longer contain REMOTE=foo settings.  Because of this change,
   THE TMCD VERSION NUMBER HAS BEEN BUMPED TO 32.

 * The client rc.mounts script will now create local versions of /users/*,
   /proj/<pid>, and /share when FSTYPE=LOCAL.  It first runs mkextrafs to
   create a large partition for these, since someday we will likely want
  ...

The commit got away from me.  I was just going to check in a bug fix but got
a lot more.  The "-s" option of create_image will use ssh to collect an image
rather than having the node copy it across NFS.  This is part of the grand
unNFS-ification option which will get checked in tomorrow.  But this part
should stand alone as of this commit.

These variables were previously constants hardwired into create_image:

images/create/maxwait:
    Max time (minutes) to allow for saving an image
images/create/idlewait:
    Max time (minutes) to allow between periods of progress
    (image file getting larger) when saving an image (should be <= maxwait)
images/create/maxsize:
    Max size (GB) of a created image

Max image size should really be in DB as a per-testbed (per-project? per-user?
per-something!) value, and not hardwired here.  In the meantime, we set this
big and let disk quotas do the dirty work of limiting size.

instead of paths. There is a script called from database-migrate that
converts paths to logfile (db) objects.

is not a directory.

Increase max wait time for images based on 6GB at 1.6MB/sec (72 minutes)
Fix a couple of print statements.