GitLab

1. Support for PUT.

The big change is support for uploading via the master server, based heavily
on the prototype that Grant did. Currently only host-based (IP-based)
authentication is done as is the case with download. Grant's SSL-based
authentication code is "integrated" but has not even been compiled in.

The PUT protocol allows for assorted gewgaws, like specifying a maximum size,
setting a timeout value, returning size and signature info, etc.

There is a new, awkwardly-named client utility "frisupload" which, like the
download client, takes an "image ID" as an argument and requests to upload
(PUT) that image via the master server. As with download, the image ID can
be either of the form "<pid>/<emulab-image-name>", to upload/update an actual
Emulab image or it can start with a "/" in which case it is considered to
be a pathname on the server.

On the server side, the master server takes PUT requests, verifies permission
to upload the image, fires up a separate instance of an upload daemon (with
the even catchier moniker "frisuploadd"), and returns the unicast addr/port
info to the client which then begins the upload. The master server also acts
as a traffic cop to make sure that downloads and uploads (or uploads and
uploads) don't overlap.

This has been integrated into the Emulab "create image" process in a
backward-compatible way (i.e., so old admin MFSes will continue to work).
Boy, was that fun. One not-so-desirable effect of this integration is that
images now traverse our network twice, once to upload from node to boss and
once for boss to write out the image file across NFS to ops. This is not
really something that should be "fixed" in frisbee, it is only "undesirable"
because we have a crappy NFS server.

What has NOT been done includes: support of hierarchical PUT operations
(we don't need it for either the elabinelab or subboss case), support for
uploading standard images stored on boss (we really want something better
than host-based authentication here), and the aforementioned support of
SSL-based authentication.

2. Other tidbits that got mixed in with PUT support:

Added two new site variables:
    images/frisbee/maxrate_std
    images/frisbee/maxrate_usr
which replace the hardwired (in mfrisbeed and frisbeelauncher before that)
bandwidth limits for image download. mfrisbeed reads these (and the
images/create/* variables) when it starts up or receives a HUP signal.
These could be read from the DB on every GET/PUT, but they really don't change
much and I needed something to test the reread-the-config-on-a-HUP code!

Fixed avoidance of "problematic multicast addresses" so it would actually
work as intended.

Lots of internal "refactoring" to make up for things I did wrong the first
time and to give the general impression that "Wow, Mike did a LOT!"

showstats page way way faster.

Add "Unused" to type enum in the wires table.
Change a few critical wires/interfaces queries to avoid logical=1.

CreateSliver(), to handle multiple accounts.  This somewhat reflects
the Geni AM API for keys, which allows the client to specify multiple
users, each with a set of ssh keys.

The keys argument to the CM now looks like the following (note that
the old format is still accepted and will be for a while).

[{'urn'   => 'urn:blabla'
  'login' => 'dopey',
  'keys'  => [ list of keys like before ]},
 {'login' => "leebee",
  'keys'  => [ list of keys ... ]}];

Key Points:

1. You can supply a urn or a login or both. Typically, it is going to
   be the result of getkeys() at the PG SA, and so it will include
   both.

2. If a login is provided, use that. Otherwise use the id from the urn.

3. No matter what, verify that the token is valid for Emulab an uid
   (standard 8 char unix login that is good on just about any unix
   variant), and transform it if not.

4. For now, getkeys() at the SA will continue to return the old format
   (unless you supply version=2 argument) since we do not want to
   default to a keylist that most CMs will barf on.

5. I have modified the AM code to transform the Geni AM version of the
   "users" argument into the above structure. Bottom line here, is
   that users of the AM interface will not actually need to do
   anything, although now multiple users are actually supported
   instead of ignored.

Still to be done are the changes to the login services structure in
the manifest. We have yet to settle on what these changes will look
like, but since people generally supply valid login ids, you probably
will not need this, since no transformation will take place.

The entries in client_services are the services that Emulab allows users
and admins to apply hooks to, or to enable/disable.

will teach me to believe what I read on the web), there is absolutely
no support at all inside. So, a little more wrapping of things and
some code changes, and presto, dataseek() works. Good that we do not
do this very often.

mostly supports vlans > 1024, but a couple do not so we have to be
careful about te tags we pick.

Still need to deal with modification of a vlan.

At least our schemadiff script assumes that unique keys have names.
mysqldump generates them as well.

per-device community string to override the community string in the
switch_stack_types table.

connections to external network points like ION.

vlans will later use when instantiated. Needed for protogeni.

rather then a specific interface that no testbed is going to have in
their DB.

mode.

bad thing to do, and has recently stopped working. Use an index
ordering slot instead.

No changes in functionally.

This should never have been there in the first place, and I don't want to
carry that hack forward into the post-frisbeelauncher world.

"tb-set-tarfiles" is like "tb-set-node-tarfiles" except that it
distributes the tarfile to all nodes rather than just one and that it
uses frisbee to distribute the file.

These changes involved 1) refactoring frisbee info from images table
into a new table, frisbee_blobs, 2) a new experiment_blobs table, and
3) a new tmcd command so the node knows how to get the files from the
server.

The changes where designed to be general purpose enough to eventually
support:
  1) Distributing arbitrary files (not just tarfiles) to nodes
  2) Perform arbitrary actions on those files
  3) Use arbitrary methods to get the files

As such the tmcd line is as follows:
  URL=* ACTION=*

where URL is currently:
  frisbee.mcast://<ADDR>/<FILE>
for example
  frisbee.mcast://234.16.184.192:18092/users/kevina/home-dir.tar.gz
and when we get around to using a master Frisbee server it could be
  frisbee://*
or it could be a file://, http://, etc.

and ACTION is currently:
  unpack:<LOCATION>
for example
  unpackt:/users
with future syntax to be determined.

how did we not see them?

Cody updated his year old elabinelab and uncovered some problems with the
update process.  Note the updates to the install/updates scripts are just
to add "MASTER_SITE_FREEBSD=1" to the make lines.  This doesn't change
anything functionally, it just speeds up the builds for older FreeBSDs where
some of the MASTER_SITEs no longer exist.