GitLab

for mere users.

This commit is intended to makes the license status of Emulab and
ProtoGENI source files more clear.  It replaces license symbols like
"EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
blocks that contain actual license statements.

This change was driven by the fact that today, most people acquire and
track Emulab and ProtoGENI sources via git.

Before the Emulab source code was kept in git, the Flux Research Group
at the University of Utah would roll distributions by making tar
files.  As part of that process, the Flux Group would replace the
license symbols in the source files with actual license statements.

When the Flux Group moved to git, people outside of the group started
to see the source files with the "unexpanded" symbols.  This meant
that people acquired source files without actual license statements in
them.  All the relevant files had Utah *copyright* statements in them,
but without the expanded *license* statements, the licensing status of
the source files was unclear.

This commit is intended to clear up that confusion.

Most Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the Affero GNU General Public License, version 3
(AGPLv3).

Most Utah-copyrighted files related to ProtoGENI are distributed under
the terms of the GENI Public License, which is a BSD-like open-source
license.

Some Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the GNU Lesser General Public License, version 2.1
(LGPL).

web interface.

Most of this commit is about making it easier to create the
descriptors (of any kind) The newimageid_ez page now takes an optional
node_id on which to base the new descriptor; most of the fields are
initialized.

You can also provide optional imageid and osinfo arguments, which
allows for slightly less info to be initialized.

Then on the ShowNode page, the link to create an image will take you
to the snapshot image page if the user has write access to the image,
otherwise to the newimageid_iz page to create and snapshot a new
image descriptor.

We had a couple of different problems actually.

* We allow users to insert html into many DB fields (say, a project or
  experiment description).

* We did not sanitize that output when displaying back.

* We did not sanitize initial page arguments that were reflected in the
  output (say, in a form).

Since no one has the time to analyze every line of code, I took a couple of
shortcuts. The first is that I changed the regex table to not allow any <>
chars to go from the user into the DB. Brutal, but in fact there are only a
couple of places where a user legitimately needs them. For example, a
startup command that includes redirection. I handle those as special
cases. As more come up, we can fix them.

I did a quick pass through all of the forms, and made sure that we run
htmlspecialchars on everything including initial form args. This was not
too bad cause of the way all of the forms are structured, with a
"formfields" array.

I also removed a bunch of obsolete code and added an update script to
actually remove them from the www directory.

Lastly, I purged some XMLRPC code I did a long time ago in the Begin
Experiment path. Less complexity, easier to grok and fix.

	modified:   sql/database-fill.sql
	modified:   sql/dbfill-update.sql

the new/edit/show pages. Only admins get to change it though.

Now the limit is pid=48 and gid=32.

Note that the FreeBSD clients croak on NFS mounts that are too long
(way shorter then MAXPATHLEN!). Sigh.

     www/newimageid_ez.php3 - The reworked PHP page.  Calls Image::NewImageId with ez=1.
     www/newimageid.php3 - Call Image::NewImageId with ez=0.
     www/imageid_defs.php - Re-use the Image::NewImageId class method, adding an 'ez' arg.
     backend/{newimageid_ez,GNUmakefile}.in configure configure.in - New backend script.
         After checks, calls OSinfo->Create and Image->Create with the same XML args array.
     db/Image.pm.in - Re-use the Image->Create method, adding an imageid over-ride arg.
     db/OSinfo.pm.in - Filter out extraneous db slot args from XML in the Create method.
     db/libdb.pm.in - Add TB_{OS,IMAGE}ID_* constants from dbdefs.php3 .
     sql/database-fill.sql - Add OS entries to the table_regex 'images' pattern set.

form processing to be done.

The gist is that I have moved all of the data checking and DB work to
the backend into a new script called utils/newosid. This script does
all the field checking that used to be done in php. It takes a simple
XML file as input and returns a set of strings to format as errors (if
there are any).

The overall goal to make a big push to move this code out of PHP and
perl.  A nice side effect is that many operations that are current
only available via the web interface will also become available
command line (and also XMLRPC with a little moew work).

Currently we declare the valid types for an image to be those that are of
class "pc" and for which we have at least one node in the testbed.

With this change, when an image is created as a snapshot from an existing
node, we look at the partition info for the existing image to further
restrict the types that the derived image will work on.

This will prevent images derived from old images like RHL90 or FBSD410
from getting newer node types such as pc745d as a valid type.

One problem with the current solution is that, we don't know til the form
is submitted once, what node you are snapshot'ing, so the first time a
user submits the form, it is almost certain to come back with an error.

Further, they have no choice but to uncheck the offending types.  If part
of their customization was to add support for a new node type...too bad,
ya' can't add it!  There are numerous ways to fix this, all of which exceed
my Friday afternoon mad hacking skilz.

widearea reloading support.

* New slot in the images table to store an access key which remote
  sites must provide in order to download an image (via https).

* tmcd returns a different kind of ADDRESS field from doloadinfo.
  Instead of the multicast stuff, return a URL that points to boss'
  web server. The URL is of the form:

   https://www.myemulab.net/spewimage.php?imageid=10013&access_key=abcdef

  which as you can see is fully specified; the client does not need
  to know anything else.

* New webpage and backend scripts appropriately called "spewimage"
  which also includes support for the http HEAD request (from wget) to
  avoid downloading images that are already on the node. I just
  learned about this HEAD request stuff today ... but otherwise these
  operate as expected, spewing the image if the access key is provided.

* Changes to rc.frisbee to deal with remote loading. In addition to
  URL support, I also added support for simple paths, the intent being
  that we will probably distribute images offline (say, at night) so
  that when a node reboots it doesn't actually have to wait 60 minutes
  for an image to download. I have not added any server side support
  for this yet though. Maybe later this week.

* Other bits and pieces and fixes to make this work.

improper ussage.

working on this a few weeks ago when I got sidetracked by the pubsub
project.

(also knows an integers).

indexed by exptidx. I also got the last of the pid and pid,gid tables.

  register_globals=1 to turn POST/GET/COOKIES arguments in local variables.
  This is known to be a terrible security risk, and we keep saying we are
  going to fix it, and now I am. In order to accomplish this on a
  transitional basis (since I don't want the entire web interface to stop
  working while I debug it), and because the code just needs the cleanup, I
  am doing it like this: Each page will sport new declarations at the top:

	RequiredPageArguments("experiment", PAGEARG_EXPERIMENT,
                              "template",   PAGEARG_TEMPLATE,
                              "instance",   PAGEARG_INSTANCE,
                              "metadata",   PAGEARG_METADATA,
                              "osinfo",     PAGEARG_OSINFO,
                              "image",      PAGEARG_IMAGE,
                              "project",    PAGEARG_PROJECT,
                              "group",      PAGEARG_GROUP,
                              "user",       PAGEARG_USER,
			      "node",       PAGEARG_NODE,
			      "yesno",      PAGEARG_BOOLEAN,
			      "message",    PAGEARG_STRING,
			      "age",        PAGEARG_INTEGER,
                              "cost",       PAGEARG_NUMERIC,
                              "formfields", PAGEARG_ARRAY,
                              "unknown",    PAGEARG_ANYTHING);

	OptionalPageArguments("canceled", PAGEARG_BOOLEAN);

  The first token in each pair is the name of the global variable to
  set, and the second token is the type. So, for "experiment" we look at
  the URL for a pid/eid or exptidx, etc, sanity check them (safe for a
  DB query), and then try to find that experiment in the DB. If it maps
  to an experiment, set global variable $experiment to the object. Since
  its a required argument, produce an error if not supplied. Similar
  treatment for optional arguments, with the obvious difference.

  The goal is to have ALL argument processing in one place, consistent,
  and correct. I've found numerous places where we leak unchecked
  arguments into queries. It also cuts out a lot of duplicated code.

* To make the above easier to deal with, I've been replacing lots of
  hardcoded URLS in the code of the form:

	foo.php3?pid=$pid&eid=$eid ...

  with

        CreateURL("foo", $experiment)

  which creates and returns the neccessary url string, by looking at
  the type of its arguments (experiment, template, instance, etc.)

  Eventually plan to replace them all so that URL handling throughout
  the code is all defined in one place (all the new URL code is in
  url_defs.php).

* I have cranked up error reporting to tell me anytime a variable is
  used before it is initialized, plus a bunch of other stuff that PHP
  deems improper. Think of it like -Wall ... and boy we get a lot of
  warnings.  A very large percentage of the diffs are to fix all these
  warnings.

  The warnings are currently going to /usr/testbed/log/php-errors.log,
  and I'll be adding a script to capture them each night and mail them
  to tbops. This file also gets errors (this will be a change for
  developers; rather then seeing errors and warnings dumped in the
  middle of web pages, they will go to this file instead).

* Major refactoring of the code. More objects (nodes, images, osids).
  Moving tons of queries into the objects in the hopes of someday
  getting to a point where we can split the web interface onto a
  different server.  Lots of general cleanup.

most of the rest of the tables in the system (still a few exceptions).
Bound to be some bugs ...

converting to locally unique ids and later globally unique ids.

Two-day boondoggle to support "/scratch", an optional large, shared filesystem
for users.  To do this, I needed to find all the instances where /proj is used
and behave accordingly.  The boondoggle part was the decision to gather up all
the hardwired instances of shared directory names ("/proj", "/users", etc.)
so that they are set in a common place (via unexposed configure variables).
This is a boondoggle because:

1. I didn't change the client-side scripts.  They need a different mechanism
   (e.g., tmcd) to get the info, configure is the wrong way.

2. Even if I had done #1 it is likely--no, certain--that something would
   fail if you tried to rename "/proj" to be "/mike".  These names are just
   too ingrained.

3. We may not even use "/scratch" as it turns out.

Note, I also didn't fix any of the .html documentation.  Anyway, it is done.
To maintain my illusion in the future you should:

1. Have perl scripts include "use libtestbed" and use the defined PROJROOT(),
   et.al. functions where possible.  If not possible, make sure they run
   through configure and use @PROJROOT_DIR@, etc.

2. Use the configure method for python, C, php and other languages.

3. There are perl (TBValidUserDir) and php (VALIDUSERPATH) functions which
   you should call to determine if an NS, template parameter, tarball or
   other file are in "an acceptable location."  Use these functions where
   possible.  They know about the optional "scratch" filesystem.  Note that
   the perl function is over-engineered to handles cases that don't occur
   in nature.

table, into a new table called node_type_attributes, which is intended
to be a more extensible way of describing nodes.

The only things left in the node_types table will be type,class and the
various isXXX boolean flags, since we use those in numerous joins all over
the system (ie: when discriminating amongst nodes).

For the most part, all of that other stuff is rarely used, or used in
contexts where the information is needed, but not for type descrimination.
Still, it made for a lot of queries to change!

Along the way I added a NodeType library module that represents the type
info as a perl object. I also beefed up the existing Node module, and
started using it in more places. I also added an Interfaces module, but I
have not done much with that yet.

I have not yet removed all the slots from the node_types table; I plan to
run the new code for a few days and then remove the slots.

Example using the new NodeType object:

	use NodeType;

	my $typeinfo = NodeType->Lookup($type);

        if ($typeinfo->control_interface(\$control_iface) ||
            !$control_iface) {
  	    warn "No control interface for $type is defined in the DB!\n";
        }

or using the Node:

	use Node;

        my $nodeobject = Node->Lookup($node_id);
        my $imageable  = $nodeobject->NodeTypeInfo()->imageable();
or
        my $rebootable = $nodeobject->isrebootable();
or
        $nodeobject->NodeTypeAttribute("control_interface", \$control_iface);

Lots of way to accomplish the same thing, but the main point is that the
Node is able to override the NodeType (if it wants to), which I think is
necessary for flexibly describing one/two of a kind things like switches, etc.

and per OSID basis.

* Added bios_waittime to node_types table and reboot_waittime to
  os_info table. Initialized them as follows:

        update node_types set bios_waittime=60 where class='pc';
        update os_info set reboot_waittime=150 where OS='Linux' or
	  OS='FreeBSD' or OS='NetBSD';
        update os_info set reboot_waittime=180 where OS=Windows';

* The bios waittime can be edited via the web interface.

* The reboot waittime can be set only by admin people right now; this
  is another case of something that maybe the user should not see
  cause its too much stuff? Instead, default values are established in
  www/osiddefs.php3.

* os_setup computes its per-node waitime as:

	(bios_waittime + reboot_waittime) * 2

  as per Mike's suggestion. If either value is not defined in the DB,
  it defaults the original 7 minute value.

the top.

user, can read it.

At Leigh's suggestion, wrap the filenames in escapeshellarg() to make
the call to SUEXEC safer.

'nodetype=mote' parameter to display only the options that matter for
a mote.

This is implemented much like the menu and the beginexp page, with a
$view hash, set from the nodetype argument, that controls
showing/hiding different parts of the page.

Added a file uplaod box to be used for mote images, which calls Tim's
webcopy script to copy the file into place.

Doesn't work quite yet - for some reason, even though the call to
webcopy is succeeding, the destination file doesn't show up. So we
have a little bit of debugging to do still.

Not linked in from anywhere yet. Not sure what the right thing is -
shoud there be a list at the top of this page of all special nodetypes
it supports? Or should we just link it in from the TinyOS tutorial or
the mote type pages?