GitLab

for mere users.

This commit is intended to makes the license status of Emulab and
ProtoGENI source files more clear.  It replaces license symbols like
"EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
blocks that contain actual license statements.

This change was driven by the fact that today, most people acquire and
track Emulab and ProtoGENI sources via git.

Before the Emulab source code was kept in git, the Flux Research Group
at the University of Utah would roll distributions by making tar
files.  As part of that process, the Flux Group would replace the
license symbols in the source files with actual license statements.

When the Flux Group moved to git, people outside of the group started
to see the source files with the "unexpanded" symbols.  This meant
that people acquired source files without actual license statements in
them.  All the relevant files had Utah *copyright* statements in them,
but without the expanded *license* statements, the licensing status of
the source files was unclear.

This commit is intended to clear up that confusion.

Most Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the Affero GNU General Public License, version 3
(AGPLv3).

Most Utah-copyrighted files related to ProtoGENI are distributed under
the terms of the GENI Public License, which is a BSD-like open-source
license.

Some Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the GNU Lesser General Public License, version 2.1
(LGPL).

We had a couple of different problems actually.

* We allow users to insert html into many DB fields (say, a project or
  experiment description).

* We did not sanitize that output when displaying back.

* We did not sanitize initial page arguments that were reflected in the
  output (say, in a form).

Since no one has the time to analyze every line of code, I took a couple of
shortcuts. The first is that I changed the regex table to not allow any <>
chars to go from the user into the DB. Brutal, but in fact there are only a
couple of places where a user legitimately needs them. For example, a
startup command that includes redirection. I handle those as special
cases. As more come up, we can fix them.

I did a quick pass through all of the forms, and made sure that we run
htmlspecialchars on everything including initial form args. This was not
too bad cause of the way all of the forms are structured, with a
"formfields" array.

I also removed a bunch of obsolete code and added an update script to
actually remove them from the www directory.

Lastly, I purged some XMLRPC code I did a long time ago in the Begin
Experiment path. Less complexity, easier to grok and fix.

	modified:   sql/database-fill.sql
	modified:   sql/dbfill-update.sql

The stand-alone desktop client is no longer supported - the GUI applet
is nearly as good (just doesn't allow user to ssh to the nodes), and
actually works.

entered.

         www/beginexp_form.php3 - No initial whitespace in plab Slice Description textarea.
         sql/database-fill.sql - Use default:fulltext for 'experiments','expt_name'
             to allow multi-line Slice Description in plab_ez->beginexp->batchexp.
         www/moduserinfo.php3 - Pass null string for optional usr_addr2 and notes fields.
         backend/moduserinfo.in - Null string attr comes from XML as an undef value.
         db/User.pm.in - Distinguish between undef and null string in ModUserInfo.

thankless job but someone has to do it. I'm expecting to finish by the
time Bush 43 leaves office.

Other stuff leaked in too ...

I did separate out a lot of tbsetup/libArchive into db/Archive, and
whats left in libArchive.pm will eventually move over into the
Template library.

Note that I have dropped archiving of plain experiments; this is not
really owrth it outside the workbench context, and it just wastes
space and makes a lot if stuff painful in the web interface.

  register_globals=1 to turn POST/GET/COOKIES arguments in local variables.
  This is known to be a terrible security risk, and we keep saying we are
  going to fix it, and now I am. In order to accomplish this on a
  transitional basis (since I don't want the entire web interface to stop
  working while I debug it), and because the code just needs the cleanup, I
  am doing it like this: Each page will sport new declarations at the top:

	RequiredPageArguments("experiment", PAGEARG_EXPERIMENT,
                              "template",   PAGEARG_TEMPLATE,
                              "instance",   PAGEARG_INSTANCE,
                              "metadata",   PAGEARG_METADATA,
                              "osinfo",     PAGEARG_OSINFO,
                              "image",      PAGEARG_IMAGE,
                              "project",    PAGEARG_PROJECT,
                              "group",      PAGEARG_GROUP,
                              "user",       PAGEARG_USER,
			      "node",       PAGEARG_NODE,
			      "yesno",      PAGEARG_BOOLEAN,
			      "message",    PAGEARG_STRING,
			      "age",        PAGEARG_INTEGER,
                              "cost",       PAGEARG_NUMERIC,
                              "formfields", PAGEARG_ARRAY,
                              "unknown",    PAGEARG_ANYTHING);

	OptionalPageArguments("canceled", PAGEARG_BOOLEAN);

  The first token in each pair is the name of the global variable to
  set, and the second token is the type. So, for "experiment" we look at
  the URL for a pid/eid or exptidx, etc, sanity check them (safe for a
  DB query), and then try to find that experiment in the DB. If it maps
  to an experiment, set global variable $experiment to the object. Since
  its a required argument, produce an error if not supplied. Similar
  treatment for optional arguments, with the obvious difference.

  The goal is to have ALL argument processing in one place, consistent,
  and correct. I've found numerous places where we leak unchecked
  arguments into queries. It also cuts out a lot of duplicated code.

* To make the above easier to deal with, I've been replacing lots of
  hardcoded URLS in the code of the form:

	foo.php3?pid=$pid&eid=$eid ...

  with

        CreateURL("foo", $experiment)

  which creates and returns the neccessary url string, by looking at
  the type of its arguments (experiment, template, instance, etc.)

  Eventually plan to replace them all so that URL handling throughout
  the code is all defined in one place (all the new URL code is in
  url_defs.php).

* I have cranked up error reporting to tell me anytime a variable is
  used before it is initialized, plus a bunch of other stuff that PHP
  deems improper. Think of it like -Wall ... and boy we get a lot of
  warnings.  A very large percentage of the diffs are to fix all these
  warnings.

  The warnings are currently going to /usr/testbed/log/php-errors.log,
  and I'll be adding a script to capture them each night and mail them
  to tbops. This file also gets errors (this will be a change for
  developers; rather then seeing errors and warnings dumped in the
  middle of web pages, they will go to this file instead).

* Major refactoring of the code. More objects (nodes, images, osids).
  Moving tons of queries into the objects in the hopes of someday
  getting to a point where we can split the web interface onto a
  different server.  Lots of general cleanup.

form).

Two-day boondoggle to support "/scratch", an optional large, shared filesystem
for users.  To do this, I needed to find all the instances where /proj is used
and behave accordingly.  The boondoggle part was the decision to gather up all
the hardwired instances of shared directory names ("/proj", "/users", etc.)
so that they are set in a common place (via unexposed configure variables).
This is a boondoggle because:

1. I didn't change the client-side scripts.  They need a different mechanism
   (e.g., tmcd) to get the info, configure is the wrong way.

2. Even if I had done #1 it is likely--no, certain--that something would
   fail if you tried to rename "/proj" to be "/mike".  These names are just
   too ingrained.

3. We may not even use "/scratch" as it turns out.

Note, I also didn't fix any of the .html documentation.  Anyway, it is done.
To maintain my illusion in the future you should:

1. Have perl scripts include "use libtestbed" and use the defined PROJROOT(),
   et.al. functions where possible.  If not possible, make sure they run
   through configure and use @PROJROOT_DIR@, etc.

2. Use the configure method for python, C, php and other languages.

3. There are perl (TBValidUserDir) and php (VALIDUSERPATH) functions which
   you should call to determine if an NS, template parameter, tarball or
   other file are in "an acceptable location."  Use these functions where
   possible.  They know about the optional "scratch" filesystem.  Note that
   the perl function is over-engineered to handles cases that don't occur
   in nature.

* Add Princeton IP subnet to the allowed list for viewing plabstats.php3
* Mention PlanetLab AUP on EZ creation page.
* Change language a bit RE PlanetLab AUP on the Portal doc page.

Pass experiment description through to Planetlab as the slice description
when creating a slice.  Also mention the Planetlab AUP and importance of
providing an accurate slice description in the documentation.  The ez
interface also briefly mentions the importance of the slice description.

Convert to a CSS layout, hopefully it should make some workbenchy
things easier.

	* www/beginexp_form.php3: Widen the nscheck window a bit so it
	looks nice.

	* www/common-style.css: Clean out some dead definitions.

	* www/GNUmakefile.in, www/csshover.htc: Some magic that makes IE
	support the :hover CSS pseudo-class.

	* www/currentusage.php3: Tweak the header on the usage table.

	* www/linktest.php3: Drop an extraneous </div>.

	* www/menu.php3, www/tutorial/mobilewireless.php3: Use CSS and
	semantic markup instead of tables.

	* www/nscheck.php3: Hide some of the extra stuff.

	* www/style-doc.css: Some styling specific to documentation.

	* www/style.css, www/style-nonmain.css: Update for changes to
	menu.php3 and try to clean out some old stuff.

* Add support for linking to the NS file that will be used, from the
  begin experiment page, when duplicating or branching an experiment.
  Ultimately we want to separate things so that user can first edit
  the NS file and then proceed to branching.

* In discussion we agreed to use the convention that a directory called
  "archive" in experiment directory, will always be saved and restored.
  This has been implemented.

* Add more of the support for branching an experiment (the archive).
  Batchexp takes a couple of new arguments:

	-c pid,eid[:tag]  or
	-c exptidx[:tag]

  The above specifies what and where to duplicate or branch. Simply
  giving pid,eid does not use the archive, but just copies right out
  of the existing experiment directory.

  Adding the -b option says to branch instead of duplicate.

users only.

The revived Plab interface is here!

Lots of updates to the plab backend, including improved plab <-> elab node
id translation and update handling.  Includes support for the current PLC
API, and the new pl_conf node manager interface API.  Several more db library
routines were ported from the perl library to the python one to support the
new code (mostly the node_id tracking stuff).  Fixes to the client side and
also a rootball creation cleanup (binaries removed from the CVS repo).

There are also enhancements to the experiment view page for experiments
including plab nodes: site and widearea hostname are now displayed along
with the other node information.

Note that the way setup timeout for vnodes is calculated has been changed a
bit.  Instead of using a hardwired base timeout, the base timeout is now
based on the reload_waittime database field, which comes from the 'OS'
(e.g., FBSD-JAIL, RHL-PLAB) the vnode runs.

The default max duration for a plab slice created through the plab_ez interface
is set to 1 year, and linktest is currently disabled and hidden through
the ez interface.

There is still work to do, but this checkin brings with it a functional
plab portal!

Exec summary: after this checkin, the infrastructure exists (once enabled)
to create swapout-time "delta" images for all machines in experiments.
There is only a single, cumulative swap image per node (i.e., all diffs
are from the base image, not from the previous swap).

What doesn't yet exist, is the mechanism for reloading the delta at
swapin time.  That is Phase III.

The nitty-gritty:

1. Keep disk image signature files for all nodes in an experiment.

   New fields in the DB to track, for each disk partition, what image the
   partition was loaded from.  This enables us at swapin or os_load time to
   create signature files in /proj/<pid>/exp/<eid>/swapinfo for the current
   contents of a node disk/partition.  All nodes with the same image loaded
   will share (via symlink) the same signature file.  TODO: no longer
   referenced signature files should be removed.

   Signature info is only collected in the swapinfo directory if the
   experiment is set to have disk state saving enabled (see #5 below).
   Info consists of the <vname>.sig file, which is the file created
   by imagehash, and <vname>.part which says what the root disk is
   for the node and whether to look at the whole disk or just a single
   partition when crafting the delta image.

2. Swapout-time hook for creating swapout image.

   If the experiment is marked as allowing disk state saving, tbswap
   will arrange to run and then monitor the create-swapimage command
   on each node.  This script will run the modified version of imagezip
   which uses the signature file to create a delta image.

   The command to run and maximum timeout are specified via sitevars
   (previously checked in).  Note that the tbswap script currently has
   special knowledge of /usr/local/bin/create-swapimage as a swapout
   time script.  If the swap/swapout_command sitevar is set to that,
   Magic Stuff shall occur (i.e. it will monitor the command and make
   periodic reports of progress).  The sitevars are a total hack and
   will disappear at some point.

3. Client-side script for creating swapout image.

   os/create-swapimage, very similar to create-image.  Uses the info
   stashed in /proj/..blahblah../swapinfo to create a delta image.

   XXX fer now hack: the script first looks in /proj/<pid>/bin for an
   imagezip binary to use.  Failing that, it uses the one in the MFS.
   This allows for easier development of the imagezip changes (i.e.,
   don't have to update the MFS every time.

4. Auto creation of signature files for new images.

   The create_image script (the one that runs on boss when creating images
   for users) has been modified to automatically create a signature via
   imagehash.  The .sig file winds up in /usr/testbed/images/sigs or
   in /proj/<pid>/images/sigs.  From there it will be copied at swapin/os_load
   time to the per-expt swapinfo directory for any node that uses the images.

   The process for creating standard system images (aka, "Mike") has not
   yet been modified.  When the image creation/installation procedure
   is formalized into a script, this will be done.

5. Web changes to set/clear saving of disk state at swapout time.

   Add a checkbox to the experiment create page to allow setting "save
   swap state".  Also added to the experiment modify page, but currently
   "if (0)"ed out as it will need some additional support.  The showstuff
   page will show it.

   Taking a page from Leigh's hack book, if EXPOSESTATESAVE in defs.php3
   is set to zero (as it is now), then the checkbox doesn't appear in the
   create experiment page except for STUDLY users.

in progress.

Make the netlab client applet available to locals.

	* configure, configure.in: Add xmlrpcpipe.php3.

	* xmlrpc/emulabserver.py.in: Add missing virtual_tables.  Add
	getareas call to get the list of robot areas.  Add node.getlist
	and node.typeinfo methods for getting information about the nodes.
	Add a "nic" argument to node.available to get the count of
	wireless nodes.  Add "exclude" argument to
	experiment.virtual_topology so we don't have to download the
	massive route table, also delete the 'pid'/'eid' fields for the
	same reason.  Don't return string output of virtual_topology, it's
	huge.  Return some more info in experiment.getlist().

	* www/GNUmakefile.in: Add xmlrpcpipe.php3.

	* www/beginexp_form.php3, www/modifyexp.php3: Add links to the
	client gui.

	* www/netlab-client.jar: The client binary.

users yet; to do that change $EXPOSELINKTEST in defs.php3.in to 1.

Okay, I implemented a primitive Knowledge Base! The current contents are
*all* the existing FAQ entries, which I entered manually. Here are the
details.

* My reason for doing this is that we need something very simple. The wiki
  is too much of a barrier, and its search capabilities are pathetic.

* The search page for the Knowledge Base is:

	https://www.emulab.net/kb-search.php3

  Fairly primitive keyword search. Turns out that mysql 4.0 has a bunch for
  really good text searching functions built in, but we run 3.23 ... so I
  had to roll it myself. So, its a simple keyword (space or comma
  separated) search, no regular expressions.

* Each DB record has a "faq_entry" flag, so creating the current FAQ on the
  fly from the DB is easy. See:

	https://www.emulab.net/kb-faq.php3

* In reddot mode, you can add new KB entries:

	https://www.emulab.net/kb-manage.php3

  The form is fairly obvious but here are details anyway:

    Section Name: Choose an existing title, or make up a new one.
    Title:        The title of the KB (or FAQ) entry.
    Faq Entry:    Check this box if the new entry should show up in the FAQ.
    X Ref Tag:    A token so you can refer to other KB entries by name,
                  instead of by its index. Within the KB entry you would
                  write: <a href=kb-show.php3?xref_tag=sometag>
    Body:         Whatever you like. I took the existing FAQ entries and
                  stuck them with no changes except for the xref_tag
                  mentioned about (since some entries referenced other
                  entries).

* Once you click on sumbit, you will see the entry as it will appear to
  users, along with a submenu to Modify/Delete/Add entries. You can modify
  the current entry from that menu. Mere users do not see this menu, only
  when in reddot mode.

* The intent here is that we can generate new entries really easy, right
  from email if you like (with appropriate <pre> or <xmp> tags around it).

* I have added sql/knowlbase-create.sql and a makefile target to
  generate that file when creating a distribution. I also added a section
  to install/boss-install to insert the entries into the new DB.

* I hooked the search function into the existing Search Documentation link.
  We know search both the Knowledge Base *and* the Documentation on doc
  searches. This probably needs a little more work to get right.

* I changed a lot of faq links to be more consistent and to reference
  the proper xref_tags (#swapping instead of #UTT-34).

threshold.

currently available to only people with stud=1 status in the DB.

* www/tbauth.php3: Add a STUDLY() function to check that bit.

* www/linktest.php3: New page to run linktest on the fly. The level
  defaults to the current level in the experiments table, but you can
  override that via the form on the page.

* www/showexp.php3: Add link to aforementioned page. STUDLY() only.

* www/beginexp_form.php3: Add an option (selection) to set the linktest
  level for create/swapin. Defaults to 0 (no linktest). STUDLY() only.

* www/editexp.php3: Add an option to edit the default linktest level
  for an experiment. STUDLY() only.

* tbsetup/batchexp.in and tbsetup/swapexp.in: Add code to optionally run
  the linktest, sending email if it fails (exists with non-zero status).
  Failure does not affect the swapin.

page.