GitLab

This commit is intended to makes the license status of Emulab and
ProtoGENI source files more clear.  It replaces license symbols like
"EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
blocks that contain actual license statements.

This change was driven by the fact that today, most people acquire and
track Emulab and ProtoGENI sources via git.

Before the Emulab source code was kept in git, the Flux Research Group
at the University of Utah would roll distributions by making tar
files.  As part of that process, the Flux Group would replace the
license symbols in the source files with actual license statements.

When the Flux Group moved to git, people outside of the group started
to see the source files with the "unexpanded" symbols.  This meant
that people acquired source files without actual license statements in
them.  All the relevant files had Utah *copyright* statements in them,
but without the expanded *license* statements, the licensing status of
the source files was unclear.

This commit is intended to clear up that confusion.

Most Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the Affero GNU General Public License, version 3
(AGPLv3).

Most Utah-copyrighted files related to ProtoGENI are distributed under
the terms of the GENI Public License, which is a BSD-like open-source
license.

Some Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the GNU Lesser General Public License, version 2.1
(LGPL).

We had a couple of different problems actually.

* We allow users to insert html into many DB fields (say, a project or
  experiment description).

* We did not sanitize that output when displaying back.

* We did not sanitize initial page arguments that were reflected in the
  output (say, in a form).

Since no one has the time to analyze every line of code, I took a couple of
shortcuts. The first is that I changed the regex table to not allow any <>
chars to go from the user into the DB. Brutal, but in fact there are only a
couple of places where a user legitimately needs them. For example, a
startup command that includes redirection. I handle those as special
cases. As more come up, we can fix them.

I did a quick pass through all of the forms, and made sure that we run
htmlspecialchars on everything including initial form args. This was not
too bad cause of the way all of the forms are structured, with a
"formfields" array.

I also removed a bunch of obsolete code and added an update script to
actually remove them from the www directory.

Lastly, I purged some XMLRPC code I did a long time ago in the Begin
Experiment path. Less complexity, easier to grok and fix.

	modified:   sql/database-fill.sql
	modified:   sql/dbfill-update.sql

and change the way we display widearea nodes a bit.  Also add a generic
typefilter list arg that can limit to multiple types beyond what the
showtype param does.

Finally, there is now a "protogeni" tabular view that might be helpful for
pgeni experimenters...

comment in nodecontrol_list for why.

redirect to the login page rather than printing a message with a link
to the page.  Otherwise send a "403 Forbidden" to keep robots from
indexing the page.  Also send appreciate HTTP responses on other
precheck errors to keep a robot from indexing the page.  In order to
do this the PAGEHEADER call needed to be moved to after
CheckLoginOrDie and Required/OptionalPageArguments on many pages.  A
warning will be printed if either CheckLoginOrDie or
Required/OptionalPageArguments detects that PAGEHEADER was already
called.

Also change the redirect in kb-show to be a permanent redirect (301)
rather than a temporary one (302) which is the default unless a status
code is given.

that type (e.g., pc3000).

Link this capability from shownodetype with a "show details" link down
where all the dots are shown.

form processing to be done.

The gist is that I have moved all of the data checking and DB work to
the backend into a new script called utils/newosid. This script does
all the field checking that used to be done in php. It takes a simple
XML file as input and returns a set of strings to format as errors (if
there are any).

The overall goal to make a big push to move this code out of PHP and
perl.  A nice side effect is that many operations that are current
only available via the web interface will also become available
command line (and also XMLRPC with a little moew work).

processing the node_history table, but that is *way* too slow to do on
the fly (say, from the web interface) cause of the number of records,
so the summary info is stored in the new node_utilization table. I
generate the summary info as new entries are added to node_history (in
SetNodeHistory) but if that becomes too messy, we can just as easily
shift to processing the table on a nightly basis.

Note that I added a new "inception" date field to the nodes table,
which will get set on new nodes, but for existing nodes I have to
derive it from the first entry in the node_history table, or else the
numbers will not make sense.

  register_globals=1 to turn POST/GET/COOKIES arguments in local variables.
  This is known to be a terrible security risk, and we keep saying we are
  going to fix it, and now I am. In order to accomplish this on a
  transitional basis (since I don't want the entire web interface to stop
  working while I debug it), and because the code just needs the cleanup, I
  am doing it like this: Each page will sport new declarations at the top:

	RequiredPageArguments("experiment", PAGEARG_EXPERIMENT,
                              "template",   PAGEARG_TEMPLATE,
                              "instance",   PAGEARG_INSTANCE,
                              "metadata",   PAGEARG_METADATA,
                              "osinfo",     PAGEARG_OSINFO,
                              "image",      PAGEARG_IMAGE,
                              "project",    PAGEARG_PROJECT,
                              "group",      PAGEARG_GROUP,
                              "user",       PAGEARG_USER,
			      "node",       PAGEARG_NODE,
			      "yesno",      PAGEARG_BOOLEAN,
			      "message",    PAGEARG_STRING,
			      "age",        PAGEARG_INTEGER,
                              "cost",       PAGEARG_NUMERIC,
                              "formfields", PAGEARG_ARRAY,
                              "unknown",    PAGEARG_ANYTHING);

	OptionalPageArguments("canceled", PAGEARG_BOOLEAN);

  The first token in each pair is the name of the global variable to
  set, and the second token is the type. So, for "experiment" we look at
  the URL for a pid/eid or exptidx, etc, sanity check them (safe for a
  DB query), and then try to find that experiment in the DB. If it maps
  to an experiment, set global variable $experiment to the object. Since
  its a required argument, produce an error if not supplied. Similar
  treatment for optional arguments, with the obvious difference.

  The goal is to have ALL argument processing in one place, consistent,
  and correct. I've found numerous places where we leak unchecked
  arguments into queries. It also cuts out a lot of duplicated code.

* To make the above easier to deal with, I've been replacing lots of
  hardcoded URLS in the code of the form:

	foo.php3?pid=$pid&eid=$eid ...

  with

        CreateURL("foo", $experiment)

  which creates and returns the neccessary url string, by looking at
  the type of its arguments (experiment, template, instance, etc.)

  Eventually plan to replace them all so that URL handling throughout
  the code is all defined in one place (all the new URL code is in
  url_defs.php).

* I have cranked up error reporting to tell me anytime a variable is
  used before it is initialized, plus a bunch of other stuff that PHP
  deems improper. Think of it like -Wall ... and boy we get a lot of
  warnings.  A very large percentage of the diffs are to fix all these
  warnings.

  The warnings are currently going to /usr/testbed/log/php-errors.log,
  and I'll be adding a script to capture them each night and mail them
  to tbops. This file also gets errors (this will be a change for
  developers; rather then seeing errors and warnings dumped in the
  middle of web pages, they will go to this file instead).

* Major refactoring of the code. More objects (nodes, images, osids).
  Moving tons of queries into the objects in the hopes of someday
  getting to a point where we can split the web interface onto a
  different server.  Lots of general cleanup.

converting to locally unique ids and later globally unique ids.

that filters on 'usrp' for GNU nodes.

allows takes you to new (admin only) page to select a project that
node will be reserved for.

* The node is not actually *reserved*, it is *pre* reserved! The node
  stays in the free pool, and is available only for use in the project
  to which it is reserved.

* The node can already be reserved to some other project when you
  pre-reserve it. It is not until the current owner releases the node
  that the pre-reservation takes effect.

* The node free counts (on the web pages) count a free a node with a
  pre-reservation, as allocated. This way people do not see a free
  count that includes a node they will never be able to get.

Add some checks for 'free' nodes that are not allocatable.

	* db/audit.in: Include the list of nodes that are not reserved but
	have an eventstate that makes them unallocatable.

	* www/dbdefs.php3.in: Add POWEROFF and ALWAYSUP node states.

	* www/nodecontrol_list.php3: Add an asterisk next to the free
	count for type(s) that have free, but unallocatable nodes.

	* www/shownodetype.php3: If a node is free, but unallocatable, put
	a yellow ball next to its name instead of a green one.

some additional privs; allowed to view/edit node logs and histories of
all nodes, plus some others.

with no other stuff on page. Notes:

* I added a new building called MEB-ROBOTS cause otherwise it would
  show up on the wireless maps since floormap gets all the floors for
  a building. Need to add type tags someplace, but not sure where yet

* I committed Russ' initial jpeg image to use in the floorimages
  table. Russ can replace that when he generates a better looking
  version.

* I linked the robots map in from the node status page ...

table).

With apologies to all those non-broadband trogs (use green dot if you want
to download fast) :-)

that I added to the DB. See revision 1.251 of sql/database-create.sql.
I have removed all of the hardwired names of buildings, images, files,
etc. I have also added MEB floors 1 and 2, but they do not show up on
the status page unlesss there are actually nodes assigned to them.
When you go to set a node location, you do see the little thumbs of
all existing buildings/floors, so you can add nodes to floors that do
not currently have any.

Still needs some work to cleanly display multiple buildings, but thats
for another day.

project permission. By default you get a summary of all the free nodes
you can use in all your project memberships, which is not too
useful. It would be more useful to see what types you can use by
project, since that is how permission is granted and how experiments
are created. So, when you are a member of more than one project,
display a list of those projects along the top so you can see the list
on a per-project basis.

More generally, if you are an admin person:

	www.emulab.net/nodecontrol_list.php3?showtype=summary&bypid=tbres

will show out the summary information for a particular project. At
some point I need to make this a subroutine that we can call from
other pages to drop in a summary table.

Remove isadmin() restriction from floormap pages; at some point I
assume we make the floormaps publically available.

Make MEB the default argument to the floormap page until such time
that we need something fancier.

Add a link to the wireless tutorial I am currently throwing up onto
paper.

node_types table, especially when setting up new testbeds. Currently,
linked off the node summary page, when in admin mode you get the edit
link instead of the show link.

ignore that count as free only.

isalive reported from pxeboot kernel when node is free).

for all nodes the user has permission to use (checks the
nodetypeXpid_permissions table).

sorting options).

and in reports.