GitLab

This commit is intended to makes the license status of Emulab and
ProtoGENI source files more clear.  It replaces license symbols like
"EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
blocks that contain actual license statements.

This change was driven by the fact that today, most people acquire and
track Emulab and ProtoGENI sources via git.

Before the Emulab source code was kept in git, the Flux Research Group
at the University of Utah would roll distributions by making tar
files.  As part of that process, the Flux Group would replace the
license symbols in the source files with actual license statements.

When the Flux Group moved to git, people outside of the group started
to see the source files with the "unexpanded" symbols.  This meant
that people acquired source files without actual license statements in
them.  All the relevant files had Utah *copyright* statements in them,
but without the expanded *license* state...

We had a couple of different problems actually.

* We allow users to insert html into many DB fields (say, a project or
  experiment description).

* We did not sanitize that output when displaying back.

* We did not sanitize initial page arguments that were reflected in the
  output (say, in a form).

Since no one has the time to analyze every line of code, I took a couple of
shortcuts. The first is that I changed the regex table to not allow any <>
chars to go from the user into the DB. Brutal, but in fact there are only a
couple of places where a user legitimately needs them. For example, a
startup command that includes redirection. I handle those as special
cases. As more come up, we can fix them.

I did a quick pass through all of the forms, and made sure that we run
htmlspecialchars on everything including initial form args. This was not
too bad cause of the way all of the forms are structured, with a
"formfields" array.

I also removed a bunch of obsolete code and added an update script to
actually remove them from the www directory.

Lastly, I purged some XMLRPC code I did a long time ago in the Begin
Experiment path. Less complexity, easier to grok and fix.

	modified:   sql/database-fill.sql
	modified:   sql/dbfill-update.sql

... we don't do it for any other error, and it's best not to inject
user input back into the HTML page; presumably totally bogus strings
could evaluate to 0, and thus look 'too small'

Turns out these were not the cause of a problem I had with PHP5, but they
should be fixed anyway.

the User Name (or id) in various tables.  This also involved adding
the field to the newproject, joinproject, and moduserinfo forms.

Note: also modified 4.149 database-migrate.txt entry to add a note
to add the necessary "slot" to table_regex.

tell caller about it.

  register_globals=1 to turn POST/GET/COOKIES arguments in local variables.
  This is known to be a terrible security risk, and we keep saying we are
  going to fix it, and now I am. In order to accomplish this on a
  transitional basis (since I don't want the entire web interface to stop
  working while I debug it), and because the code just needs the cleanup, I
  am doing it like this: Each page will sport new declarations at the top:

	RequiredPageArguments("experiment", PAGEARG_EXPERIMENT,
                              "template",   PAGEARG_TEMPLATE,
                              "instance",   PAGEARG_INSTANCE,
                              "metadata",   PAGEARG_METADATA,
                              "osinfo",     PAGEARG_OSINFO,
                              "image",      PAGEARG_IMAGE,
                              "project",    PAGEARG_PROJECT,
                              "group",      PAGEARG_GROUP,
                              "user",       PAGEARG_USER,
			      "node",       PAGEARG_NODE,
			      "yesno",      PAGEARG_BOOLEAN,
			      "message",    PAGEARG_STRING,
			      "age",        PAGEARG_INTEGER,
                              "cost",       PAGEARG_NUMERIC,
                              "formfields", PAGEARG_ARRAY,
                              "unknown",    PAGEARG_ANYTHING);

	OptionalPageArguments("canceled", PAGEARG_BOOLEAN);

  The first token in each pair is the name of the global variable to
  set, and the second token is the type. So, for "experiment" we look at
  the URL for a pid/eid or exptidx, etc, sanity check them (safe for a
  DB query), and then try to find that experiment in the DB. If it maps
  to an experiment, set global variable $experiment to the object. Since
  its a required argument, produce an error if not supplied. Similar
  treatment for optional arguments, with the obvious difference.

  The goal is to have ALL argument processing in one place, consistent,
  and correct. I've found numerous places where we leak unchecked
  arguments into queries. It also cuts out a lot of duplicated code.

* To make the above easier to deal with, I've been replacing lots of
  hardcoded URLS in the code of the form:

	foo.php3?pid=$pid&eid=$eid ...

  with

        CreateURL("foo", $experiment)

  which creates and returns the neccessary url string, by looking at
  the type of its arguments (experiment, template, instance, etc.)

  Eventually plan to replace them all so that URL handling throughout
  the code is all defined in one place (all the new URL code is in
  url_defs.php).

* I have cranked up error reporting to tell me anytime a variable is
  used before it is initialized, plus a bunch of other stuff that PHP
  deems improper. Think of it like -Wall ... and boy we get a lot of
  warnings.  A very large percentage of the diffs are to fix all these
  warnings.

  The warnings are currently going to /usr/testbed/log/php-errors.log,
  and I'll be adding a script to capture them each night and mail them
  to tbops. This file also gets errors (this will be a change for
  developers; rather then seeing errors and warnings dumped in the
  middle of web pages, they will go to this file instead).

* Major refactoring of the code. More objects (nodes, images, osids).
  Moving tons of queries into the objects in the hopes of someday
  getting to a point where we can split the web interface onto a
  different server.  Lots of general cleanup.

converting to locally unique ids and later globally unique ids.

of adding copyrights to them.

does his own commit.

in new project names.

Jay has "comments"), but I do not want it hanging around in my source
tree. Here is my mail message:

* The "My Mailing Lists" is context sensitive (copied from Tim's
  changes to the My Bug Databases). It takes you to the *archives* for
  the current project (or subgroup) list. Or it takes you to your
  first joined project.

* The showproject and showgroup pages have direct links to the project
  and group specific archives. If you are in reddot mode, you also
  get a link to the admin page for the list. Note that project and
  group leaders are just plain members of these lists.

* The interface to create a new "user" list is:

	https://www.emulab.net/dev/stoller/newmmlist.php3

  We do not store the password, but just fire it over in the list
  creation process.

  Anyone can create their own mailing lists. They are not associated
  with projects, but just the person creating the list. That person
  is the list administrator and is given permission to access the
  configuration page.

  This page is not hooked in yet; not sure where.

* Once you have your own lists, you user profile page includes a link
  in the sub menu: Show Mailman Lists. From this page you can delete
  lists, zap to the admin page, or change the admin password (which is
  really just a subpage of the admin page).

* As usual, in reddot mode you can mess with anyone else's mailman lists,
  (via the magic of mailman cookies).

* Note on cross machine login. The mailman stuff has a really easy way
  to generate the right kind of cookie to give users access. You can
  generate a cookie to give user access, or to the admin interface for
  a list (a different cookie). Behind the scenes, I ssh over and get
  the cookie, and set it in the user's browser from boss. When the
  browser is redirected over to ops, that cookie goes along and gives
  the user the requested access. No passwords need be sent around,
  since we do the authentication ourselves.

some details can be found in the advanced tutorial that I wrote up.
See this link:

http://www.emulab.net/tutorial/docwrapper.php3?docname=advanced.html#Tracing

The basic idea is that each virt_lan entry gets a couple of new slots
describing the type of tracing that is desired.

  traced tinyint(1) default '0',
  trace_type enum('header','packet','monitor') NOT NULL default 'header',
  trace_expr tinytext,
  trace_snaplen int(11) NOT NULL default '0',
  trace_endnode tinyint(1) NOT NULL default '0',

There is a new physical table called "traces" that is a little bit
like the current delays table. A new tmcd command returns the trace
configuration to the client nodes (tmcd/common/config/rc.trace).

The delays table got a new boolean called "noshaping" that tells the
delay node to bridge, but not set up any pipes. This allows us to
capture traffic at the delay node, but without much less overhead on
the packets.

The pcapper got bloated up to do packet capture and more event stuff.
I also had to add some mutex locking around calls into the pcap
library and around malloc, since the current setup used linuxthreads,
which is not compatable with the standard libc_r library. I was
getting all kinds of memory corruption, and I am sure that if someone
breathes on the pcapper again, it will break in some new way.

for the near future. Two big changes:

* Add WikiOnly accounts. An external user can register for an account on
  the wiki. Rather then use the registration stuff that comes with TWiki,
  redirect to new Emulab web page so we can manage all of the wiki accounts
  from one place. I modified the joinproject page to spit out a subset of
  the required fields so that its simple to get a wiki only account (just a
  few things to fill in).

  In keeping with current security practices, we still generate a
  verification email message to ensure the email address works. However,
  when the user completes the verification, the wiki account is created right
  away, rather then waiting for someone to approve it (since that would
  defeat the entire point of the wiki).

  Aside: I have not thought much about the conversion from a wiki-only
  account to a real account. That is going to happen, and it would be nice
  if that step did not require one of use to go in and hack the DB. Will
  cross that moat later.

  Aside: Rather beat up on the modify user info page too much, I continue
  to spit out the same form, but mark most of the fields as not required,
  and allow wiki-only people to not specify them.

* Both the joinproject and newproject pages sport a new WikiName field so
  that users can select their own WikiName. I added some JavaScript to
  both pages that generate a suitable wikiname from the FullName field, so
  that as soon as the user clicks out of the FullName, a default wikiname is
  inserted in the field.

  Both pages verify the wikinames by checking to make sure it is not
  already in use, and that it meets the WikiRules for WikiTopic names.
  (someone please shoot me if I continue to use WikiNotation).

redoing the entire editimage page.

anchored (force them to be).

node_types table, especially when setting up new testbeds. Currently,
linked off the node summary page, when in admin mode you get the edit
link instead of the show link.

files. Then use it in the newosid page to allow setting of the
nextosid field (by admin people only). This will make it a tiny tiny bit
easier to set up new testbeds.

a multiline text field.

of problem in PHP with the unsigned int value from the regex table
for default:int, which I need to look at, but in the meantime add
proper bounds checks for the numeric fields in the new project table.

zillions of DB fields that we have to set. My solution was to add a
meta table that describes what is a legal value for each table/slot
for which we take from user input. The table looks like this right
now, but is likely to adapt as we get more experience with this
approach (or it might get tossed if it turns out to be a pain in the
ass!).

	CREATE TABLE table_regex (
	  table_name varchar(64) NOT NULL default '',
	  column_name varchar(64) NOT NULL default '',
	  column_type enum('text','int','float') default NULL,
	  check_type enum('regex','function','redirect') default NULL,
	  check tinytext NOT NULL,
	  min int(11) NOT NULL default '0',
	  max int(11) NOT NULL default '0',
	  comment tinytext,
	  UNIQUE KEY table_name (table_name,column_name)
	) TYPE=MyISAM;

Entries in this table look like this:

	('virt_nodes','vname','text','regex','^[-\\w]+$',1,32,NULL);

Which says that the vname slot of the virt_nodes table (which we trust the
user to give us in some form) is a text field to be checked with the given
regex (perlre of course), and that the min/max length of the text field is
1 and 32 chars respectively.

Now, you wouldn't want to write the same regex over and over, and since we
use the same fields in many tables (like pid, eid, vname, etc) there is an
option to redirect to another entry (recursively). So, for "PID" I do this:

        ('eventlist','pid','text','redirect','projects:pid',0,0,NULL);

which redirects to:

	('projects','pid','text','regex','^[a-zA-Z][-\\w]+$',2,12,NULL);

And, for many fields you just want to describe generically what could go
into it. For that I have defined some default fields. For example, a user
description:

        ('experiment,'usr_name','text','redirect','default:tinytext',0,0,NULL);

which redirects to:

	('default','tinytext','text','regex','^[\\040-\\176]*$',0,256,NULL);

and this says that a tinytext (in our little corner of the database
universe) field can have printable characters (but not a newline), and
since its a tinytext field, its maxlen is 256 chars.

You also have integer fields, but these are little more irksome in the
details.

	('default','tinyint,'int,'regex','^[\\d]+$',-128,127,NULL);

and you would use this anyplace you do not care about the min/max values
being something specific in the tinyint range. The range for a float is of
course stated as an integer, and thats kinda bogus, but we do not have many
floats, and they generally do not take on specific values anyway.

A note about the min/max fields and redirecting. If the initial entry has
non-zero min/max fields, those are the min mac fields used. Otherwise they
come from the default. So for example, you can do this:

    ('experiments','mem_usage','int','redirect','default:tinyint',0,5,NULL);

So, you can redirect to the standard "tinyint" regular expression, but you
still get to define min/max for the specific field.

Isn't this is really neat and really obtuse too? Sure, you can say it.

Anyway, xmlconvert now sends all of its input through these checks (its
all wrapped up in library calls), and if a slot does not have an entry, it
throws an error so that we are forced to define entries for new slots as we
add them.

In the web page, I have changed all of the public pages (login, join
project, new project, and a couple of others) to also use these checks.
As with the perl code, its all wrapped up in a library. Lots more code
needs to be changed of course, but this is a start.