currently available to only people with stud=1 status in the DB.

* www/tbauth.php3: Add a STUDLY() function to check that bit.

* www/linktest.php3: New page to run linktest on the fly. The level
  defaults to the current level in the experiments table, but you can
  override that via the form on the page.

* www/showexp.php3: Add link to aforementioned page. STUDLY() only.

* www/beginexp_form.php3: Add an option (selection) to set the linktest
  level for create/swapin. Defaults to 0 (no linktest). STUDLY() only.

* www/editexp.php3: Add an option to edit the default linktest level
  for an experiment. STUDLY() only.

* tbsetup/batchexp.in and tbsetup/swapexp.in: Add code to optionally run
  the linktest, sending email if it fails (exists with non-zero status).
  Failure does not affect the swapin.

Change ptopgen to look at the eventstate of a node; a node is not considered
free unless it is ISUP or PXEWAIT.

Add TBAvailablePCs() to libdb, removing the corresponding code from
assign_wrapper. This routine does the equiv of ptopgen, returning the
number of PCs that are available for use (looking at eventstate).

Change TBFreePCs in the web interface accordingly.

The above changes correspond to an upcoming change in stated.

error - it's too hard to destingish a string from 0 in php, and every
place we call this function, we've already made sure the group exists
anyway.

account, and direct user to Forgot Username/Password page.

Changed the Forgot page to allow user to *either* reset password, or
just find out their username if that is what they forgot.

zillions of DB fields that we have to set. My solution was to add a
meta table that describes what is a legal value for each table/slot
for which we take from user input. The table looks like this right
now, but is likely to adapt as we get more experience with this
approach (or it might get tossed if it turns out to be a pain in the
ass!).

	CREATE TABLE table_regex (
	  table_name varchar(64) NOT NULL default '',
	  column_name varchar(64) NOT NULL default '',
	  column_type enum('text','int','float') default NULL,
	  check_type enum('regex','function','redirect') default NULL,
	  check tinytext NOT NULL,
	  min int(11) NOT NULL default '0',
	  max int(11) NOT NULL default '0',
	  comment tinytext,
	  UNIQUE KEY table_name (table_name,column_name)
	) TYPE=MyISAM;

Entries in this table look like this:

	('virt_nodes','vname','text','regex','^[-\\w]+$',1,32,NULL);

Which says that the vname slot of the virt_nodes table (which we trust the
user to give us in some form) is a text field to be checked with the given
regex (perlre of course), and that the min/max length of the text field is
1 and 32 chars respectively.

Now, you wouldn't want to write the same regex over and over, and since we
use the same fields in many tables (like pid, eid, vname, etc) there is an
option to redirect to another entry (recursively). So, for "PID" I do this:

        ('eventlist','pid','text','redirect','projects:pid',0,0,NULL);

which redirects to:

	('projects','pid','text','regex','^[a-zA-Z][-\\w]+$',2,12,NULL);

And, for many fields you just want to describe generically what could go
into it. For that I have defined some default fields. For example, a user
description:

        ('experiment,'usr_name','text','redirect','default:tinytext',0,0,NULL);

which redirects to:

	('default','tinytext','text','regex','^[\\040-\\176]*$',0,256,NULL);

and this says that a tinytext (in our little corner of the database
universe) field can have printable characters (but not a newline), and
since its a tinytext field, its maxlen is 256 chars.

You also have integer fields, but these are little more irksome in the
details.

	('default','tinyint,'int,'regex','^[\\d]+$',-128,127,NULL);

and you would use this anyplace you do not care about the min/max values
being something specific in the tinyint range. The range for a float is of
course stated as an integer, and thats kinda bogus, but we do not have many
floats, and they generally do not take on specific values anyway.

A note about the min/max fields and redirecting. If the initial entry has
non-zero min/max fields, those are the min mac fields used. Otherwise they
come from the default. So for example, you can do this:

    ('experiments','mem_usage','int','redirect','default:tinyint',0,5,NULL);

So, you can redirect to the standard "tinyint" regular expression, but you
still get to define min/max for the specific field.

Isn't this is really neat and really obtuse too? Sure, you can say it.

Anyway, xmlconvert now sends all of its input through these checks (its
all wrapped up in library calls), and if a slot does not have an entry, it
throws an error so that we are forced to define entries for new slots as we
add them.

In the web page, I have changed all of the public pages (login, join
project, new project, and a couple of others) to also use these checks.
As with the perl code, its all wrapped up in a library. Lots more code
needs to be changed of course, but this is a start.

devel tree.

select the default user interface for a project. The choice is current
'emulab or 'plab', defaults to 'emulab'. New users that join emulab
get the default user interface from the first project they join.

Also generalize the plab_user bit as new "user_interface" slot of the
users table, which is an enum of interface tokens, currently either
'emulab' or 'plab', defaults to 'emulab'. The plab_user bit will be
removed later.

planetlab nodes.

of the CHECKEMAIL function as a result.

state machine (state). All of the stuff that was previously handled by
using batchstate is now embedded into the one state machine. Of
course, these mostly overlapped, so its not that much of a change,
except that we also redid the machine, adding more states (for
example, modify phases are now explicit. To get a picture of the
actual state machine, on boss:

		stategraph -o newstates EXPTSTATE
		gv newstates.ps

Things to note:

* The "batchstate" slot of the experiments table is now used solely to
  provide a lock for batch daemon. A secondary change will be to
  change the slot name to something more appropriate, but it can
  happen anytime after this new stuff is installed.

* I have left expt_locked for now, but another later change will be to remove
  expt_locked, and change it to active_busy or some such new state name in
  the state machine. I have removed most uses of expt_locked, except those
  that were necessary until there is a new state to replace it.

* These new changes are an implementation of the new state machine,
  but I have not done anything fancy. Most of the code is the same as
  it was before.

* I suspect that there are races with the batch daemon now, but they
  are going to be rare, and the end result is probably that a
  cancelation is delayed a little bit.

for key upload). Add more regexes and addslashes().

* Use superglobals for page/form arguments.

* Add regex functions for email and phone number.

* Remove stripslashes calls; not needed and actually incorrect for
  data returned from the DB.

* Add TBvalid_uid() function to regex uid's. To be used throughout the
  system. Eventually add routines for checking other things like pids
  and eids, etc.

* Regex the uid value we get from the cookie, and switch to $_COOKIE
  superglobal.

* Strict regex checking in DOLOGIN() of uid.

* Change login.php to use superglobals, and general tightening of
  parameter checking.

* install-rpm, install-tarfile, spewrpmtar.php3, spewrpmtar.in: Pumped
  up even more! The db file we store in /var/db now records both the
  timestamp (of the file, or if remote the install time) and the MD5
  of the file that was installed. Locally, we can get this info when
  accessing the file via NFS (copymode on or off). Remote, we use wget
  to get the file, and so pass the timestamp along in the URL request,
  and let spewrpmtar.in determine if the file has changed. If the
  timestamp it gets is >= to the timestamp of the file, an error code
  of 304 (Not Modifed) is returned. Otherwise the file is returned.

  If the timestamps are different (remote, server sends back an actual
  file), the MD5 of the file is compared against the value stored. If
  they are equal, update the timestamp in the db file to avoid
  repeated MD5s (or server downloads) in the future. If the MD5 is
  different, then reinstall the tarball or rpm, and update the db file
  with the new timestamp and MD5. Presto, we have auto update capability!

  Caveat: I pass along the old MD5 in the URL, but it is currently
  ignored. I do not know if doing the MD5 on the server is a good
  idea, but obviously it is easy to add later. At the moment it
  happens on the node, which means wasted bandwidth when the timestamp
  has changed, but the file has not (probably not something that will
  happen in typical usage).

  Caveat: The timestamp used on remote nodes is the time the tarfile
  is installed (GM time of course). We could arrange to return the
  timestamp of the local file back to the node, but that would mean
  complicating the protocol (or using an http header) and I was not in
  the mood for that. In typical usage, I do not think that people will
  be changing tarfiles and rpms so rapidly that this will make a
  difference, but if it does, we can change it.

* node_update.in, client side watchdog, and various web pages:
  Deflated node_update, removing all of the older ssh code. We now
  assume that all nodes will auto update on a periodic basis, via the
  watchdog that runs on all client nodes, including plab nodes.

  Changed the permission check to look for new UPDATE permission (used
  to be UPDATEACCOUNT). As before, it requires local_root or better.
  The reason for this is that node_update now implies more than just
  updating the accounts/mounts. The web pages have been changed to
  explain that in addition to mounts/accounts, rpms and tarfiles will
  also be updated. At the moment, this is still tied to a single
  variable (update_accounts) in the nodes table, but as Kirk requested
  at the meeting, it will probably be nice to split these out in the
  future.

  Added the ability to node_update a single node in an experiment (in
  addition to all nodes option on the showexp page). This has been
  added to the shownode webpage menu options.

  Changed locking code to use the newer wrapper states, and to move
  the experiment to RUNNING_LOCKED until the update completes. This is
  to prevent mayhem in the rest of the system (which could be dealt
  with, but is not worth the trouble; people have to wait until their
  initiated update is complete, before they can swap out the
  experiment).

  Added "short" mode to shownode routine, equiv to the recently added
  short mode for showexp. I use this on the confirmation page for
  updating a single node, giving the user a couple of pertinent (feel
  good) facts before they comfirm.

plus a lock field. The lock field was a simple "experiment locked, go away"
slot that is easy to use when you do not care about the actual state that
an experiment is in, just that it is in "transition" and should not be
messed with.

The other two state variables are "state" and "batchstate". The former
(state) is the original variable that Chris added, and was used by the tb*
scripts to make sure that the experiment was in the state each particular
script wanted them to be in. But over time (and with the addition of so
much wrapper goo around them), "state" has leaked out all over the place to
determine what operations on an experiment are allowed, and if/when it
should be displayed in various web pages. There are a set of transition
states in addition to the usual "active", "swapped", etc like "swapping"
that make testing state a pain in the butt.

I added the other state variable ("batchstate") when I did the batch
system, obviously! It was intended as a wrapper state to control access to
the batch queue, and to prevent batch experiments from being messed with
except when it was really okay (for example, its okay to terminate a
swapped out batch experiment, but not a swapped in batch experiment since
that would confuse the batch daemon). There are fewer of these states, plus
one additional state for "modifying" experiments.

So what I have done is change the system to use "batchstate" for all
experiments to control entry into the swap system, from the web interface,
from the command line, and from the batch daemon. The other state variable
still exists, and will be brutally pushed back under the surface until its
just a vague memory, used only by the original tb* scripts. This will
happen over time, and the "batchstate" variable will be renamed once I am
convinced that this was the right thing to do and that my changes actually
work as intended.

Only people who have bothered to read this far will know that I also added
the ability to cancel experiment swapin in progress. For that I am using
the "canceled" flag (ah, this one was named properly from the start!), and
I test that at various times in assign_wrapper and tbswap. A minor downside
right now is that a canceled swapin looks too much like a failed swapin,
and so tbops gets email about it. I'll fix that at some point (sometime
after the boss complains).

I also cleaned up various bits of code, replacing direct calls to exec
with calls to the recently improved SUEXEC interface. This removes
some cruft from each script that calls an external script.

Cleaned up modifyexp.ph3 quite a bit, reformatting and indenting.
Also fixed to not run the parser directly! This was very wrong; should
call nscheck instead. Changed to use "nobody" group instead of group
flux (made the same change in nscheck).

There is a script in the sql directory called newstates.pl. It needs
to be run to initialize the batchstate slot of the experiments table
for all existing experiments.

tmcd (which is bad, since tying up the tmcd threads blocks all nodes
in the testbed). The old functionality is left in tmcd for now.

On the server side, a new web page (www/spewrpmtar.php3) receives a
request for a file, along with the nodeid (pcXXX) making the request,
and the secret key that is generated for each new experiment and
transfered to the node via tmcd. If the key matches, the operation is
handed off to tbsetup/spewrpmtar.in which verifies that the file is in
the list of rpm/tar files for that node, and then spits it out to
stdout. The web page uses fpassthru() to send the file out to the
client. The client is using wget, and is required to use https (the
web page checks).

At present, the external script is run as the creator of the
experiment, and gid of the experiment. Perhaps this is not a good
idea. In any event, the file must be in the list of rpm/tarfiles,
either owned by the experiment creator or with a group of the
experiment, and the file must reside in either /proj or /groups.
I use the realpath() function to make sure there are no symlink tricks
pointing to outside those filesystems. I use the standard NFS read goo to
prevent transient mount problems that we all know and love.

nodetypeXpid_permissions table, since most users will be confused
to see free PCs that they cannot actually use.

This query is starting to become a bit complicated to be doing on
every page.

based page that looks like the original Begin Experiment page. Be sure
to look at the page in both admin mode and non-admin mode since I had
some trouble determining how swappable is treated these days.

Oh, added the ability to convert non-batch experiments into batch, and
back. The experiment must be unlocked and in the swapped state to go
in either direction.

Also added the cpu_usage and mem_usage slots for editing. I added a
comment about planetlab only, since otherwise we would just confuse
normal users who have no idea what they mean. I could conditionalize
them on having plab nodes, but thats difficult to figure out in the
web page when the experiment is swapped out, so lets not worry about
it.

are couple left in the newnodes code that Rob will need to help me
with.

Soon image when the image is not available (cause the renderer is not
finished yet).

The default thumbsize is now 160, so do not call out to the renderer
to generate the thumb on the showexp page; just take it from the DB.

showexp page that its a batch experiment, by the menu options. Same
deal in the swapexp output, plus some other minor cleanup. The only
bug I found while trying to figure out the batchmode problem reported
this morning by the FileMover people, is that the cancelflag is not
cleared after swaping a running batch experiment out, so even after
reinjecting it into the queue, it will not run. Still, that does seem
to be what the FileMover people reported.

can I say? This page is public, and linked from various places.

Added site variables ('sitevars').
These are stored in the sitevariables database table.
Each one has a name, a description (NOT OPTIONAL!), a
default value, and a current value.
If the current value is NULL, the default value is used.

Soon, a mechanism will be added to the install process to
ensure all needed site variables exist before installing;
more on that when it is committed.

 - Added 'editsitevars.php3' page, accessable to admins
   via the 'Edit Site Variables' menu option.

 - Added 'setsitevar' script,
   an interface for listing, viewing in detail, and setting
   site variables.

 - Web interface now uses 'web/nologins' and 'web/message'
   instead of one-off database tables.

NOTE that setting a variable to the default value and
setting a variable to a value which is string-identical
are NOT the same thing.
(This doesn't matter yet, but when we push default values out to
 remote sites as part of our install, it will.)

the number of hours correctly. (Was using idlehours mod 24.)

In addition to the usual checks, project_root and group_root (in the
project) get the same permission as if they were in the subgroups with
group_root permission. This means they can swap/terminate/reboot etc
in subgroups even though they are not members of the subgroups.
There is still some copy problems with files, but leaving that till
later to deal with.

how long they've been idle and what the time of last activity was on their
nodes. And now do proper wordwrapping on the message instead of static
apriori newline placement.

Also add the ability to do some formatting on the idle time to those
functions, and use it to make the swap request message a little better.
("H hours, M minutes" instead of "X.Y hours").

only pid, to pid/gid like most other things in the testbed. Also add a
"global" slot to denote images that are globally available to all
projects (system images). The older "shared" attribute is now used to
denote images that are shared within a project (available to all
subgroups in the project). The migration path for existing DBs is
given in the migrate file. Be sure to run those commands on an
existing testbed or things will break!

www/newimageid, www/newimageid_ez: A bunch of changes for
shared/global attributes. Added a group menu to the form so users can
create images in subgroups. Beefed up the Java code that constructs
the path name to use the gid, shared, and global attributes of the
form to give the user the best possible path that we can. Improved the
pathname checking code so that we do not allow just any old path in
case the user elects to disregard the path we carefully constructed
for them. Also check the proj/group membership, and setup defaults for
users that have permission in just one pid/gid to create images.

libdb.in: Changed permission check in TBImageIDAccessCheck() to
reflect shared/global attribute changes.

os_load: Get rid of test that checked path of the image. The path
checking is done in the web interface anyway, so why duplicate in 4
places. Other minor changes reflecting shared->global name change.
Also note that images can come from the group directory now.

create_image: Get rid of test that checked path of the image. The path
checking is done in the web interface anyway, so why duplicate in 4
places. Also note that images can come from the group directory now.

www/dbdefs: Changed permission check in TBImageIDAccessCheck() to
reflect shared/global attribute changes.

www/showimageid_list, www/showstuff: Minor global/shared attribute
changes.

www/menu: Change osids/imageids pointer to point to the image list,
not the osid list. This is more reasonable for mere users who have
access to the EZ form, and thus never really need to concern
themselves with osids.

www/editimageid: Add proper pathname checking. There were no checks at
all before!

 - Added a bunch of functions to get time of last activity, hours idle,
   and staleness for nodes and experiments.
 - Fixed the showexp_list page to use the new idle measures, including
   idle view. Idle view now runs in about 1.3 seconds instead of 13.
 - Changed all the "show" functions for nodes and expts to show idleness.

More changes will be coming, but I wanted to commit this before I install
it for real.

More rework on the groups system.

* BESTOWGROUPROOT permission added to dbdefs.

* Permissions criteria for group operations changed in dbdefs
  (consult code for full explanation.)

* Approveuser and Editgroup now check for BESTOWGROUPROOT
  permissions before allowing changes to group_root.

* approveuser_form and editgroup_form do not show "Group Root"
  as an option unless you are allowed to set it (or it is already set.)

* editgroup does not UPDATE rows where trust has not been changed.

* showgroup does a correct check to see whether to show the
  "group options" subpage.

* Altered consistency checks to treat any root as equivalent
  (so, if you're project_root in the default group, but group_root in
   a group, that won't be a problem)

* Moved consistency checks, which were done in two different places into
  dbdefs TBCheckGroupTrustConsistency()

* Added preemptive checks, so if 'user' or '*_root' are not valid
  trusts, they aren't displayed as options in editgroup_form and
  approveuser_form (using above function)

* In approveuser, a new approval may now be sent to group_root.

Split notion of "EDITGROUP" permission into two:
"EDITGROUP" and "GROUPGRABUSERS".

"EDITGROUP" is easier to obtain;
            it is now given to group_root for the group.
"GROUPGRABUSERS" is how "EDITGROUP" _used_ to be:
                 only given to default-group_root or project_root.

The ability to add users to a group who have not requested membership
now requires "GROUPGRABUSERS".

Removing or editing members still requires only EDITGROUP.

So, the upshot is, now group_root users can edit and remove members from
their own groups.
But they still can't 'grab' users who haven't asked to join the group.
(which would enable them to mount arbitrary users' home dirs as
 root, which would be a Bad Thing.)

  leader wants to be in the subgroup, he has to do it via the editgroup
  page. This required minor changes in editgroup pages, since I was special
  casing the project leader to not allow removal/addition.

* Allow mere users to be the head of a group. This was previously not
  allowed, and is totally wrong since the entire group trust mechanism
  is based on giving subgroup members *more* privs then they have in
  the default (project) group.

* Change permission check in the showgroup page to allow non group members
  to look at the group if they have group_root or better in the default
  group. I noticed that once I took myself out of a group, I could no longer
  look at the group even though I had group_root in the project.

  Also change so that the edit/del menu does not appear unless the user
  has permission to do those things.

* Change consistency check when adding a group member. New test is simpler
  and makes sure that the user does not have root privs in the project and
  user privs in the subgroup. The reverse is of course okay, and the expected
  manner in which groups should be used.

* newgroup page now spits out a redirect to showgroup page, rather then
  printing the group info itself. Avoids duplication and gets rid of the
  form post from the history. Ditto for editgroup page.