GitLab

to numbered update scripts. To install updates:

	boss> cd /your/srcdir/sql/updates
	boss> perl /your/objdir/db/dbupdate tbdb

For developers, please look in the sql/updates/4/1 file for an example
of how to do updates. For each revision in database-create.sql, create
a corresponding script and add it to the repository. The scripts will
be applied in numeric order (see db/dbupdate.in), starting with the
version recorded in the version_info table. This table is updated by
dbupdate as it runs each script.

instead of paths. There is a script called from database-migrate that
converts paths to logfile (db) objects.

sandbox, and that I did. It falls back to the older archive when
the template is older then CVS repos.

But along the way I got annoyed with the fact that template instantiation
does not provide a logfile to the web interface. The reason is that
the current logfile stuff is very experiment centric; there has to be an
experiment and an attached logfile. An instance does not have an experiment
until really late in the game so the code was just not bothering.

Anyway, I've started to generalize the logfile stuff with a new table
and the approach that a logfile is named by a random key, and if you
know the key you can look at the logfile in the web (since without an
experiment it is hard to do permission checks unless we make logfiles
uid/gid owned, and I did not want to do that.

menus that used to be on the left side, are now drop down menus along
the top. The banner across the top will also be smaller and more space
efficient.

The primary motivation for these changes is to leave more room for
actual content in the main window.

Tested with Firefox (1.5 and 2.0), Safari (2.0 and 3.0) and on
Internet Explorer Version 7. IE6 has some visual issues that might
make it a little annoying, but should be viewed as incentive to
upgrade to IE7 since it has been out a long time.

  register_globals=1 to turn POST/GET/COOKIES arguments in local variables.
  This is known to be a terrible security risk, and we keep saying we are
  going to fix it, and now I am. In order to accomplish this on a
  transitional basis (since I don't want the entire web interface to stop
  working while I debug it), and because the code just needs the cleanup, I
  am doing it like this: Each page will sport new declarations at the top:

	RequiredPageArguments("experiment", PAGEARG_EXPERIMENT,
                              "template",   PAGEARG_TEMPLATE,
                              "instance",   PAGEARG_INSTANCE,
                              "metadata",   PAGEARG_METADATA,
                              "osinfo",     PAGEARG_OSINFO,
                              "image",      PAGEARG_IMAGE,
                              "project",    PAGEARG_PROJECT,
                              "group",      PAGEARG_GROUP,
                              "user",       PAGEARG_USER,
			      "node",       PAGEARG_NODE,
			      "yesno",      PAGEARG_BOOLEAN,
			      "message",    PAGEARG_STRING,
			      "age",        PAGEARG_INTEGER,
                              "cost",       PAGEARG_NUMERIC,
                              "formfields", PAGEARG_ARRAY,
                              "unknown",    PAGEARG_ANYTHING);

	OptionalPageArguments("canceled", PAGEARG_BOOLEAN);

  The first token in each pair is the name of the global variable to
  set, and the second token is the type. So, for "experiment" we look at
  the URL for a pid/eid or exptidx, etc, sanity check them (safe for a
  DB query), and then try to find that experiment in the DB. If it maps
  to an experiment, set global variable $experiment to the object. Since
  its a required argument, produce an error if not supplied. Similar
  treatment for optional arguments, with the obvious difference.

  The goal is to have ALL argument processing in one place, consistent,
  and correct. I've found numerous places where we leak unchecked
  arguments into queries. It also cuts out a lot of duplicated code.

* To make the above easier to deal with, I've been replacing lots of
  hardcoded URLS in the code of the form:

	foo.php3?pid=$pid&eid=$eid ...

  with

        CreateURL("foo", $experiment)

  which creates and returns the neccessary url string, by looking at
  the type of its arguments (experiment, template, instance, etc.)

  Eventually plan to replace them all so that URL handling throughout
  the code is all defined in one place (all the new URL code is in
  url_defs.php).

* I have cranked up error reporting to tell me anytime a variable is
  used before it is initialized, plus a bunch of other stuff that PHP
  deems improper. Think of it like -Wall ... and boy we get a lot of
  warnings.  A very large percentage of the diffs are to fix all these
  warnings.

  The warnings are currently going to /usr/testbed/log/php-errors.log,
  and I'll be adding a script to capture them each night and mail them
  to tbops. This file also gets errors (this will be a change for
  developers; rather then seeing errors and warnings dumped in the
  middle of web pages, they will go to this file instead).

* Major refactoring of the code. More objects (nodes, images, osids).
  Moving tons of queries into the objects in the hopes of someday
  getting to a point where we can split the web interface onto a
  different server.  Lots of general cleanup.

methods.

Change a bunch of queries that join on the users table to use the idx
instead of the uid. This will need to be completed before we can really
archive users away, but close at this point.

most of the rest of the tables in the system (still a few exceptions).
Bound to be some bugs ...

converting to locally unique ids and later globally unique ids.

of adding copyrights to them.

table, into a new table called node_type_attributes, which is intended
to be a more extensible way of describing nodes.

The only things left in the node_types table will be type,class and the
various isXXX boolean flags, since we use those in numerous joins all over
the system (ie: when discriminating amongst nodes).

For the most part, all of that other stuff is rarely used, or used in
contexts where the information is needed, but not for type descrimination.
Still, it made for a lot of queries to change!

Along the way I added a NodeType library module that represents the type
info as a perl object. I also beefed up the existing Node module, and
started using it in more places. I also added an Interfaces module, but I
have not done much with that yet.

I have not yet removed all the slots from the node_types table; I plan to
run the new code for a few days and then remove the slots.

Example using the new NodeType object:

	use NodeType;

	my $typeinfo = NodeType->Lookup($type);

        if ($typeinfo->control_interface(\$control_iface) ||
            !$control_iface) {
  	    warn "No control interface for $type is defined in the DB!\n";
        }

or using the Node:

	use Node;

        my $nodeobject = Node->Lookup($node_id);
        my $imageable  = $nodeobject->NodeTypeInfo()->imageable();
or
        my $rebootable = $nodeobject->isrebootable();
or
        $nodeobject->NodeTypeAttribute("control_interface", \$control_iface);

Lots of way to accomplish the same thing, but the main point is that the
Node is able to override the NodeType (if it wants to), which I think is
necessary for flexibly describing one/two of a kind things like switches, etc.

number of pcs in use by the project at the top of the project page.

I also changed to the project listing page to show both all nodes and
PCs in use, sortable on either.

but that stuff is still not totally hooked in yet.

a huge amount of work, but really needs someone to test drive it.

until I figure out the rest of deleted users wrt stats. I was working on
this at some point, and some crisis must have intervened ...

on initial page load; mysqld has gone catatonic and we find out about
it many other ways, so lets not add more email to the flood.

does his own commit.

would be no big deal, except that we want to retain user_stats for
deleted users, and rather then a deleted_user_stats table, I want to
retain stats for deleted users in the user_stats table, since that
is a more natural place for them.

The main problem is that we use the login (uid) as the cross table
reference slot all over the DB, which is fundamentally incorrect, if
we want to be able reuse uids and still know what historical data
refers to.

So, I have taken a few baby steps towards weaning us off the uid, and
towards permanently unique key for users, using the unix_uid integer
for now, but probably something slightly different later.

The user_stats is now indexed on this new key (called uid_idx in the
users_stats table) instead of the plain uid.

The unix_uid slot in the users table is no longer an auto_increment
field, but instead uses the emulab_indicies table for the next
available index.

allows takes you to new (admin only) page to select a project that
node will be reserved for.

* The node is not actually *reserved*, it is *pre* reserved! The node
  stays in the free pool, and is available only for use in the project
  to which it is reserved.

* The node can already be reserved to some other project when you
  pre-reserve it. It is not until the current owner releases the node
  that the pre-reservation takes effect.

* The node free counts (on the web pages) count a free a node with a
  pre-reservation, as allocated. This way people do not see a free
  count that includes a node they will never be able to get.

for more then one user) as this messes up mailman. Besides, we will
eventually have to switch to email uids at some point, so they will
need to be unique anyway.

Add some checks for 'free' nodes that are not allocatable.

	* db/audit.in: Include the list of nodes that are not reserved but
	have an eventstate that makes them unallocatable.

	* www/dbdefs.php3.in: Add POWEROFF and ALWAYSUP node states.

	* www/nodecontrol_list.php3: Add an asterisk next to the free
	count for type(s) that have free, but unallocatable nodes.

	* www/shownodetype.php3: If a node is free, but unallocatable, put
	a yellow ball next to its name instead of a green one.

Jay has "comments"), but I do not want it hanging around in my source
tree. Here is my mail message:

* The "My Mailing Lists" is context sensitive (copied from Tim's
  changes to the My Bug Databases). It takes you to the *archives* for
  the current project (or subgroup) list. Or it takes you to your
  first joined project.

* The showproject and showgroup pages have direct links to the project
  and group specific archives. If you are in reddot mode, you also
  get a link to the admin page for the list. Note that project and
  group leaders are just plain members of these lists.

* The interface to create a new "user" list is:

	https://www.emulab.net/dev/stoller/newmmlist.php3

  We do not store the password, but just fire it over in the list
  creation process.

  Anyone can create their own mailing lists. They are not associated
  with projects, but just the person creating the list. That person
  is the list administrator and is given permission to access the
  configuration page.

  This page is not hooked in yet; not sure where.

* Once you have your own lists, you user profile page includes a link
  in the sub menu: Show Mailman Lists. From this page you can delete
  lists, zap to the admin page, or change the admin password (which is
  really just a subpage of the admin page).

* As usual, in reddot mode you can mess with anyone else's mailman lists,
  (via the magic of mailman cookies).

* Note on cross machine login. The mailman stuff has a really easy way
  to generate the right kind of cookie to give users access. You can
  generate a cookie to give user access, or to the admin interface for
  a list (a different cookie). Behind the scenes, I ssh over and get
  the cookie, and set it in the user's browser from boss. When the
  browser is redirected over to ops, that cookie goes along and gives
  the user the requested access. No passwords need be sent around,
  since we do the authentication ourselves.

have the user go through a set of hard to explain steps, just push
them through it using the web interface.

* New sitevars to control a little state machine used by the web
  interface.

* When first setting up a testbed, the sitevar value will force the
  web interface to present the user with a single menu option "Create
  New Project" and the "Home" link will take the user to that page.
  The user is instructed to login is as elabman.

* The user fills in the form as directed in setup-ops.txt. Even though
  he is logged in as elabman, the newproject form has been altered to
  operate as if no one is logged in. I also default a bunch more of
  the fields in this case.

* The user submits the form. Rather then pend the new project, just
  jump straight into approveproject. That grinds along as usual, and
  when it is done, the elabman account is frozen and the user logged
  out. The user gets a link inviting him to log back in as the user
  just created.

* Side effects of this new process:

	* The user is made an admin user (admin=1) automatically.
	* The user is added to the emulab-ops project as group_root.
	* The user verification process is skipped.
	* The user is added to the unixgroups wheel and tbadmin.

* I reworked this entire section of setup-db.txt ...

* The user still needs to give himself a real shell and password on
  boss, but I left that for the user to do explicitly. I also drop in
  a pointer to the shellonboss.txt. I might automate this part too at
  some point. Not sure yet.

Firewalled experiments (see tbsetup/elabinelab.in for the other stuff).

* To support firewalled experiments, needed to add a new virt_firewalls
  table to split the existing firewalls table up, which included both
  virtual and physical stuff. There are the usual frontend changes and a
  few other things scattered around, including tmcd.c.

* The firewall code in tbswap got some beefing up to support adding and
  deleting nodes from the its special control net vlan. Note that I have
  not made any progress on containment of deleted nodes, just as we do not
  do anything now for teardown (unless its paniced, in which case the
  experiment cannot be modified anyway).

* ptopgen and assign_wrapper got some interesting modifications: Unlike
  regular swapmod, we cannot just tear down all the vlans since that would
  interrupt everything inside the inner elab. Instead, leave the vlans as
  is. The problem is that when assign runs, it can just as easily pick
  different interfaces on the same nodes, which would be a royal pain in
  the ass to deal with! So, ptopgen got a new option (-u) that assign
  wrapper uses to tell ptopgen that it should prune out unused interfaces
  from nodes that are already allocated to the experiment. This is, at
  best, as pathetically gross hack, but it makes sure that all the
  interfaces stay the same across swapmods.

* The unrelated revision of elabinelab has a bunch of new code for adding
  and deleting nodes from the inner elab. Mostly it deals with dhcpd (inner
  and outer, waiting for nodes to reboot, etc). It also deals with updating
  the vlans table in the DB, pruning out any nodes (ports) that are deleted
  but for which there are still interfaces in existing vlans. Said ports
  are them moved back to the default vlan with calls to snmpit. Also under
  another revision a a couple of weeks ago are the web interface changes to
  support the newnode MFS inside an inner Emulab.

* swapexp and endexp got some more checks for firewalled and paniced
  experiments, which were missing.

some additional privs; allowed to view/edit node logs and histories of
all nodes, plus some others.

for the near future. Two big changes:

* Add WikiOnly accounts. An external user can register for an account on
  the wiki. Rather then use the registration stuff that comes with TWiki,
  redirect to new Emulab web page so we can manage all of the wiki accounts
  from one place. I modified the joinproject page to spit out a subset of
  the required fields so that its simple to get a wiki only account (just a
  few things to fill in).

  In keeping with current security practices, we still generate a
  verification email message to ensure the email address works. However,
  when the user completes the verification, the wiki account is created right
  away, rather then waiting for someone to approve it (since that would
  defeat the entire point of the wiki).

  Aside: I have not thought much about the conversion from a wiki-only
  account to a real account. That is going to happen, and it would be nice
  if that step did not require one of use to go in and hack the DB. Will
  cross that moat later.

  Aside: Rather beat up on the modify user info page too much, I continue
  to spit out the same form, but mark most of the fields as not required,
  and allow wiki-only people to not specify them.

* Both the joinproject and newproject pages sport a new WikiName field so
  that users can select their own WikiName. I added some JavaScript to
  both pages that generate a suitable wikiname from the FullName field, so
  that as soon as the user clicks out of the FullName, a default wikiname is
  inserted in the field.

  Both pages verify the wikinames by checking to make sure it is not
  already in use, and that it meets the WikiRules for WikiTopic names.
  (someone please shoot me if I continue to use WikiNotation).

message requesting information about an experiment (why has it been
swapped in for 6000 hours, tell us before we swap it out).

table that will prevent an experiment from being swapped/modified. The
toggle is on the showexp page, and the toggle is *not* admin
over-ridable; you must turn the toggle off (and of course, you must be
an admin to do that).

* Add new DB table "webcams" which hold the id of the webcam, the
  server it is attached to, and the last update time.

* Add new sitevars webcam/anyone_can_view and webcam/admins_can_view.
  Should be obvious what they mean.

* Add trivial script grabwebcams (invoked from cron) to grab the images
  from the servers and stash in /usr/testbed/webcams. The images are
  grabbed with scp, protected by a 5 second timeout. Fine for a couple
  of cameras.

* Add web page stuff to display webcams, linked from the robot mape page.

Permission to view the webcams is currently admin, or in a project that is
allowed to use a robot. We can tighten this up later as needed.

A bunch of engineering on the robot code.  I'm sure I've broken something,
but the majority of it is done and I wanted to get a checkpoint in.

	* GNUmakerules: Add rpcgen rules.

	* Makeconf.in: Add PATH and host_cpu variables so
	cross-compilation works properly.  Add JAVAC and JAR for java
	compilation.  Add BRAINSTEM_DIR that refers to a brainstem build
	directory to be used for the robot build.

	* configure, configure.in: Prepend the arm cross-compile dir to
	PATH.  Detect java for building applets.  Add --with-brainstem to
	specify the brainstem build dir.  Add --enable-mezzanine to turn
	on the mezzanine build.

	* robots/GNUmakefile.in: Add client target that builds the
	subdirs.

	* robots/emc/GNUmakefile.in, robots/emc/emcd.h, robots/emc/emcd.c,
	test_emcd.sh.in, robots/rmcd/GNUmakefile.in, robots/rmcd/rmcd.c,
	robots/rmcd/test_rmcd.sh.in, robots/vmcd/test_vmc-client.sh.in,
	robots/vmcd/test_vmcd.sh.in, robots/vmcd/test_vmcd2.sh,
	robots/vmcd/test_vmcd3.sh, robots/vmcd/test_vmcd4.sh,
	robots/vmcd/vmc-client.c, robots/vmcd/vmcd.c: Updates for the mtp
	switch to using rpcgen.

	* robots/emc/test_emcd.config: Restore missing config line.

	* robots/mtp/GNUmakefile.in, robots/mtp/global_bound.java,
	robots/mtp/mtp.h, robots/mtp/mtp.c, robots/mtp/mtp.java,
	robots/mtp/mtp.x, robots/mtp/mtp_command_goto.java,
	robots/mtp/mtp_command_stop.java, robots/mtp/mtp_config_rmc.java,
	robots/mtp/mtp_config_vmc.java, robots/mtp/mtp_control.java,
	robots/mtp/mtp_dump.c, robots/mtp/mtp_garcia_telemetry.java,
	robots/mtp/mtp_opcode_t.java, robots/mtp/mtp_packet.java,
	robots/mtp/mtp_payload.java, robots/mtp/mtp_recv.c,
	robots/mtp/mtp_request_id.java,
	robots/mtp/mtp_request_position.java,
	robots/mtp/mtp_robot_type_t.java, robots/mtp/mtp_role_t.java,
	robots/mtp/mtp_send.c, robots/mtp/mtp_status_t.java,
	robots/mtp/mtp_telemetry.java, robots/mtp/mtp_update_id.java,
	robots/mtp/mtp_update_position.java, robots/mtp/robot_config.java,
	robots/mtp/robot_position.java, robots/mtp/test_mtp.sh: Replace
	hand-generated stubs with xdr stubs for C and java.  Java stubs
	were generated by "remotetea's" jrpcgen.

	* robots/primotion/GNUmakefile.in,
	robots/primotion/buttonManager.hh,
	robots/primotion/buttonManager.cc, robots/primotion/dashboard.hh,
	robots/primotion/dashboard.cc, robots/primotion/flash-user-led.cc,
	robots/primotion/garcia-pilot.cc, robots/primotion/garciaUtil.hh,
	robots/primotion/garciaUtil.cc, robots/primotion/ledManager.hh,
	robots/primotion/ledManager.cc,
	robots/primotion/pilotButtonCallback.hh,
	robots/primotion/pilotButtonCallback.cc,
	robots/primotion/pilotClient.hh, robots/primotion/pilotClient.cc,
	robots/primotion/watch-user-button.cc,
	robots/primotion/wheelManager.hh,
	robots/primotion/wheelManager.cc: Replace gorobot with
	garcia-pilot, a beefed up daemon for controlling the robots.
	Improvements include: making use of the user LED and button to
	give some feedback and let the wrangler run a test sequence,
	reboot, and shutdown the robot; Logging of the battery level, how
	often the robot has moved and for how long, and the distance
	traveled; telemetry is sent back to emulab clients; movements are
	now just pivot-move instead of pivot-move-pivot, since the second
	pivot ends up being extra work most of the time; the robot will
	move backwards to cut down on the amount of rotation; and just
	generic cleanups to the code.

	* robots/primotion/garcia.config: The configuration file currently
	used on the garcias.

	* www/GNUmakefile.in: Add garcia-telemetry subdir to the build.

	* www/dbdefs.php3.in: Add TBNodeClass and TBNodeStatus functions.

	* www/garcia-telemetry/Base64.java,
	www/garcia-telemetry/GNUmakefile.in,
	www/garcia-telemetry/GarciaTelemetry.java,
	www/garcia-telemetry/UpdateThread.java,
	www/garcia-telemetry/main.xml: A telemetry applet for the garcia,
	it displays readouts for the various sensors and other bits of data
	gathered by the garcia-pilot daemon.  Hopefully, it will make a
	handy debugging tool.

	* www/garcia-telemetry.jar, www/mtp.jar, www/oncrpc.jar,
	www/thinlet.jar: Java jars used by the robot telemetry applet.

	* www/servicepipe.php3: A slightly enhanced version of
	ledpipe.php3 that can be used for other services, like robot
	telemetry.

	* www/shownode.php3: Add "Show Telemetry" menu item to robot
	nodes.

	* www/telemetry.php3: Telemetry page for the garcia-telemetry
	applet.

* tbsetup/panic.in: New backend script to implement the panic button
  feature. When used, it will cut the severe the connection to the
  firewall node by using snmpit to disable the port. Sets the panic
  bit (and date) in the experiments table, and changes the state of
  the experiment from "active" to "paniced" to ensure that the
  experiment cannot be messed with (swapped out or modified). Sends
  email to tbops when the panic button is pressed.

  Used with -r option, reverses the above. State is set back to
  active, the panic bit is cleared, and the port is renabled with
  snmpit.

* tbsetup/tbswap.in: During swapout, a firewalled experiment that has
  been paniced will get a cleaning; The nodes are powered off, then
  the osids for all the nodes are reset (with os_select) so that they
  will boot the MFS, and then the nodes are powered on. Then the
  control network is turned back on, and then I wait for the nodes to
  reboot (this is simply cause we do not record in the DB that a node
  is turned off, and if I do not wait, the reload daemon will end
  hitting the power button again if they do not reboot in time. We can
  fix this later.

  I am not planning to apply this to general firewalled experiments
  yet as the power cycling is going to be hard on the nodes, so would
  rather that we at least have a 1/2 baked plan before we do that.

* www/showexp.php3: If experiment is firewalled, show the Panic
  Button, linked to the panic button web script. If the experiment has
  already had the panic button pressed, show a big warning message and
  explain that user must talk to tbops to swap the experiment out.
  Also fiddle with menu options so that the terminate link is gone,
  and the swap link is visible only in admin mode. In other words, only
  an admin person can swap an experiment once it is paniced. And of
  course, an admin person can the backend panic script above with the
  -r option, but thats not something to be done lightly.

* db/libdb.pm.in: Add "paniced" as an experiment state (EXPTSTATE_PANICED).
  Add utility functions: TBExptSetPanicBit(), TBExptGetPanicBit(), and
  TBExptClearPanicBit().

* tbsetup/swapexp.in: Minor state fiddling so that an experiment can
  be swapped while in paniced state, but only when in admin mode. Also
  clear the panic bit when experiment is swapped out.

* www/dbdefs.php3.in: Add "paniced" as an experiment state. Add a
  utility function TBExptFirewall() to see if experiment is firewalled.

* www/panicbutton.php3: New web script to invoke the backend panic
  script mentioned above, after the usual confirm song and dance.

* www/panicbutton.gif: New gif of a red panic button that I stole off
  the net. If anyone has sees/has a better one, feel free to replace
  this one.

* utils/node_statewait.in: Add -s option so that I can pass in the
  state I want to wait for (used from tbswap above to wait for nodes
  to reach ISUP after power on).

TBImageIDAccessCheck, otherwise they seem to return true all the
time...

currently available to only people with stud=1 status in the DB.

* www/tbauth.php3: Add a STUDLY() function to check that bit.

* www/linktest.php3: New page to run linktest on the fly. The level
  defaults to the current level in the experiments table, but you can
  override that via the form on the page.

* www/showexp.php3: Add link to aforementioned page. STUDLY() only.

* www/beginexp_form.php3: Add an option (selection) to set the linktest
  level for create/swapin. Defaults to 0 (no linktest). STUDLY() only.

* www/editexp.php3: Add an option to edit the default linktest level
  for an experiment. STUDLY() only.

* tbsetup/batchexp.in and tbsetup/swapexp.in: Add code to optionally run
  the linktest, sending email if it fails (exists with non-zero status).
  Failure does not affect the swapin.