GitLab

but that stuff is still not totally hooked in yet.

a huge amount of work, but really needs someone to test drive it.

until I figure out the rest of deleted users wrt stats. I was working on
this at some point, and some crisis must have intervened ...

on initial page load; mysqld has gone catatonic and we find out about
it many other ways, so lets not add more email to the flood.

does his own commit.

would be no big deal, except that we want to retain user_stats for
deleted users, and rather then a deleted_user_stats table, I want to
retain stats for deleted users in the user_stats table, since that
is a more natural place for them.

The main problem is that we use the login (uid) as the cross table
reference slot all over the DB, which is fundamentally incorrect, if
we want to be able reuse uids and still know what historical data
refers to.

So, I have taken a few baby steps towards weaning us off the uid, and
towards permanently unique key for users, using the unix_uid integer
for now, but probably something slightly different later.

The user_stats is now indexed on this new key (called uid_idx in the
users_stats table) instead of the plain uid.

The unix_uid slot in the users table is no longer an auto_increment
field, but instead uses the emulab_indicies table for the next
available index.

allows takes you to new (admin only) page to select a project that
node will be reserved for.

* The node is not actually *reserved*, it is *pre* reserved! The node
  stays in the free pool, and is available only for use in the project
  to which it is reserved.

* The node can already be reserved to some other project when you
  pre-reserve it. It is not until the current owner releases the node
  that the pre-reservation takes effect.

* The node free counts (on the web pages) count a free a node with a
  pre-reservation, as allocated. This way people do not see a free
  count that includes a node they will never be able to get.

for more then one user) as this messes up mailman. Besides, we will
eventually have to switch to email uids at some point, so they will
need to be unique anyway.

Add some checks for 'free' nodes that are not allocatable.

	* db/audit.in: Include the list of nodes that are not reserved but
	have an eventstate that makes them unallocatable.

	* www/dbdefs.php3.in: Add POWEROFF and ALWAYSUP node states.

	* www/nodecontrol_list.php3: Add an asterisk next to the free
	count for type(s) that have free, but unallocatable nodes.

	* www/shownodetype.php3: If a node is free, but unallocatable, put
	a yellow ball next to its name instead of a green one.

Jay has "comments"), but I do not want it hanging around in my source
tree. Here is my mail message:

* The "My Mailing Lists" is context sensitive (copied from Tim's
  changes to the My Bug Databases). It takes you to the *archives* for
  the current project (or subgroup) list. Or it takes you to your
  first joined project.

* The showproject and showgroup pages have direct links to the project
  and group specific archives. If you are in reddot mode, you also
  get a link to the admin page for the list. Note that project and
  group leaders are just plain members of these lists.

* The interface to create a new "user" list is:

	https://www.emulab.net/dev/stoller/newmmlist.php3

  We do not store the password, but just fire it over in the list
  creation process.

  Anyone can create their own mailing lists. They are not associated
  with projects, but just the person creating the list. That person
  is the list administrator and is given permission to access the
  configuration page.

  This page is not hooked in yet; not sure where.

* Once you have your own lists, you user profile page includes a link
  in the sub menu: Show Mailman Lists. From this page you can delete
  lists, zap to the admin page, or change the admin password (which is
  really just a subpage of the admin page).

* As usual, in reddot mode you can mess with anyone else's mailman lists,
  (via the magic of mailman cookies).

* Note on cross machine login. The mailman stuff has a really easy way
  to generate the right kind of cookie to give users access. You can
  generate a cookie to give user access, or to the admin interface for
  a list (a different cookie). Behind the scenes, I ssh over and get
  the cookie, and set it in the user's browser from boss. When the
  browser is redirected over to ops, that cookie goes along and gives
  the user the requested access. No passwords need be sent around,
  since we do the authentication ourselves.

have the user go through a set of hard to explain steps, just push
them through it using the web interface.

* New sitevars to control a little state machine used by the web
  interface.

* When first setting up a testbed, the sitevar value will force the
  web interface to present the user with a single menu option "Create
  New Project" and the "Home" link will take the user to that page.
  The user is instructed to login is as elabman.

* The user fills in the form as directed in setup-ops.txt. Even though
  he is logged in as elabman, the newproject form has been altered to
  operate as if no one is logged in. I also default a bunch more of
  the fields in this case.

* The user submits the form. Rather then pend the new project, just
  jump straight into approveproject. That grinds along as usual, and
  when it is done, the elabman account is frozen and the user logged
  out. The user gets a link inviting him to log back in as the user
  just created.

* Side effects of this new process:

	* The user is made an admin user (admin=1) automatically.
	* The user is added to the emulab-ops project as group_root.
	* The user verification process is skipped.
	* The user is added to the unixgroups wheel and tbadmin.

* I reworked this entire section of setup-db.txt ...

* The user still needs to give himself a real shell and password on
  boss, but I left that for the user to do explicitly. I also drop in
  a pointer to the shellonboss.txt. I might automate this part too at
  some point. Not sure yet.

Firewalled experiments (see tbsetup/elabinelab.in for the other stuff).

* To support firewalled experiments, needed to add a new virt_firewalls
  table to split the existing firewalls table up, which included both
  virtual and physical stuff. There are the usual frontend changes and a
  few other things scattered around, including tmcd.c.

* The firewall code in tbswap got some beefing up to support adding and
  deleting nodes from the its special control net vlan. Note that I have
  not made any progress on containment of deleted nodes, just as we do not
  do anything now for teardown (unless its paniced, in which case the
  experiment cannot be modified anyway).

* ptopgen and assign_wrapper got some interesting modifications: Unlike
  regular swapmod, we cannot just tear down all the vlans since that would
  interrupt everything inside the inner elab. Instead, leave the vlans as
  is. The problem is that when assign runs, it can just as easily pick
  different interfaces on the same nodes, which would be a royal pain in
  the ass to deal with! So, ptopgen got a new option (-u) that assign
  wrapper uses to tell ptopgen that it should prune out unused interfaces
  from nodes that are already allocated to the experiment. This is, at
  best, as pathetically gross hack, but it makes sure that all the
  interfaces stay the same across swapmods.

* The unrelated revision of elabinelab has a bunch of new code for adding
  and deleting nodes from the inner elab. Mostly it deals with dhcpd (inner
  and outer, waiting for nodes to reboot, etc). It also deals with updating
  the vlans table in the DB, pruning out any nodes (ports) that are deleted
  but for which there are still interfaces in existing vlans. Said ports
  are them moved back to the default vlan with calls to snmpit. Also under
  another revision a a couple of weeks ago are the web interface changes to
  support the newnode MFS inside an inner Emulab.

* swapexp and endexp got some more checks for firewalled and paniced
  experiments, which were missing.

some additional privs; allowed to view/edit node logs and histories of
all nodes, plus some others.

for the near future. Two big changes:

* Add WikiOnly accounts. An external user can register for an account on
  the wiki. Rather then use the registration stuff that comes with TWiki,
  redirect to new Emulab web page so we can manage all of the wiki accounts
  from one place. I modified the joinproject page to spit out a subset of
  the required fields so that its simple to get a wiki only account (just a
  few things to fill in).

  In keeping with current security practices, we still generate a
  verification email message to ensure the email address works. However,
  when the user completes the verification, the wiki account is created right
  away, rather then waiting for someone to approve it (since that would
  defeat the entire point of the wiki).

  Aside: I have not thought much about the conversion from a wiki-only
  account to a real account. That is going to happen, and it would be nice
  if that step did not require one of use to go in and hack the DB. Will
  cross that moat later.

  Aside: Rather beat up on the modify user info page too much, I continue
  to spit out the same form, but mark most of the fields as not required,
  and allow wiki-only people to not specify them.

* Both the joinproject and newproject pages sport a new WikiName field so
  that users can select their own WikiName. I added some JavaScript to
  both pages that generate a suitable wikiname from the FullName field, so
  that as soon as the user clicks out of the FullName, a default wikiname is
  inserted in the field.

  Both pages verify the wikinames by checking to make sure it is not
  already in use, and that it meets the WikiRules for WikiTopic names.
  (someone please shoot me if I continue to use WikiNotation).

message requesting information about an experiment (why has it been
swapped in for 6000 hours, tell us before we swap it out).

table that will prevent an experiment from being swapped/modified. The
toggle is on the showexp page, and the toggle is *not* admin
over-ridable; you must turn the toggle off (and of course, you must be
an admin to do that).

* Add new DB table "webcams" which hold the id of the webcam, the
  server it is attached to, and the last update time.

* Add new sitevars webcam/anyone_can_view and webcam/admins_can_view.
  Should be obvious what they mean.

* Add trivial script grabwebcams (invoked from cron) to grab the images
  from the servers and stash in /usr/testbed/webcams. The images are
  grabbed with scp, protected by a 5 second timeout. Fine for a couple
  of cameras.

* Add web page stuff to display webcams, linked from the robot mape page.

Permission to view the webcams is currently admin, or in a project that is
allowed to use a robot. We can tighten this up later as needed.

A bunch of engineering on the robot code.  I'm sure I've broken something,
but the majority of it is done and I wanted to get a checkpoint in.

	* GNUmakerules: Add rpcgen rules.

	* Makeconf.in: Add PATH and host_cpu variables so
	cross-compilation works properly.  Add JAVAC and JAR for java
	compilation.  Add BRAINSTEM_DIR that refers to a brainstem build
	directory to be used for the robot build.

	* configure, configure.in: Prepend the arm cross-compile dir to
	PATH.  Detect java for building applets.  Add --with-brainstem to
	specify the brainstem build dir.  Add --enable-mezzanine to turn
	on the mezzanine build.

	* robots/GNUmakefile.in: Add client target that builds the
	subdirs.

	* robots/emc/GNUmakefile.in, robots/emc/emcd.h, robots/emc/emcd.c,
	test_emcd.sh.in, robots/rmcd/GNUmakefile.in, robots/rmcd/rmcd.c,
	robots/rmcd/test_rmcd.sh.in, robots/vmcd/test_vmc-client.sh.in,
	robots/vmcd/test_vmcd.sh.in, robots/vmcd/test_vmcd2.sh,
	robots/vmcd/test_vmcd3.sh, robots/vmcd/test_vmcd4.sh,
	robots/vmcd/vmc-client.c, robots/vmcd/vmcd.c: Updates for the mtp
	switch to using rpcgen.

	* robots/emc/test_emcd.config: Restore missing config line.

	* robots/mtp/GNUmakefile.in, robots/mtp/global_bound.java,
	robots/mtp/mtp.h, robots/mtp/mtp.c, robots/mtp/mtp.java,
	robots/mtp/mtp.x, robots/mtp/mtp_command_goto.java,
	robots/mtp/mtp_command_stop.java, robots/mtp/mtp_config_rmc.java,
	robots/mtp/mtp_config_vmc.java, robots/mtp/mtp_control.java,
	robots/mtp/mtp_dump.c, robots/mtp/mtp_garcia_telemetry.java,
	robots/mtp/mtp_opcode_t.java, robots/mtp/mtp_packet.java,
	robots/mtp/mtp_payload.java, robots/mtp/mtp_recv.c,
	robots/mtp/mtp_request_id.java,
	robots/mtp/mtp_request_position.java,
	robots/mtp/mtp_robot_type_t.java, robots/mtp/mtp_role_t.java,
	robots/mtp/mtp_send.c, robots/mtp/mtp_status_t.java,
	robots/mtp/mtp_telemetry.java, robots/mtp/mtp_update_id.java,
	robots/mtp/mtp_update_position.java, robots/mtp/robot_config.java,
	robots/mtp/robot_position.java, robots/mtp/test_mtp.sh: Replace
	hand-generated stubs with xdr stubs for C and java.  Java stubs
	were generated by "remotetea's" jrpcgen.

	* robots/primotion/GNUmakefile.in,
	robots/primotion/buttonManager.hh,
	robots/primotion/buttonManager.cc, robots/primotion/dashboard.hh,
	robots/primotion/dashboard.cc, robots/primotion/flash-user-led.cc,
	robots/primotion/garcia-pilot.cc, robots/primotion/garciaUtil.hh,
	robots/primotion/garciaUtil.cc, robots/primotion/ledManager.hh,
	robots/primotion/ledManager.cc,
	robots/primotion/pilotButtonCallback.hh,
	robots/primotion/pilotButtonCallback.cc,
	robots/primotion/pilotClient.hh, robots/primotion/pilotClient.cc,
	robots/primotion/watch-user-button.cc,
	robots/primotion/wheelManager.hh,
	robots/primotion/wheelManager.cc: Replace gorobot with
	garcia-pilot, a beefed up daemon for controlling the robots.
	Improvements include: making use of the user LED and button to
	give some feedback and let the wrangler run a test sequence,
	reboot, and shutdown the robot; Logging of the battery level, how
	often the robot has moved and for how long, and the distance
	traveled; telemetry is sent back to emulab clients; movements are
	now just pivot-move instead of pivot-move-pivot, since the second
	pivot ends up being extra work most of the time; the robot will
	move backwards to cut down on the amount of rotation; and just
	generic cleanups to the code.

	* robots/primotion/garcia.config: The configuration file currently
	used on the garcias.

	* www/GNUmakefile.in: Add garcia-telemetry subdir to the build.

	* www/dbdefs.php3.in: Add TBNodeClass and TBNodeStatus functions.

	* www/garcia-telemetry/Base64.java,
	www/garcia-telemetry/GNUmakefile.in,
	www/garcia-telemetry/GarciaTelemetry.java,
	www/garcia-telemetry/UpdateThread.java,
	www/garcia-telemetry/main.xml: A telemetry applet for the garcia,
	it displays readouts for the various sensors and other bits of data
	gathered by the garcia-pilot daemon.  Hopefully, it will make a
	handy debugging tool.

	* www/garcia-telemetry.jar, www/mtp.jar, www/oncrpc.jar,
	www/thinlet.jar: Java jars used by the robot telemetry applet.

	* www/servicepipe.php3: A slightly enhanced version of
	ledpipe.php3 that can be used for other services, like robot
	telemetry.

	* www/shownode.php3: Add "Show Telemetry" menu item to robot
	nodes.

	* www/telemetry.php3: Telemetry page for the garcia-telemetry
	applet.

* tbsetup/panic.in: New backend script to implement the panic button
  feature. When used, it will cut the severe the connection to the
  firewall node by using snmpit to disable the port. Sets the panic
  bit (and date) in the experiments table, and changes the state of
  the experiment from "active" to "paniced" to ensure that the
  experiment cannot be messed with (swapped out or modified). Sends
  email to tbops when the panic button is pressed.

  Used with -r option, reverses the above. State is set back to
  active, the panic bit is cleared, and the port is renabled with
  snmpit.

* tbsetup/tbswap.in: During swapout, a firewalled experiment that has
  been paniced will get a cleaning; The nodes are powered off, then
  the osids for all the nodes are reset (with os_select) so that they
  will boot the MFS, and then the nodes are powered on. Then the
  control network is turned back on, and then I wait for the nodes to
  reboot (this is simply cause we do not record in the DB that a node
  is turned off, and if I do not wait, the reload daemon will end
  hitting the power button again if they do not reboot in time. We can
  fix this later.

  I am not planning to apply this to general firewalled experiments
  yet as the power cycling is going to be hard on the nodes, so would
  rather that we at least have a 1/2 baked plan before we do that.

* www/showexp.php3: If experiment is firewalled, show the Panic
  Button, linked to the panic button web script. If the experiment has
  already had the panic button pressed, show a big warning message and
  explain that user must talk to tbops to swap the experiment out.
  Also fiddle with menu options so that the terminate link is gone,
  and the swap link is visible only in admin mode. In other words, only
  an admin person can swap an experiment once it is paniced. And of
  course, an admin person can the backend panic script above with the
  -r option, but thats not something to be done lightly.

* db/libdb.pm.in: Add "paniced" as an experiment state (EXPTSTATE_PANICED).
  Add utility functions: TBExptSetPanicBit(), TBExptGetPanicBit(), and
  TBExptClearPanicBit().

* tbsetup/swapexp.in: Minor state fiddling so that an experiment can
  be swapped while in paniced state, but only when in admin mode. Also
  clear the panic bit when experiment is swapped out.

* www/dbdefs.php3.in: Add "paniced" as an experiment state. Add a
  utility function TBExptFirewall() to see if experiment is firewalled.

* www/panicbutton.php3: New web script to invoke the backend panic
  script mentioned above, after the usual confirm song and dance.

* www/panicbutton.gif: New gif of a red panic button that I stole off
  the net. If anyone has sees/has a better one, feel free to replace
  this one.

* utils/node_statewait.in: Add -s option so that I can pass in the
  state I want to wait for (used from tbswap above to wait for nodes
  to reach ISUP after power on).

TBImageIDAccessCheck, otherwise they seem to return true all the
time...

currently available to only people with stud=1 status in the DB.

* www/tbauth.php3: Add a STUDLY() function to check that bit.

* www/linktest.php3: New page to run linktest on the fly. The level
  defaults to the current level in the experiments table, but you can
  override that via the form on the page.

* www/showexp.php3: Add link to aforementioned page. STUDLY() only.

* www/beginexp_form.php3: Add an option (selection) to set the linktest
  level for create/swapin. Defaults to 0 (no linktest). STUDLY() only.

* www/editexp.php3: Add an option to edit the default linktest level
  for an experiment. STUDLY() only.

* tbsetup/batchexp.in and tbsetup/swapexp.in: Add code to optionally run
  the linktest, sending email if it fails (exists with non-zero status).
  Failure does not affect the swapin.

Change ptopgen to look at the eventstate of a node; a node is not considered
free unless it is ISUP or PXEWAIT.

Add TBAvailablePCs() to libdb, removing the corresponding code from
assign_wrapper. This routine does the equiv of ptopgen, returning the
number of PCs that are available for use (looking at eventstate).

Change TBFreePCs in the web interface accordingly.

The above changes correspond to an upcoming change in stated.

error - it's too hard to destingish a string from 0 in php, and every
place we call this function, we've already made sure the group exists
anyway.

account, and direct user to Forgot Username/Password page.

Changed the Forgot page to allow user to *either* reset password, or
just find out their username if that is what they forgot.

zillions of DB fields that we have to set. My solution was to add a
meta table that describes what is a legal value for each table/slot
for which we take from user input. The table looks like this right
now, but is likely to adapt as we get more experience with this
approach (or it might get tossed if it turns out to be a pain in the
ass!).

	CREATE TABLE table_regex (
	  table_name varchar(64) NOT NULL default '',
	  column_name varchar(64) NOT NULL default '',
	  column_type enum('text','int','float') default NULL,
	  check_type enum('regex','function','redirect') default NULL,
	  check tinytext NOT NULL,
	  min int(11) NOT NULL default '0',
	  max int(11) NOT NULL default '0',
	  comment tinytext,
	  UNIQUE KEY table_name (table_name,column_name)
	) TYPE=MyISAM;

Entries in this table look like this:

	('virt_nodes','vname','text','regex','^[-\\w]+$',1,32,NULL);

Which says that the vname slot of the virt_nodes table (which we trust the
user to give us in some form) is a text field to be checked with the given
regex (perlre of course), and that the min/max length of the text field is
1 and 32 chars respectively.

Now, you wouldn't want to write the same regex over and over, and since we
use the same fields in many tables (like pid, eid, vname, etc) there is an
option to redirect to another entry (recursively). So, for "PID" I do this:

        ('eventlist','pid','text','redirect','projects:pid',0,0,NULL);

which redirects to:

	('projects','pid','text','regex','^[a-zA-Z][-\\w]+$',2,12,NULL);

And, for many fields you just want to describe generically what could go
into it. For that I have defined some default fields. For example, a user
description:

        ('experiment,'usr_name','text','redirect','default:tinytext',0,0,NULL);

which redirects to:

	('default','tinytext','text','regex','^[\\040-\\176]*$',0,256,NULL);

and this says that a tinytext (in our little corner of the database
universe) field can have printable characters (but not a newline), and
since its a tinytext field, its maxlen is 256 chars.

You also have integer fields, but these are little more irksome in the
details.

	('default','tinyint,'int,'regex','^[\\d]+$',-128,127,NULL);

and you would use this anyplace you do not care about the min/max values
being something specific in the tinyint range. The range for a float is of
course stated as an integer, and thats kinda bogus, but we do not have many
floats, and they generally do not take on specific values anyway.

A note about the min/max fields and redirecting. If the initial entry has
non-zero min/max fields, those are the min mac fields used. Otherwise they
come from the default. So for example, you can do this:

    ('experiments','mem_usage','int','redirect','default:tinyint',0,5,NULL);

So, you can redirect to the standard "tinyint" regular expression, but you
still get to define min/max for the specific field.

Isn't this is really neat and really obtuse too? Sure, you can say it.

Anyway, xmlconvert now sends all of its input through these checks (its
all wrapped up in library calls), and if a slot does not have an entry, it
throws an error so that we are forced to define entries for new slots as we
add them.

In the web page, I have changed all of the public pages (login, join
project, new project, and a couple of others) to also use these checks.
As with the perl code, its all wrapped up in a library. Lots more code
needs to be changed of course, but this is a start.

devel tree.

select the default user interface for a project. The choice is current
'emulab or 'plab', defaults to 'emulab'. New users that join emulab
get the default user interface from the first project they join.

Also generalize the plab_user bit as new "user_interface" slot of the
users table, which is an enum of interface tokens, currently either
'emulab' or 'plab', defaults to 'emulab'. The plab_user bit will be
removed later.

planetlab nodes.

of the CHECKEMAIL function as a result.

state machine (state). All of the stuff that was previously handled by
using batchstate is now embedded into the one state machine. Of
course, these mostly overlapped, so its not that much of a change,
except that we also redid the machine, adding more states (for
example, modify phases are now explicit. To get a picture of the
actual state machine, on boss:

		stategraph -o newstates EXPTSTATE
		gv newstates.ps

Things to note:

* The "batchstate" slot of the experiments table is now used solely to
  provide a lock for batch daemon. A secondary change will be to
  change the slot name to something more appropriate, but it can
  happen anytime after this new stuff is installed.

* I have left expt_locked for now, but another later change will be to remove
  expt_locked, and change it to active_busy or some such new state name in
  the state machine. I have removed most uses of expt_locked, except those
  that were necessary until there is a new state to replace it.

* These new changes are an implementation of the new state machine,
  but I have not done anything fancy. Most of the code is the same as
  it was before.

* I suspect that there are races with the batch daemon now, but they
  are going to be rare, and the end result is probably that a
  cancelation is delayed a little bit.

for key upload). Add more regexes and addslashes().

* Use superglobals for page/form arguments.

* Add regex functions for email and phone number.

* Remove stripslashes calls; not needed and actually incorrect for
  data returned from the DB.

* Add TBvalid_uid() function to regex uid's. To be used throughout the
  system. Eventually add routines for checking other things like pids
  and eids, etc.

* Regex the uid value we get from the cookie, and switch to $_COOKIE
  superglobal.

* Strict regex checking in DOLOGIN() of uid.

* Change login.php to use superglobals, and general tightening of
  parameter checking.