GitLab

being taken.

The basic strategy is to have node_reboot (when -p option supplied)
invoke a special command on the node that will cause the shutdown
procedure to run prepare as it goes single user, but before the
network is turned off and the machine rebooted. The output of the
prepare run is capture and send back via the tmcd BOOTLOG command and
stored in the DB, so that create_image can dump that to the logfile
(so that the person taking the image can know for certain that the
prepare ran and finished okay).

On linux this is pretty easy to arrange since reboot is actually
shutdown and shutdown runs the K scripts in /etc/rc.d/rc6.d, and at
the end the node is basically single user mode. I just added a new
script to run prepare and send back the output.

On FreeBSD this is a lot harder since there are no decent hooks.
Instead, I had to hack up init (see tmcd/freebsd/init/{4,5,6}) with
some simple code that looks for a command to run instead of going to a
single user shell. The command (script) runs prepare, sends the output
back to tmcd, and then does a real reboot.

Okay, so how to get -p passed to node_reboot? I hacked up the
libadminmfs code slightly to do that, with new 'prepare' argument
option. This may not be the best approach; might have to do this as a
real state transition if problems develop. I will wait and see.

Also, I changed www/loadimage.php3 to spew the output of the
create_image to the browser.

and the makefile changes to get it installed.

version of TWiki, which will not run on ops until it is updated.

except that we do not "skip grant tables" on ops.

    - added a new defs var, TBROBOCOPSEMAIL

  * tbsetup/power_mail.pm.in:
    - add some new info to robot powerup mails

  * db/libdb.pm.in:
    - add a new function to determine if an experiment contains nodes of a
      given class/type

  * tbsetup/swapexp.in:
    - check if exp is a robot exp; that is, if it has robots or motes; if
      so, cc error msgs to TBROBOCOPSEMAIL in addition to TBOPS

GNUmakefile auto generated.

Parse the NS file with the real NS parser so we can make sure linktest is
doing the "right" thing.

	* configure, configure.in: Add tbsetup/nsverify files.

	* tbsetup/GNUmakefile.in: Add nsverify subdir.

	* tbsetup/tbprerun.in: Run verify-ns on the experiments NS file.

	* tbsetup/ns2ir/nstb_compat.tcl: Bring up-to-date with the current
	world.

	* tbsetup/nsverify/GNUmakefile.in: Makefile.

	* tbsetup/nsverify/ns-2.27.patch: Patch file for NS version 2.27.

	* tbsetup/nsverify/nstbparse.in: Wrapper for the NS parser.

	* tbsetup/nsverify/tb_compat.tcl: Different version of
	tb_compat.tcl that is used to verify linktest parameters.

	* tbsetup/nsverify/verify-ns.in: Script that runs on boss and
	verifies that the testbed parser worked correctly.

	* tbsetup/ns2ir/parse-ns.in, tbsetup/ns2ir/parse.proxy.in: Tweaked
	a bit so parse.proxy can be used to run the regular NS parser in
	addition to the testbed one.

figuring out how to cope with the new dhclient in FreeBSD 6 and above which
only handles a single interface.

of the archive.

First cut at a daemon that does regular checkups of the testbed
hardware/software.

	* configure, configure.in: Add tbsetup/checkup directory.

	* db/audit.in: Add a listing of stuck checkups.

	* install/boss-install.in: Add 'elabckup' user.

	* rc.d/3.testbed.sh.in: Startup the checkup_daemon.

	* sql/database-create.sql, sql/database-migrate.txt: Add the
	checkups tables.

	* tbsetup/GNUmakefile.in: Descend into the checkup directory.

	* tbsetup/checkup: The checkup daemon, man page, and
	  associated scripts.

	* tbsetup/ptopgen.in: Add a feature with a value of 0.9 to
	  prereserved nodes to keep them from being allocated unless
	  they're really wanted.

	* utils/firstuser.in: Add some other options so the script can be
	  used to create other pseudo users.

Updates to to Error Logging API Code.

You should start seeing much better error messages coming from my
system.  Errors coming from parse.proxy and assign (the two most
frequent sources of errors) should now be concise and to the point.
Errors coming from libosload/libreboot (the next most frequent source
of errors) should now also be much better, but not perfect.  Getting
perfect errors will likely a rework of how errors are handled in
libosload/libreboot, just adding tberror/tbwarn/tbnotice calls is not
enough.  I can do this at a latter date if necessary.

A few minor database changes.

Some changes to the API.  A few bug fixes. Lots of tberror/tbwarn/tbnotice
added to scripts.

Since assign is a C program, and at this time my API is perl only, I wrote a
second wrapper around assign, assign_wrapper2.  When assign fails errors are
now parsed in assign_wrapper2, sent to stderr and logged.  This means that
RunAssign() just returns when assign fails rather than echoing some of
assign.log output and then quiting.  The output to the activity log remains
unchanged.

Since "parse.proxy" is run from ops I couldn't use my API in it, even though
it is a perl program.  Instead I parse the errors coming form it in
parse-ns.

The revived Plab interface is here!

Lots of updates to the plab backend, including improved plab <-> elab node
id translation and update handling.  Includes support for the current PLC
API, and the new pl_conf node manager interface API.  Several more db library
routines were ported from the perl library to the python one to support the
new code (mostly the node_id tracking stuff).  Fixes to the client side and
also a rootball creation cleanup (binaries removed from the CVS repo).

There are also enhancements to the experiment view page for experiments
including plab nodes: site and widearea hostname are now displayed along
with the other node information.

Note that the way setup timeout for vnodes is calculated has been changed a
bit.  Instead of using a hardwired base timeout, the base timeout is now
based on the reload_waittime database field, which comes from the 'OS'
(e.g., FBSD-JAIL, RHL-PLAB) the vnode runs.

The default max duration for a plab slice created through the plab_ez interface
is set to 1 year, and linktest is currently disabled and hidden through
the ez interface.

There is still work to do, but this checkin brings with it a functional
plab portal!

Update SSH_ARGS to remove "Protocol 1" specification.  Everyone needs to
reconfig their object trees!

to use a delay value that is too low.

Make the netlab client applet available to locals.

	* configure, configure.in: Add xmlrpcpipe.php3.

	* xmlrpc/emulabserver.py.in: Add missing virtual_tables.  Add
	getareas call to get the list of robot areas.  Add node.getlist
	and node.typeinfo methods for getting information about the nodes.
	Add a "nic" argument to node.available to get the count of
	wireless nodes.  Add "exclude" argument to
	experiment.virtual_topology so we don't have to download the
	massive route table, also delete the 'pid'/'eid' fields for the
	same reason.  Don't return string output of virtual_topology, it's
	huge.  Return some more info in experiment.getlist().

	* www/GNUmakefile.in: Add xmlrpcpipe.php3.

	* www/beginexp_form.php3, www/modifyexp.php3: Add links to the
	client gui.

	* www/netlab-client.jar: The client binary.

Install NFS tracing stuff so we can gather some data.

	* configure, configure.in: Add some nfstrace files.

	* sensors/GNUmakefile.in: Add nfstrace directory to the build.

	* sensors/nfstrace/GNUmakefile.in, nfs.sql, nfsdump2db,
	nfstrace.init.in, nfstrace.proxy: Prototype of scripts to trace
	NFS traffic from experimental nodes and build a list of the files
	that are accessed.

	* sensors/nfstrace/nfsdump2/*: Slightly customized version of the
	nfsdump program.

	* tbsetup/GNUmakefile.in: Install nfstrace control script which is
	used to talk to the proxy on ops.

* Add libadminmfs.pm with routines for entering/exiting and executing
  commands in, the admin MFS.  Node admin and firewall swapout (see
  below) now use this, the image creation process does not yet.

* Add swapout time hooks for running an admin mode process, likely to
  be used to collect swapout time state.  Currently controlled globally
  by two new sitevars.

* Modified node_admin to use the library and added a "-c <command>"
  option to have nodes go into admin mode and run a command.  I don't
  really expect this to be useful, it was just a testing vehicle for
  the library.

2. Improved the swapout process for firewalled experiments.  Largely
   just generalized what we already did for paniced experiments.
   At swapout, firewalled nodes are:

   - powered off
   - set to boot into admin mode and run a disk zapper
   - powered on

  The swapout process then waits for all nodes to successfully complete
  disk zapage, at which point the nodes are nfree'ed as usual.  Any
  failure of the above process, marks the experiment as panic'ed (to
  ensure that we are involved in cleanup) and sends mail to testbed-ops
  describing the state of the nodes.

3. Added the aforementioned disk zapper, a little C program in the MFS
   which zeroes out the MBR and partition boot blocks (but not the MBR
   partition table or FS superblocks).  This is added insurance that if
   a node somehow gets diverted after being nfree'd but before getting
   the disk reloaded (e.g., goes to hwdown), that we cannot accidentally
   boot from the disk.  This program gets installed in the admin MFS.

4. Related to firewalls, modified swapin to use the new documented
   "snmpit -N" to get the firewall VLAN number rather than parsing the
   output that was a side-effect of VLAN creation.

remote site (currently just www.datapository.net) wants to verify the
user (log the user in), it redirects the user to:

	https://www.emulab.net/login_redirect.php?redirect_to=http://www.datapository.net/login.rhtml

Emulab verifies the user, and then issues an ssh to the host in the
URL, giving it the uid and a freshly generated hash string. Emulab
then redirects the users browser back over to the redirect_to URL,
appending the user name and the hash.

	http://www.datapository.net/login.rhtml?user=$uid&auth='$hash'

What the remote site does with all of this is up to that site.

emulab version of iperf.

Linux makefile stuff not working yet.

New node_attributes facility and table.

Auxiliary node attributes, such as service tag #, BIOS version, etc., are
should now be placed into the node_attributes table.  This can be accomplished
by either using the node_attributes command line tool, or by using the
modnodeattributes_form.php3 form (not linked in anywhere yet, but will be
in a moment).  Attribute names and values are checked for sanity using
table_regex entries.  Also note that I started with the nodecontrol stuff
as a template.

The command line tool and web form (which simply calls the command line tool
to actually do the modifications) can add, delete, and/or remove attributes.

Finally, note that the bios_version column has been moved from the nodes
table to the node_attributes table.  The Node Information page will show
the list of current attributes at the bottom of the info table.

Ready for local people to play with.

The current implementation is that we munge the mysql DB on ops directly,
underneath jabberd. We add/del users from the authreg table, and set up
buddy lists in the roster-items and roster-groups tables. modgroups will
invoke the modjabberbuddies whenever a user is added or removed from a
group, although currently I am building buddy lists for just the top level
projects.

The "My IM" link in the collaboration menu will tell the user their
jabber ID on the Emulab chat server (jabber.emulab.net) and also give
them their plain text password to plug into their chat client.

I also installed a java applet (Jeti) that is a simple chat client that
I found off the jabberware page. Like all applets, it exhibits a degree
of flakiness, but I really do not expect too many people to use it.

Entailed new instructions for manual setup as well as integration into
elabinelab framework.  First, the manual path:

setup.txt, setup-boss.txt, setup-ops.txt and new setup-fs.txt:
    Updated to reflect potential for separate fs node.  The org here
    is a little dicey and could be confusing with ops+fs vs. ops and fs.
    Has not been field tested yet.

*/GNUmakefile.in: new fs-install target.

configure, configure.in, defs-*:
    Somewhat unrelated, make min uid/gid to use be a defs setting.
    Also add config of fs-install.in script.

boss-install.in, ops-install.in and new fs-install.in:
    Handle distinct fs node.  If you have one, fs-install is run before
    ops-install.  All scripts rely on the defs file settings of FSNODE
    and USERNODE to determine if the fs node is seperate.

utils/checkquota.in:
    Just return "ok" if quotas are not used (i.e., if defs file FS_WITH_QUOTA
    string is null.

install/ports/emulab-fs:
    Meta port for fs node specific stuff.  Also a patch for the samba port
    Makefile so it doesn't drag in CUPs, etc.  Note that the current samba
    port Makefile has this change, I am just backporting to our version.

Elabinelab specific changes:

elabinelab-withfs.ns:
    NS fragment used in conjunction with
	tb-elab-in-elab-topology "withfs"
    to setup inner-elab with fs node.

elabinelab.ns:
    The hard work on the boss side.  Recognize seperate-fs config and handle
    running of rc.mkelab on that node.  fs setup happens before ops setup.

rc.mkelab:
    The hard work on the client side.  Recognize FsNode setup as well as
    differentiate ops+fs from ops setup.

Related stuff either not part of the repo or checked in previously:
    emulab-fs package

XMLRPC server on boss.

Bring linktest in line with the new event system world order and do some
generic cleanup.  I'm pretty sure I broke backwards compatibility with
old disk images though...

	* configure, configure.in: Remove linktest.h.in from the list of
	template files.

	* event/linktest/GNUmakefile.in: Don't make subdirectories for
	executables since it can mess up the dependencies.  Don't install
	tb_compat.tcl stuff anymore.  Check for '/usr/local/bin/pathrate'
	and friends, and print out a warning if they are not found.  Make
	a version file for the linktest daemon.

	* event/linktest/linktest.c: Only subscribe to the events we care
	about 'STOP/KILL'.  Subscribe to TIME START events and kill any
	linktests that are in progress.  Send events through the local
	elvind instead of running tevc.

	* event/linktest/linktest.pl.in: Don't run the modified NS anymore
	to generate the topology file, use gentopofile to do that.  Send
	COMPLETE events to indicate that the tests were finished instead
	of STOP so it works with event-sequences.  Use emulab-sync to
	report errors between linktest daemons.  In addition to sending
	REPORT events, send LOG events to the SIMULATOR agent so the
	messages end up in the report e-mail.  Fix wait_all so it collects
	child processes immediately and doesn't leave zombies lying
	around.  Make the static route test work with topologies that
	aren't fully connected.  Change ownership of the linktest error
	files from root to the swapper.  Bump latency tolerance up to
	2.5ms.

	* event/linktest/ltevent.c: Use event_schedule instead of
	event_notify so the events go through the scheduler.

	* event/linktest/run_linktest.pl.in: Wait for COMPLETE events
	insetad of STOPs.

	* event/linktest/linktest.h.in: Removed.

	* event/sched/error-record.c: Include linktest error output in the
	report e-mail.

	* event/sched/event-sched.c: Linktest now sends COMPLETE events
	for START events.

	* tbsetup/gentopofile.in, tmcd/common/config/rc.topomap: Piggy
	back the generation and management of the linktest config file
	onto the generation of the topomap.

	* tbsetup/ns2ir/sim.tcl.in: Add "linktest" event to the Simulator
	object that runs linktest at level 3 (latency, static routing, and
	loss).

	* tmcd/common/libsetup.pm: Add TMLTMAP constant that contains the
	path to the node local linktest config file.

	* tmcd/common/rc.linktest: Pass '-u' option to the linktest daemon
	so it knows what user should to change ownership of files to.

	* tmcd/freebsd/jail/mkjail.pl: Copy the linktest config file
	(ltmap) into the jail.

Jay has "comments"), but I do not want it hanging around in my source
tree. Here is my mail message:

* The "My Mailing Lists" is context sensitive (copied from Tim's
  changes to the My Bug Databases). It takes you to the *archives* for
  the current project (or subgroup) list. Or it takes you to your
  first joined project.

* The showproject and showgroup pages have direct links to the project
  and group specific archives. If you are in reddot mode, you also
  get a link to the admin page for the list. Note that project and
  group leaders are just plain members of these lists.

* The interface to create a new "user" list is:

	https://www.emulab.net/dev/stoller/newmmlist.php3

  We do not store the password, but just fire it over in the list
  creation process.

  Anyone can create their own mailing lists. They are not associated
  with projects, but just the person creating the list. That person
  is the list administrator and is given permission to access the
  configuration page.

  This page is not hooked in yet; not sure where.

* Once you have your own lists, you user profile page includes a link
  in the sub menu: Show Mailman Lists. From this page you can delete
  lists, zap to the admin page, or change the admin password (which is
  really just a subpage of the admin page).

* As usual, in reddot mode you can mess with anyone else's mailman lists,
  (via the magic of mailman cookies).

* Note on cross machine login. The mailman stuff has a really easy way
  to generate the right kind of cookie to give users access. You can
  generate a cookie to give user access, or to the admin interface for
  a list (a different cookie). Behind the scenes, I ssh over and get
  the cookie, and set it in the user's browser from boss. When the
  browser is redirected over to ops, that cookie goes along and gives
  the user the requested access. No passwords need be sent around,
  since we do the authentication ourselves.

the repo in case I decide to stay in Switzerland, eating chocolate and
cheeseballs.

quota check, and parses the output. Any over limit causes the script
to exit with non-zero status.

Not hooked in yet ...

enabled in the defs file:

	CVSSUPPORT=1

each project gets a stub CVS tree created (using 'cvs init') in
/proj/$pid/CVS. It is up to users obviously to do something with
that tree, and of course they have to either set their CVSROOT
env variable, or use the -d option to cvs.

The showproject page gets a link to the per-project CVS tree, using
the cvsweb interface, which I hacked up a bit to allow restricted
access to specific project trees, via a ?pid=$pid argument to the URL.
Without the ?pid argument, it falls back to normal behaviour, which is
check the cvsallowed bit in the users table, and provide access to the
Emulab source repo.

If you are curious, go here:

	https://www.emulab.net/cvsweb/cvsweb.php3/?pid=testbed

Initial checkin of a "repositioning" daemon that moves robots back to
their pens on swapout.

	* configure, configure.in: Add tbsetup/repos_daemon.

	* db/libdb.pm.in: Add constants for the
	repositionpending/repositioning experiments.

	* db/nfree.in: When freeing garcias, send them to
	repositionpending instead of reloadpending.

	* event/sched/event-sched.c: Deal with the rare case of no
	SIMULATOR object being in the agent list for an experiment.

	* robots/emc/emcd.c, robots/emc/locpiper.in: Fix some typos.

	* robots/rmcd/masterController.h, robots/rmcd/masterController.c,
	robots/rmcd/obstacles.h, robots/rmcd/obstacles.c: Ignore dynamic
	obstacles that are far away and remove dynamic obstacles where the
	robot is inside the natural obstacle area.

	* sql/database-create.sql, sql/database-migrate.txt: Add a
	reposition_status table that tracks the status of robots that are
	being moved back to their pens.

	* tbsetup/GNUmakefile.in: Install the repos_daemon script.

	* tbsetup/reload_daemon.in: Move robots to the repositionpending
	experiment, if they haven't already reached their pen.

	* tbsetup/repos_daemon.in: Daemon that takes care of seeing robots
	back to their pens after they are freed from an experiment.