GitLab

Jay has "comments"), but I do not want it hanging around in my source
tree. Here is my mail message:

* The "My Mailing Lists" is context sensitive (copied from Tim's
  changes to the My Bug Databases). It takes you to the *archives* for
  the current project (or subgroup) list. Or it takes you to your
  first joined project.

* The showproject and showgroup pages have direct links to the project
  and group specific archives. If you are in reddot mode, you also
  get a link to the admin page for the list. Note that project and
  group leaders are just plain members of these lists.

* The interface to create a new "user" list is:

	https://www.emulab.net/dev/stoller/newmmlist.php3

  We do not store the password, but just fire it over in the list
  creation process.

  Anyone can create their own mailing lists. They are not associated
  with projects, but just the person creating the list. That person
  is the list administrator and is given permission to access the
  configuration page.

  This page is not hooked in yet; not sure where.

* Once you have your own lists, you user profile page includes a link
  in the sub menu: Show Mailman Lists. From this page you can delete
  lists, zap to the admin page, or change the admin password (which is
  really just a subpage of the admin page).

* As usual, in reddot mode you can mess with anyone else's mailman lists,
  (via the magic of mailman cookies).

* Note on cross machine login. The mailman stuff has a really easy way
  to generate the right kind of cookie to give users access. You can
  generate a cookie to give user access, or to the admin interface for
  a list (a different cookie). Behind the scenes, I ssh over and get
  the cookie, and set it in the user's browser from boss. When the
  browser is redirected over to ops, that cookie goes along and gives
  the user the requested access. No passwords need be sent around,
  since we do the authentication ourselves.

the repo in case I decide to stay in Switzerland, eating chocolate and
cheeseballs.

quota check, and parses the output. Any over limit causes the script
to exit with non-zero status.

Not hooked in yet ...

enabled in the defs file:

	CVSSUPPORT=1

each project gets a stub CVS tree created (using 'cvs init') in
/proj/$pid/CVS. It is up to users obviously to do something with
that tree, and of course they have to either set their CVSROOT
env variable, or use the -d option to cvs.

The showproject page gets a link to the per-project CVS tree, using
the cvsweb interface, which I hacked up a bit to allow restricted
access to specific project trees, via a ?pid=$pid argument to the URL.
Without the ?pid argument, it falls back to normal behaviour, which is
check the cvsallowed bit in the users table, and provide access to the
Emulab source repo.

If you are curious, go here:

	https://www.emulab.net/cvsweb/cvsweb.php3/?pid=testbed

Initial checkin of a "repositioning" daemon that moves robots back to
their pens on swapout.

	* configure, configure.in: Add tbsetup/repos_daemon.

	* db/libdb.pm.in: Add constants for the
	repositionpending/repositioning experiments.

	* db/nfree.in: When freeing garcias, send them to
	repositionpending instead of reloadpending.

	* event/sched/event-sched.c: Deal with the rare case of no
	SIMULATOR object being in the agent list for an experiment.

	* robots/emc/emcd.c, robots/emc/locpiper.in: Fix some typos.

	* robots/rmcd/masterController.h, robots/rmcd/masterController.c,
	robots/rmcd/obstacles.h, robots/rmcd/obstacles.c: Ignore dynamic
	obstacles that are far away and remove dynamic obstacles where the
	robot is inside the natural obstacle area.

	* sql/database-create.sql, sql/database-migrate.txt: Add a
	reposition_status table that tracks the status of robots that are
	being moved back to their pens.

	* tbsetup/GNUmakefile.in: Install the repos_daemon script.

	* tbsetup/reload_daemon.in: Move robots to the repositionpending
	experiment, if they haven't already reached their pen.

	* tbsetup/repos_daemon.in: Daemon that takes care of seeing robots
	back to their pens after they are freed from an experiment.

I fixed a couple of minor problems, but mostly this worked fine. Note that
I have tested this with the installed perl, *NOT* perl 5.8. I am just
making sure this stuff gets committed before too much more bitrot sets in.

in the defs file, so that it is easier to export a default value.
The local defs-default turns it on for all of us, while the default in
configure.in is off. It is currently off by default in defs-example.

I have left the --enable-windows and --disable-windows arguments for
backwards compatability; they just override whatever the defs file
says but otherwise have no effect if not specified.

Mote and robot related stuff.  The main thing is the addition of relay
capabilities to capture and related things.

	* GNUmakefile.in: Add the capture and tip subdirectories to the
	client and client-install targets.

	* configure, configure.in, config.h.in: Detect srandomdev() for
	capture and add "mote/newmote" script.

	* capture/GNUmakefile.in, capture/capture.c: Add "relay"
	capabilities to capture.

	* capture/capquery.c: Query the capserver for the relay receiver's
	port number.

	* capture/capserver.c: Small hack to return the port number
        for a node.

	* db/libdb.pm.in, db/xmlconvert.in: Add virt_tiptunnels table.

	* event/program-agent/program-agent.c: Change log file names to
	something a little more user-friendly.  Add a "MODIFY" event
	handler that lets the user set agent attributes (command, tag,
	timeout) without having to run a program.

	* event/sched/GNUmakefile.in, event/sched/console-agent.cc,
	event/sched/console-agent.h, event/sched/event-sched.c: Add
	console agents that can be used to snapshot a section of the
	capture log file.

	* event/sched/node-agent.cc: Some minor cleanup.

	* event/sched/simulator-agent.cc, event/sched/simulator-agent.h:
	Add the config data to the report mail.  Add a "RESET" event
	handler that runs "loghole clean".  Save the report mail in a file
	so it gets archived with the rest of the logs.

	* lib/libtb/tbdefs.h: Add CONSOLE object type.

	* mote/GNUmakefile.in, mote/newmote: Add newmote script, just a
	quick hack to add motes to the DB.

	* mote/tbuisp.in: Add another backend for loading motes through
	their relay capture server.

	* robots/mtp/mtp_dump.c: Dump the min/max values for x and y,
	handy for figuring out the bounds of the camera.

	* sql/database-fill.sql: Change the RELOAD-MOTE/SHUTDOWN ->
	ALWAYSUP/SHUTDOWN mode transition to ALWAYSUP/ISUP since stated
	doesn't seem to run triggers after a state change by a mode
	transition.

	* tbsetup/tbreport.in: Change the ordering of the eventlist so it
	displays event-sequences appropriately.

	* tbsetup/ns2ir/GNUmakefile.in, tbsetup/ns2ir/console.tcl,
	tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/parse.tcl.in,
	tbsetup/ns2ir/sim.tcl.in: Add a "console" agent that represents
	the serial console for a node.

	* tbsetup/ns2ir/sequence.tcl: Add an "append" method so it is
	easier to build sequences dynamically.

	* tbsetup/ns2ir/topography.tcl: Make checkdest available to
	regular users.

	* tip/GNUmakefile.in, tip/tiptunnel.c: Add support for uploading a
	file to a relay version of capture and exporting the end
	connection as a pty.

	* tmcd/decls.h, tmcd/common/libsetup.pm: Bump version number since
	the dosubnodelist change is not backwards compatible.

	* tmcd/tmcd.c: Make dosubnodelist and dosubconfig callable even
	when a node isn't allocated.  Add dotiptunnels command that
	returns which serial consoles are to be mounted on a node.  Add
	mote version of subconfig that returns information needed to
	startup the relay version of capture.

	* tmcd/common/bootsubnodes: For motes, startup the relay version
	of capture (XXX stargate specific).

	* tmcd/common/libsetup.pm, tmcd/common/libtmcc.pm,
	tmcd/common/config/rc.config, tmcd/common/config/rc.tiptunnels:
	Client side changes for mounting another nodes serial line.

	* tmcd/common/rc.bootsetup: Always boot the subnodes, even when
	free.  This is used for motes since their capture needs to be up
	for reloading at the time.

	* tmcd/linux/ixpboot: Shuffle some code around so the script
	doesn't fail if the ixp isn't allocated.

	* utils/loghole.in: Add "digest.out" and "report.mail" as global
	logs to be saved in archives and display the "report.mail" file
	when showing a loghole archive.

	* xmlrpc/emulabserver.py.in: Scrub more of the return values to
	get rid of "None"s.

grabswitchconfig, suitable for daily running from cron.

and stashes it in a local file.

Wiki support is turned on in the defs file with a WIKISUPPORT=1 directive.

Sends wake-on-lan packets to a speicifc MAC address in order to
trigger our custom whack-on-lan hardware.

Checkin some changes related to experiment automation and vnode feedback:

	* configure, configure.in: Add sensors/canaryd/feedbacklogs
	template.

	* db/libdb.pm.in, db/xmlconvert.in: Add "virt_user_environment"
	table that holds environment variable names and values.

	* event/lib/event.c: Allocate memory of the right size for
	event_notifications.

	* event/program-agent/GNUmakefile.in: Add version.c file and
	add install targets for the man page.

	* event/program-agent/program-agent.8: Man page describing the
	program-agent daemon.

	* event/program-agent/program-agent.c: Add a bunch of convenience
	features: let the user specify the working directory for commands;
	save output to separate files on every invocation of an agent; let
	the user specify a timeout for a command; make the set of
	environment variables sane and add vars given in the NS file in
	the opt array; a "status" file containing process information is
	written out when children are collected.  Internal changes: child
	processes are collected immediately, instead of waiting for the
	next START event, so we can send back COMPLETE events; the daemon
	now runs with a real-time priority, to increase the chances of
	receiving events.

	* event/proxy/evproxy.c: Made it bidirectional so the
	program-agent's COMPLETE events make it back to the scheduler.

	* event/sched/error-record.c: Change the default log directory.

	* event/sched/event-sched.h, event/sched/event-sched.c: Setup an
	environment similar to a program-agent to run the user's log
	digester.

	* event/sched/node-agent.cc: Add a handler for the SNAPSHOT event
	that runs create_image for the node.

	* event/sched/simulator-agent.h, event/sched/simulator-agent.cc:
	Let the user specify a "DIGESTER" script that digests the log
	files into a summary of the results.  Add event handler for
	remapping a vnode experiment.

	* event/sched/timeline-agent.c: Accept the RUN event as well as
	the START event.

	* os/GNUmakefile.in: Install the install-tarfile.1 man page.

	* os/install-tarfile: Automatically chown/chgrp any files that do
	not have valid user or group IDs, the new owner will be the user
	that swapped in the experiment.  Include the install directory in
	the DB file.  Add a "list" mode that just dumps what files have
	been installed and where.  Add a "force" option so the user can
	forcefully install the file, even though the DB says its already
	there.

	* os/install-tarfile.1: Man page describing the install-tarfile
	tool.

	* os/syncd/GNUmakefile.in: Install man pages on ops.

	* sensors/canaryd/GNUmakefile.in: Link canaryd statically and
	install "feedbacklogs" tool.

	* sensors/canaryd/canaryd.c: Dump dummynet pipe data.

	* sensors/canaryd/canarydEvents.c: Log errors.

	* sensors/canaryd/feedbacklogs.in: Tool used to generate feedback
	data from canaryd log files.

	* sensors/slothd/GNUmakefile.in: Install digest-slothd on ops.

	* sensors/slothd/digest-slothd: Fix some bugs and write out an
	"alert" file with all the nodes/links that were overloaded.

	* tbsetup/os_load.in, tbsetup/libosload.pm.in: Add "waitmode"
	argument that lets you specify that you want to wait for the disk
	to finish loading and/or wait for the node to come back up in the
	new OS.

	* tbsetup/power.in: Remove debugging printf.

	* tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/program.tcl,
	tbsetup/ns2ir/sequence.tcl, tbsetup/ns2ir/sim.tcl.in: Fix some
	quoting problems with event-sequences.  Add -expected-exit-code
	and -tag options to the "$program run" event.  Add -digester to
	the "$ns report" event that lets the user specify a program to run
	to digest the log files.

	* tbsetup/ns2ir/tb_compat.tcl.in: Change the initial scaling
	factor for feedback nodes to 1%, instead of 100%.

	* tmcd/tmcd.c, tmcd/common/libtmcc.pm: Add "userenv" command that
	returns the values in "virt_user_environment".  Return new program
	agent fields: dir, timeout, and expected_exit_code.

	* tmcd/common/GNUmakefile.in: Install rc.canaryd.

	* tmcd/common/bootvnodes: Add hack to boost the program-agents to
	a real-time priority, they can't do it from inside the jail.

	* tmcd/common/rc.canaryd: Rc script for canaryd.

	* tmcd/common/watchdog: Don't fail outright if there is a bad line
	in the battery.log

	* tmcd/common/rc.progagent: Append "userenv" data to the
	program-agent config file.

	* utils/GNUmakefile.in: Install loghole and its man page on ops.

	* utils/loghole.1: Document "clean" command and the change in
	loghole directories.

	* utils/loghole.in: Add "clean" command and parallelization.

	* xmlrpc/emulabserver.py.in: Add "virt_user_environment" table.
	Order the eventlist by "idx" and time, needed for sequences.  And
	removed unnecessary nologin checks.

burgeoning Java Applet skills to to write a little applet to decode
the motion jpeg stream that the webcams spit out. Not exactly
efficient, but at least it works everyplace I tried, even with
Internet Explorer.

For the web page, the default view is still static, with an option
near the top to use the applet version of the page. The nice thing is
that when you leave the page (go someplace else) the data stream
actually stops (cause the applet is stopped).

append ".exe" in favor of the --enable-windowsclient option.

Enable windows support by default now.  This ensures exports_setup creates
samba shares.

* rc.mkelab - modify logic s.t. --disable-windows is passed to configure if
  WINSUPPORT is not set.

Note that the java compiler on boss is too old to compile this applet
(not that I am using anything special) so I am committing a compiled
version. Use the jar target in the robots/tracker build directory if
you happen to have a modern version of java on your machine.

Fixed up ops-install for new way of handling smb.conf file.  We copy in an
smb.conf.head file now, and copy that to smb.conf initially on ops.
exports_setup[.proxy] takes care of updating the share permissions for
samba dynamically.

directory.

The python.org Python distribution has the executable at the top level, not in a bin subdir, with include below it.

Support for reloading garcias.

Currently this is a total hack; we simply rsync the stargate as it is freed
from the experiment.  However, until we unravel os_load dependancies between
nodes and subnodes (motes and stargates in this case), doing this through
the appropriate setup channels won't work.

The tbrsync script borrows much of its infrastructure from Rob's tbuisp
script.

can avoid building robots/tbsetdest.

Stargate client-side boot support.

Similar to linux, but the sg linux distribution is based on debian, so some
things are a little different.

WARNING: if you do a client-install in a cross-compile environment
(with DESTDIR set), watch out for bogus symlinks in <root>/etc/rc*.d!
Have to fix these up by hand in this case.

Also note that the stargate rc script doesn't yet start the robot pilot
daemon - Tim said he needed to do finalize some stuff w.r.t. this, so
he would take care of modifying the rc script.

The ntp settings are still not ideal.  Even though the root fs is on a
journalling jffs filesys, its still not good to write to it regularly.  Need
to point the ntp drift file off into /var (ramfs)

sitevars to determine if the Robot Lab is open or closed. The sitevars:

* 'robotlab/override' - Override other settings and forcibly turn the lab
  "on" or "off" (open or close). When the lab is turned off, new
  experiments cannot swap in and the current experiment is immediately
  swapped out.

* 'robotlab/exclusive' - The robot lab is exclusive use. Best to not mess
  with this sitevar :-)

* 'robotlab/opentime' - The time that the robot lab opens in the
  morning. The default is 07:00, but feel free to change this as you like.

* 'robotlab/closetime' - The time that the robot lab closes in the
  evening. The default is 18:00, but feel free to change this as you like.

* 'robotlab/open' - The robot lab is open or closed. DO NOT MESS WITH THIS!
  It is updated by the robomonitord script and intended to be used by
  admission control (not done yet).

The robomonitord script runs and periodically (every 2 minutes) wakes up
and looks at the various sitevars above. The lab is open during the day,
Monday through Friday, and closed on weekends. It is also supposed to be
closed on holidays, but I have not added that yet.

15 minutes before the lab is to be closed, a warning message is sent to the
swapper of the experiment running on the robot testbed, that their
experiment is going to be swapped soon. When the Robot lab is closed
(either cause the close time was reached, or because the lab was forcibly
closed with the override), the current experiment is immediately swapped
out.

I know, this is hopelessly bogus, but it will do until we feel like adding
a "Lab" datatype to the system.

Add a Samba template config file.  The listening interface, globally allowed
access list, and workgroup are defined as follows:

   workgroup = @OURDOMAIN@
   hosts allow = @CONTROL_NETWORK@/@CONTROL_NETMASK@
   interfaces = @FSNODE_IP@

I believe this is what we want, but am not sure about the workgroup setting.
Note that this should ultimately just be the head of the smb.conf file, with
the tail generated vis-a-vis /etc/exports .  For now it has some hard wired
shares for /proj and /users, which is OK for the very short term on an
inner elab.

Robot related stuff: power via e-mail, client-install fixups, checking
coords against camera boundaries.

	* configure, configure.in: Add tbsetup/power_mail.pm to the list
	of template files.

	* doc/cross-compiling.txt: More stargate notes.

	* event/sched/rpc.cc: Updates for the addition of the cameras
	table.

	* robots/GNUmakefile.in, robots/emc/GNUmakefile.in,
	robots/mtp/GNUmakefile.in, robots/rmcd/GNUmakefile.in,
	robots/tbsetdest/GNUmakefile.in, robots/vmcd/GNUmakefile.in:
	client-install fixups.

	* tbsetup/GNUmakefile.in: Add power_mail.pm.

	* tbsetup/os_setup.in: Don't skip reboot of robots anymore.

	* tbsetup/power.in: Add special case for a power_id of "mail",
	which calls into the power_mail.pm backend.

	* tbsetup/power_mail.pm.in: E-mail backend for power, it sends an
	e-mail to tbops and waits for the outlets.last_power value to be
	updated from the power.php3 web page.

	* tbsetup/ns2ir/parse-ns.in: Add the contents of the cameras table
	to the TBCOMPAT namespace.

	* tbsetup/ns2ir/sim.tcl.in: More checking of "setdest" inputs.

	* tbsetup/ns2ir/topography.tcl: Update the checkdest method to
	check destination points against the camera list.

	* www/powertime.php3: Webpage used to update the last power time
	for nodes.

	* www/shownode.php3: Add "Update Power Time" menu button.

table by hand anymore! Update the group_policies table and then run
the script to update the permissions table (sbin/update_permissions).

Details:

My original thought when I started this was that I would be able to
replace the existing nodetypeXpid_permissions table with this new
stuff. Well, it turns out that this was not a good thing to do, for a
couple of reasons:

  * Engineering: We access the nodetypeXpid_permissions table from three
    different languages, and no way I wanted to rewrite this library in
    in python and php!

  * Performance: We access the nodetypeXpid_permissions from the web
    interface, on every single page load. In fact, we access it twice if
    if you count the FreePCs() count that we put at the top of the menu.
    Going through this library on each page load would be a serious drag.

So, rather then actually get rid of the nodetypeXpid_permissions table, I
decided to keep it as a "cache" of permissions stored in the group
policies table. Each time you update the policy tables, we need to run
the update_permissions script which will call into this library (see the
TBUpdateNodeTypeXpidPermissions() routine) to reconstruct the permissions
table. I have whacked the grantnodetype script to do exactly that.

Note that we could proably do the same thing for users by creating an
equivalent nodetypeXuid_permissions table, mapping users to types they
are allowed to use. That would be a lot rows, but the amount of data in
the table is small. That would give us very fine grained control of what
we show people in the web interface. Not sure it is worth it though.

I also added some instructions to previous commit in database-migrate.txt
on populating the new group_policies table from the existing
permissions table.

Add tbsetdest utility for generating random "setdest" events for the
robots, it's just based on the original NS version.

	* configure, configure.in: Add tbsetdest stuff to the list of
	template files.  Fix from my last checkin, change error for
	missing rsync to a warning.

	* robots/GNUmakefile.in: Add tbsetdest subdir.

	* robots/tbsetdest/GNUmakefile.in, robots/tbsetdest/tbsetdest.h,
	robots/tbsetdest.cc: Original version of the NS "setdest"
	utility.