GitLab

enabled in the defs file:

	CVSSUPPORT=1

each project gets a stub CVS tree created (using 'cvs init') in
/proj/$pid/CVS. It is up to users obviously to do something with
that tree, and of course they have to either set their CVSROOT
env variable, or use the -d option to cvs.

The showproject page gets a link to the per-project CVS tree, using
the cvsweb interface, which I hacked up a bit to allow restricted
access to specific project trees, via a ?pid=$pid argument to the URL.
Without the ?pid argument, it falls back to normal behaviour, which is
check the cvsallowed bit in the users table, and provide access to the
Emulab source repo.

If you are curious, go here:

	https://www.emulab.net/cvsweb/cvsweb.php3/?pid=testbed

some details can be found in the advanced tutorial that I wrote up.
See this link:

http://www.emulab.net/tutorial/docwrapper.php3?docname=advanced.html#Tracing

The basic idea is that each virt_lan entry gets a couple of new slots
describing the type of tracing that is desired.

  traced tinyint(1) default '0',
  trace_type enum('header','packet','monitor') NOT NULL default 'header',
  trace_expr tinytext,
  trace_snaplen int(11) NOT NULL default '0',
  trace_endnode tinyint(1) NOT NULL default '0',

There is a new physical table called "traces" that is a little bit
like the current delays table. A new tmcd command returns the trace
configuration to the client nodes (tmcd/common/config/rc.trace).

The delays table got a new boolean called "noshaping" that tells the
delay node to bridge, but not set up any pipes. This allows us to
capture traffic at the delay node, but without much less overhead on
the packets.

The pcapper got bloated up to do packet capture and more event stuff.
I also had to add some mutex locking around calls into the pcap
library and around malloc, since the current setup used linuxthreads,
which is not compatable with the standard libc_r library. I was
getting all kinds of memory corruption, and I am sure that if someone
breathes on the pcapper again, it will break in some new way.

tree we keep on boss (which is update each night). Still need to
arrange for tmcd to pass back CVSSRCTAG=xyz in the emulabconfig
request, but if it did, that tag is passed along to the web interface,
which calls new backend script to do the checkout, tar it up, gzip
it, and spit to stdout.

that picked up single images with scp, and replaced it with stuff that
can be used to capture movies.

	Usage: grabwebcams [-d] [-v] [-m] [-t timeout]
	       grabwebcams [-d] [-v] [-t timeout] [-k] pid eid
	switches and arguments:
	-d      - Debug mode, use to prevent daemonization
	-v      - Verbose mode (causes vlc to spit lots of goo)
	-m      - Movie option; create a 10fps movie from each camera
	-t <N>  - Terminate automatically and N seconds
	-k      - Kill a daemonized grabwebcams (only use with pid/eid)
	pid eid - Project and Experiment (for use with swapin)

The first form allows you to capture a low frame rate movie from the
webcams. Low means 2fps. The files are written in the current
directory. Without the -d option, the script goes into the background
and runs forever, unless you give it a -t option, in which case the
movies will be terminated after that many seconds.

Add the -m option, and you get 15fps movies, suitable for showing off
to people. This option IS VERY CPU INTENSIVE. Use it sparingly. It
might get better once I hear back from the axis people, telling me if
there is a way to turn up the mpeg fps via a URL (like most everything
else can be on these cameras). Until then, I am transcoding mjpeg into
mpeg, and that sucks.

The second form is for use from the swapexp path, to start capturing
movies when an experiment is swapped in, and then to then to kill it
off (-k) when the experiment is swapped out. At 2fps, it generates
about 1MB per camera per minute. Thats a lot of data, so I am not
hooking this in quite yet; want to give it some more thought.

We may want to export an interface to the web (and commandline) to
that people can take movies of their experiments at particular times.
Not sure yet.

Mote and robot related stuff.  The main thing is the addition of relay
capabilities to capture and related things.

	* GNUmakefile.in: Add the capture and tip subdirectories to the
	client and client-install targets.

	* configure, configure.in, config.h.in: Detect srandomdev() for
	capture and add "mote/newmote" script.

	* capture/GNUmakefile.in, capture/capture.c: Add "relay"
	capabilities to capture.

	* capture/capquery.c: Query the capserver for the relay receiver's
	port number.

	* capture/capserver.c: Small hack to return the port number
        for a node.

	* db/libdb.pm.in, db/xmlconvert.in: Add virt_tiptunnels table.

	* event/program-agent/program-agent.c: Change log file names to
	something a little more user-friendly.  Add a "MODIFY" event
	handler that lets the user set agent attributes (command, tag,
	timeout) without having to run a program.

	* event/sched/GNUmakefile.in, event/sched/console-agent.cc,
	event/sched/console-agent.h, event/sched/event-sched.c: Add
	console agents that can...

grabswitchconfig, suitable for daily running from cron.

and stashes it in a local file.

Checkin some changes related to experiment automation and vnode feedback:

	* configure, configure.in: Add sensors/canaryd/feedbacklogs
	template.

	* db/libdb.pm.in, db/xmlconvert.in: Add "virt_user_environment"
	table that holds environment variable names and values.

	* event/lib/event.c: Allocate memory of the right size for
	event_notifications.

	* event/program-agent/GNUmakefile.in: Add version.c file and
	add install targets for the man page.

	* event/program-agent/program-agent.8: Man page describing the
	program-agent daemon.

	* event/program-agent/program-agent.c: Add a bunch of convenience
	features: let the user specify the working directory for commands;
	save output to separate files on every invocation of an agent; let
	the user specify a timeout for a command; make the set of
	environment variables sane and add vars given in the NS file in
	the opt array; a "status" file containing process information is
	written out when children are collected.  Internal changes: child
	processes are c...

setdest events for robots. See www/robotrack/setdest.php3

* Add new DB table "webcams" which hold the id of the webcam, the
  server it is attached to, and the last update time.

* Add new sitevars webcam/anyone_can_view and webcam/admins_can_view.
  Should be obvious what they mean.

* Add trivial script grabwebcams (invoked from cron) to grab the images
  from the servers and stash in /usr/testbed/webcams. The images are
  grabbed with scp, protected by a 5 second timeout. Fine for a couple
  of cameras.

* Add web page stuff to display webcams, linked from the robot mape page.

Permission to view the webcams is currently admin, or in a project that is
allowed to use a robot. We can tighten this up later as needed.

Display a new 'Blinky Lights' button on the showexp page. In order to
do this, I have to get a list of which classes/types are in use in
the experiment.

This leads to moteleds.php3, which displays the blink lights using
Tim's cool Java applet.

emulab by waiting for them to reach PXEWAIT. At that point, the inner
emulab is ready to go and we can launch an experiment.

nodes itself.

of all references to a node. Mainly intended for when you have a
mishap with the newnode stuff and need to clean it up.

Added a big list of which tables contain information about physical
nodes to libdb, so that this and other scripts can find it all.

supporting both a shell script driven interface, plus the newer XMLRPC
interface. This change removes the script driven interface from boss,
replacing it with just the XMLRPC interface. Since we like to maintain
backwards compatability with interfaces we have advertised to users (and
which we know are being used), I have implemented a script wrapper that
exports the same interface, but which converts the operations into XMLRPC
requests to the server. This wrapper is written in python and uses our
locally grown xmlrpc-over-ssh library. Like the current "demonstation"
client, you can take this wrapper to your machine that has python and ssh
installed, and use it there; you do not need to use these services from
just users.emulab.net. Other things to note:

* The wrapper is a single python script that has a "class" for each wrapped
  script. Running the wrapper without any arguments will list all of the
  operations it supports. You can invoke the wrapper with the operation as
  its argument:

    {987} stoller$ script_wrapper.py swapexp --help
    swapexp -e pid,eid in|out
    swapexp pid eid in|out
    where:
         -w   - Wait for experiment to finish swapping
         -e   - Project and Experiment ID
         in   - Swap experiment in  (must currently be swapped out)
        out   - Swap experiment out (must currently be swapped in)

    Wrapper Options:
        --help      Display this help message
        --server    Set the server hostname
        --login     Set the login id (defaults to $USER)
        --debug     Turn on semi-useful debugging

   But more convenient is to create a set of symlinks so that you can just
   invoke the operation by its familiar scriptname. This is what I have
   done on users.emulab.net.

    {987} stoller$ /usr/tesbed/bin/swapexp --help
    swapexp -e pid,eid in|out
    swapexp pid eid in|out


* For those of you talking directly to the RPC server from python, I have
  added a wrapper class so that you can issue requests to any of the
  modules from a single connection. Instead using /xmlrpc/modulename, you
  can use just /xmlrpc, and use method names of the form experiment.swapexp,
  node.reboot, etc.

  Tim this should be useful for the netlab client which I think opens up
  multiple ssh connections?

* I have replaced the paperbag shell with a stripped down xmlrpcbag shell
  that is quite a bit simpler since we no longer allow access to anything
  but the RPC server. No interactive mode, no argument processing, no
  directory changing, etc. My main reason for reworking the bag is to make
  it easier to understand, maintain, and verify that it is secure. The new
  bag also logs all connections to syslog (something we should have done in
  the orginal). I also added some setrlimit calls (core, maxcpu). I also
  thought about niceing the server down, but that would put RPC users at a
  disadvantage relative to web interface users. When we switch the web
  interface to use the XMLRPC backend, we can add this (reniceing from the
  web server would be a pain cause of its scattered implementation).

Add wrapper script around dhcpd to restart it if it dies; whenever dhcpd dies, the testbed becomes unusable.

very specific to wireless links in general, and to iwconfig on Redhat
9.0. It allows you to control the entire lan or an individual member
of a wireless lan via the event system. For example to change the
accesspoint of a wireless lan, you could do this:

	tevc -e foo/bar now lan0 modify accesspoint=00:09:5B:93:0B:A4

The agent deciphers the event arguments and calls iwconfig with the
appropriate as needed. Note that there are many ways to make the lan
unusable doing this, so you want to be careful. You can get the MAC
addresses from the experiment info page (tbreport).

New script called link_config, which might be badly named since it
implies generality) to front end tevc. Operates mostly like
delay_config in that it will change the physical table settings, and
optionally (-m) the virtual table entries. So,

	link_config testbed two-wireless lan0 accesspoint=00:09:5B:93:0B:A4

You can change individual members of a lan too:

	link_confi...

night, and process it into a file to stick into the testbed www
directory. This replaces the current approach of wishfully thinking
that people will remember to add their commit log messages to another
file and them install it. I'm one of the worst offenders these days ...

So, take the commit log, skip up to the first entry in 2004/01, then
take the rest of the file and reverse the entries. Copy the resulting
file over to the www directory.

This script is run from cron on our site only!

Note to Opera users; opera does not respect <XMP> properly, and so the
page is somewhat scrogged by HTML entities that are supposed to be
ignored when inside an XMP. Works okay on other browsers.

1. "make clean" will just remove stuff built in the process of a regular build
2. "make distclean" will also clean out configure generated files.

This is how it was always supposed to be, there was just some bitrot.

Grant permission to use types/class of nodes. If given a specific
"type", then grant permission to use just that type. If its a class, then
permission to use all of the types in that class, the class itself, and any
aux nodetypes for the type/class (node_types_auxtypes table).

For example:

	wap grantnodetype -p testbed pc2000
	wap grantnodetype -p testbed pcvm
	wap grantnodetype -p testbed pcplab

Also added webnsgen as a frontend for web pages that need to call
nsgen.

 - fix bad indenting to a uniform 4 spaces (before was 2, 4 and 8 mixed)
 - Move ping-for-isup functionality into a separate script
 - Make sure every transition triggered by stated (directly or indirectly)
   sends an event, instead of taking shortcuts.

This called for a new script, eventping, which just pings until the node
is pingable, then sends an ISUP event. Stated runs this in the background
where necessary, and nothing else should run it.

Adding eventping meant modifying configure and the utils makefile, too.

history).

to an external perl script, and use ssh-keygen to attempt conversion
off SSH2/SECSH key formats. This is actually a simplification of the
php code, which is not generally very good at this kind of thing (or
maybe I mean perl is just better at it). The parsing and error
handling it also much improved.

flips the uid/gid to nobody/nobody. It would be good to run this in a
chroot shell, but that would be difficult given that we cannot easily
rebuild cvsupd (modula-3). The right solution is to either run it in a
jail or to move it to ops.

Note, files in the sup tree obviously have to be world readable for
cvsupd to send them off.

commands to fill a few tables that should be common across most
sites, so that this data can be distributed to them.

The simple single-column table exported_tables controls which tables
are output.

links that do not need to get created in the ops install target.

sshtb also checks for unified control nodes, and if the destination
host is one of them, just runs the command, rather than invoking it
remotely via ssh.

admin privs.  The idea is that you have to be explicit about when you
want to make use of your super powers, to prevent accidents.

Use the new withadminprivs script to get your admin privs, as in:
withadminprivs inuse
or
withadminprivs node_reboot -e testbed,foo

script that checks the database to see if local or remote. The problem
with this is that the ssh syntax makes it hard to determine the host
name by inspection. Would need to parse all the ssh args (bad idea),
ot work backwards and try to figure out the difference between the
command (which is not a string but a sequence of args) and the host
and the preceeding ssh args. Hell with that! Changed sshtb to require
a specific -host argument. Read the args and look for it. Error out of
not found, to catch improper usage.

The moral of this update: "sshtb [ssh args] -host <host> [more args ...]

* tmcd/ron: A new directory of client code, based on the freebsd
  client code, but scaled back to the bare minimum. Does only account
  and group file maintenance. I redid the account stuff so that only
  emulab accounts are operated on. Does not require a stub file, but
  instead keeps a couple of local dbm files recording what groups and
  accounts were added by Emulab. There is a ton of paranoia checking
  to make sure that local accounts are not touched.

  The update script that runs on the client node detaches so that the
  ssh from boss returns immediately. update can also be run from the
  node periodically and at boottime. The script is installed setuid
  root, but checks to make sure that *only* root or "emulabman" has
  invoked it.

* utils/sshremote: New file. For remote nodes, instead of using sshtb,
  use sshremote, which ssh's in as "emulabman", which needs to be a
  local non-root user, but with an authorized_keys file containing
  boss' public key.

* web interface changes: Allow user to specify his own public key in
  addition to the emulab key.

  Add option in showexp page to update accounts on nodes in the
  experiment. I was originally intending to do this from approveuser,
  but this was easier and faster. I will add an option to do it on the
  approveuser page later.

* libdb.pm: Add a TBIsNodeRemote() query to see if a node is in the
  local testbed or a pcRemote node. Currently, this test is hardwired
  to a check for class=pcRemote, but this will need to change to a
  node_types property at some point.

* node_update: Reorg so that there is a maximum number of children
  created. Previously, a child was forked for each node, but that
  could chew up too many processes, especially for remote nodes which
  might hang up. For the same reason, we need to "lock" the experiment
  so that it cannot be terminated while a node_update is in progress.
  Might be to relax that, but this was easy for now. Also add
  distinction between local and remote, since for remote we use
  sshremote insted of sshtb. Various cleanup stuff

* mkacct; When generating a new account, include user supplied pub key
  in the authorized keys file, in addition to the eumlab generated
  key. Both keys are stored in the DB in the users table. Anytime we
  update an account, get a fresh copy of the emulab pub key, in case
  user changes it.

supporting autocreating and autoloading images. The imageid form now
sports a field to specify a nodeid to create the image from; If set,
the backend create_image script is invoked. Thats the easy part.
Slightly harder is autoloading images based on the osid specified in
the NS file. To support this, I have added a new DB table called
osidtoimageid, which holds the mapping from osid/pctype to imageid.
When users create images, they must specify what node types that image
is good for. Obviously, the mappings have to be unique or it would be
impossible to figure it out! Anyway, once that image mapping is
in place and the image created, the user can specify that ID in the NS
file. I've changed os_setup to to look for IDs that are not loaded,
and to try and find one in the osidtoimageid. If found, it invokes
os_load. To keep things running in parallel as much as possible,
os_setup issues all the loads/reboots (could be more than a single set
of loads is multiple IDs are in the NS file) at once, and waits for
all the children to exit. I've hacked up os_load a bit to try and be
more robust in the face of PXE failures, which still happen and are
rather troublsesome. Need an event system!

Contained in this revision are unrelated changed to make the OS and
Image IDs per-project unique instead of globally unique, since thats a
pain for the users. This turns out to be very messy, since underneath
we do not want to pass around pid/ID in all the various places its
used. Rather, I create a globally unique name and extened the OS and
Image tables to include pid/name/ID. The user selects pid/name, and I
create the globally unique ID. For the most part this is invisible
throughout the system, except where we interface with the user, say in
the web pages; the user should see his chosen name where possible, and
the should invoke scripts (os_load, create_image, etc) using his/her
name not the internal ID. Also, in the front end the NS file should
use the user name not the ID. All in all, this accounted for a number
of annoying changes and some special cases that are unavoidable.

testbed boot process and the freebsd pxeboot/MFS kernel. I'm hoping
this will be useful enough for users to load their own disk images,
but we shall see. Usage is simple:

	/usr/testbed/bin/node_admin on|off pcXXX

The "on" state boots the MFS, and the "off" states sets it back to
normal.