Modify the event scheduler to use the BSD-licensed xmlrpc-c library
instead of the LGPL-licensed ulxmlrpcpp.

Delay agent won't build on FreeBSD 8.x right now due to dummynet API changes.
Not even sure we will bother to fix this since we have a newer, more OS
independent agent.

From: Leigh Stoller <stoller@flux.utah.edu>
Date: July 1, 2010 11:35:08 AM MDT
Subject: Re: Event System Issues

Mike and I exchanged a couple of email messages. Mike has indicated
that we should drop all Elvin support. A nice goal, but not possible
cause of the basic mistake I made in "version 0" of the event code.

What we *can* do is stop *generating* the elvin hashes completely in
Version 1. We also drop the elvin_gateway, thus no longer supporting
ancient images that are still using the real elvin libraries. I am
okay with this. Comment if you have objections.

Version 0 elvin-compat (these are pubsub clients with ELVIN_COMPAT=1)
binaries will work cause of the magic ___elvin_ordered___ flag, which
actually tells the client that the hmac is a pubsub ordered hmac. (I
know, I am just great at naming things). I just add this little flag
to events in version 1.

Version 0 non-elvin-compat binaries will not work, but this is okay.
The only case this matters right now is Protogeni, where we need to be
able to talk to non-elvin-compat binaries at remote sites. I have
solved this with a version0 gateway as described in the previous
message. ops will run a secondary pubsubd on another port, and the
protogeni client startup code will have clients connect to that
pubsubd instead. This is mostly tested, and it can roll out to other
sites as needed, once we roll out cooked mode.

Version 1 clients are fully interoperable.

Lastly, we still need to be able to compare elvin HMACs coming from
existing version 0 elvin-compat binaries (from our many many system
and custom images). Thats cause all those images are still going to be
generating the HMACs in elvin order, and so the server programs on ops
(event scheduler, tevc, etc) need that elvin hashing code built into
it. See first paragraph.

Anyway, I have all this done and tested on my elabinelab. I had wanted
to make it in time for code freeze, but not enough time to get proper
debugging, so I will push once code freeze is over.

Lbs

install.

utilities (tevc, linktest, libraries).

will give me a little breathing space (stage things in slower) by
starting just with boss.

so that users can schedule program events to run there. For example:

	set myprog [new Program $ns]
	$myprog set node "ops"
	$myprog set command "/usr/bin/env >& /tmp/foo"

	$ns at 10 "$myprog start"
or
	tevc -e pid/eid now myprog start

Since the program agent cannot talk to tmcd from ops, there are new
routines to create the config files that the program agent uses, in
the expertment tbdata directory.

I also rewrote the eventsys.proxy script that starts the event
scheduler on ops; I rolled the startup of the program agent into this
script, via new -a option which is passed over from boss when an ops
program agent is detected in the virt topology. This keep the number
of new processes on ops to a small number.

Also part of the above rewrite is that we now catch when event
scheduler (or the program agent) exits abnormally, sending email to
tbops and the swapper of the experiment. We have been seeing abnormal
exits of the scheduler and it would good to detect and see if we can
figure out what is going wrong.

Other small bug fixes in experiment run.

ops as the user, but that breaks for admin people who have real shells
on boss and no unencrypted key in the .ssh dir. So, switched it to a
root ssh, and a simple proxy on the other side that flips to the user
and invokes the run_linktest stuff.

* The per-experiment event scheduler now runs on ops instead of boss.
  Boss still runs elvind and uses events internally, but the user part
  of the event system has moved.

* Part of the guts of eventsys_control moved to new script, eventsys.proxy,
  which runs on ops and fires off the event scheduler. The only tricky part
  of this is that the scheduler runs as the user, but killing it has to be
  done as root since a different person might swap out the experiment. So,
  the proxy is a perl wrapper invoked from a root ssh from boss, which
  forks, writes the pid file into /var/run/emulab/evsched/$pid_$eid.pid,
  then flips to the user and execs the event scheduler (which is careful
  not to fork). Obviously, if the kill is done as root, the pid file has to
  be stored someplace the user is not allowed to write.

* The event scheduler has been rewritten to use Tim's C++ interface to the
  sshxmlrpc server on boss. Actually, I reorg'ed the scheduler so that it
  can be built either as a mysql client, or as RPC client. Note that it can
  also be built to use the SSL version of the XMLRPC server, but that will
  not go live until I finish the server stuff up. Also some goo for dealing
  with building the scheduler with C++.

* Changes to several makefiles to install the ops binaries over NFS to
  /usr/testbed/opsdir. Makes life easier, but only if boss and ops are
  running the same OS. For now, using static linking on the event scheduler
  until ops upgraded to same rev as boss.

* All of the event clients got little tweaks for dealing with the new CNAME
  for the event system server (event-sever). Will need to build new images
  at some point. Old images and clients will continue to work cause of an
  inetd hack on boss that uses netcat to transparently redirect elvind
  connections to ops.

* Note that eventdebug needs some explaining. In order to make the inetd
  redirect work, elvind cannot be listening on the standard port. So, the
  boss event system uses an alternate port since there are just a few
  subsystems on boss that use the server, and its easy to propogate changes
  on boss. Anyway, the default for eventdebug is to connect to the standard
  port on localhost, which means it will work as expected on ops, but will
  require -b argument on boss.

* Linktest changes were slightly more involved. No longer run linktest on
  boss when called from the experiment swapin path, but ssh over to ops to
  fire it off. This is done as the user of course, and there are some
  tricks to make it possible to kill a running linktest and its ssh when
  experiment swapin is canceled (or from the command line) by forcing
  allocation of a tty. I will probably revisit this at some point, but I
  did not want to spend a bunch of time on linktest.

* The upgrade path detailed in doc/UPDATING is necessarily complicated and
  bound to cause consternation at remote sites doing an upgrade.

possible to:

	gmake client
	sudo gmake client-install

on a FBSD4, FBSD5, RHL7.3, and RHL9.0 client node.

There are still some dependencies that are not explicit and which would
prevent a build/install from working on a "clean" OS.  Two that I know of are:
you must install our version of the elvin libraries and you must install boost.

very specific to wireless links in general, and to iwconfig on Redhat
9.0. It allows you to control the entire lan or an individual member
of a wireless lan via the event system. For example to change the
accesspoint of a wireless lan, you could do this:

	tevc -e foo/bar now lan0 modify accesspoint=00:09:5B:93:0B:A4

The agent deciphers the event arguments and calls iwconfig with the
appropriate as needed. Note that there are many ways to make the lan
unusable doing this, so you want to be careful. You can get the MAC
addresses from the experiment info page (tbreport).

New script called link_config, which might be badly named since it
implies generality) to front end tevc. Operates mostly like
delay_config in that it will change the physical table settings, and
optionally (-m) the virtual table entries. So,

	link_config testbed two-wireless lan0 accesspoint=00:09:5B:93:0B:A4

You can change individual members of a lan too:

	link_config -s nodew1 testbed two-wireless lan0 txpower=50

Currently no web interface; too much work. I will add an xmlrpc
interface though since that is easy to do.

1. "make clean" will just remove stuff built in the process of a regular build
2. "make distclean" will also clean out configure generated files.

This is how it was always supposed to be, there was just some bitrot.

the top level.  This will build all the necessary binaries and then install
them.  This works on FBSD4 and RHL7.3.  It still doesn't work on FBSD5
(newer compiler that no longer supports a style of use of _FUNCTION_ in the
event lib) or RHL9 (event lib needs SSL lib which has a bad dependency
on Kerberos).  Notes:

- requires that elvin libraries be installed on nodes (they are) to build
  event agents, requires linuxthreads be installed on FBSD (it is now) to
  build imagezip (which is installed, but is not strictly necessary)

- installed event-agents and other binaries are stripped

- added a few missing files to the source tree for bsd (healthd.conf)
  and linux (healthd.conf, rc.local)

- the only thing that doesn't get rebuilt in /usr/local/etc/emulab is
  healthd, I couldn't quickly find how it gets built

- uses a scaled down version of libtb with no DB functions (since mysql
  isn't installed on nodes).  N.B. DO NOT DO A CLIENT INSTALL FROM YOUR
  REGULAR OBJ TREE OR ELSE YOU MAY WIND UP WITH A NEUTERED VERSION OF
  libtb.a!

The build-as-well-as-install semantics are counter to the regular install
targets, but this is what we gotta do for now.  Once the TB source builds
under Linux and newer BSDs, we could undo this and just require that people
do a regular "make" followed by "make client-install"  OTOH, there should
be no reason to require installation of mysql and other server-side packages
just to build clients (or make them sit through the compilation of assign),
so maybe we will keep the client build special.

pain.

work in the general case.

is to add HMACs to events to ensure they that events cannot be
injected into an experiment by an unauthorized client.

* The frontend now generates a secret key for each experiment and
  stores that into a file and in the DB.

* Each of the event clients, as well as the event producers
  (scheduler, tevc) have a new -k option to specify the name of the
  file. Two new event library functions were added for clients to give
  the key:

    event_handle_t
    event_register_withkeyfile(char *name, int threaded, char *keyfile);

    event_handle_t
    event_register_withkeydata(char *name, int threaded,
	   		       unsigned char *keydata, int keylen);

* When the library is in possesion of a key, it will generate an HMAC
  and attach it to outgoing notifications. A client receiving a
  notification will compute an HMAC and compare it against the HMAC in
  the notification. If they do not compare, the notification is
  dropped with a warning message printed (the client callback never
  gets the notification). If the client has not provided a key, then
  the HMAC in the incoming notification is ignored.

* The scheduler also takes a -k option, and will compute HMACs for all
  of the static events ahead of time. That keeps it off the critical
  path.

* The tevc client also takes a -k option. However, tevc will always
  try to find the keyfile (default path) so that it can attach the
  HMAC to dynamic events before sending them to the scheduler (which
  will check to make sure it matches). The scheduler will not accept
  dynamic events without unless the HMAC is present and matches.

* I have rebuilt the elvin librarys, removing all of the X goop and
  the SSL goop. Smaller binaries. So, I had to add -lcrypto to all of
  the client makefiles to that programs link.

* The program-agent got a few more changes. The command string is no
  longer passed inside the event; it comes in when the program agent
  is started, via a config file generated from tmcd data. This gets
  rid of our mostly insecure remote execution facility.

on linux. Add a makefile ifdef for Linux/Freebsd that sets up the
proper set of programs. Note that the new image I'm making will have
the event libraries installed!

of stuff in the new image. Also added client side install targets
every place I could think of.

into nsetrafgen for a client install, not a regular install.

actually build nse, "make buildnse" needs to be done.
make install puts {nse,nseinput.tcl,startnse} in a hardwired path whose base is
currently /usr/testbed/sup/sup/FBSD45-STD/root/ . nse is put usr/local/bin
under the above. nseinput.tcl and startnse are put into etc/testbed