GitLab

If the libevent package is installed, there will be /usr/local/include/event.h
which conflicted with our event.h in the clientside/lib/event directory.

This commit is intended to makes the license status of Emulab and
ProtoGENI source files more clear.  It replaces license symbols like
"EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
blocks that contain actual license statements.

This change was driven by the fact that today, most people acquire and
track Emulab and ProtoGENI sources via git.

Before the Emulab source code was kept in git, the Flux Research Group
at the University of Utah would roll distributions by making tar
files.  As part of that process, the Flux Group would replace the
license symbols in the source files with actual license statements.

When the Flux Group moved to git, people outside of the group started
to see the source files with the "unexpanded" symbols.  This meant
that people acquired source files without actual license statements in
them.  All the relevant files had Utah *copyright* statements in them,
but without the expanded *license* statements, the licensing status of
the source files was unclear.

This commit is intended to clear up that confusion.

Most Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the Affero GNU General Public License, version 3
(AGPLv3).

Most Utah-copyrighted files related to ProtoGENI are distributed under
the terms of the GENI Public License, which is a BSD-like open-source
license.

Some Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the GNU Lesser General Public License, version 2.1
(LGPL).

clientside requres it to be there. As noted in previous revision, the
full emulab build always includes the clientside subdir, but a
clientside build will not necessarily include the rest of Emulab.

things for which we previously linked -static.

Fixup some clean targets in a couple of cases as well.

Get capture to compile without SSL (capture-nossl).  This is untested.

Mote and robot related stuff.  The main thing is the addition of relay
capabilities to capture and related things.

	* GNUmakefile.in: Add the capture and tip subdirectories to the
	client and client-install targets.

	* configure, configure.in, config.h.in: Detect srandomdev() for
	capture and add "mote/newmote" script.

	* capture/GNUmakefile.in, capture/capture.c: Add "relay"
	capabilities to capture.

	* capture/capquery.c: Query the capserver for the relay receiver's
	port number.

	* capture/capserver.c: Small hack to return the port number
        for a node.

	* db/libdb.pm.in, db/xmlconvert.in: Add virt_tiptunnels table.

	* event/program-agent/program-agent.c: Change log file names to
	something a little more user-friendly.  Add a "MODIFY" event
	handler that lets the user set agent attributes (command, tag,
	timeout) without having to run a program.

	* event/sched/GNUmakefile.in, event/sched/console-agent.cc,
	event/sched/console-agent.h, event/sched/event-sched.c: Add
	console agents that can be used to snapshot a section of the
	capture log file.

	* event/sched/node-agent.cc: Some minor cleanup.

	* event/sched/simulator-agent.cc, event/sched/simulator-agent.h:
	Add the config data to the report mail.  Add a "RESET" event
	handler that runs "loghole clean".  Save the report mail in a file
	so it gets archived with the rest of the logs.

	* lib/libtb/tbdefs.h: Add CONSOLE object type.

	* mote/GNUmakefile.in, mote/newmote: Add newmote script, just a
	quick hack to add motes to the DB.

	* mote/tbuisp.in: Add another backend for loading motes through
	their relay capture server.

	* robots/mtp/mtp_dump.c: Dump the min/max values for x and y,
	handy for figuring out the bounds of the camera.

	* sql/database-fill.sql: Change the RELOAD-MOTE/SHUTDOWN ->
	ALWAYSUP/SHUTDOWN mode transition to ALWAYSUP/ISUP since stated
	doesn't seem to run triggers after a state change by a mode
	transition.

	* tbsetup/tbreport.in: Change the ordering of the eventlist so it
	displays event-sequences appropriately.

	* tbsetup/ns2ir/GNUmakefile.in, tbsetup/ns2ir/console.tcl,
	tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/parse.tcl.in,
	tbsetup/ns2ir/sim.tcl.in: Add a "console" agent that represents
	the serial console for a node.

	* tbsetup/ns2ir/sequence.tcl: Add an "append" method so it is
	easier to build sequences dynamically.

	* tbsetup/ns2ir/topography.tcl: Make checkdest available to
	regular users.

	* tip/GNUmakefile.in, tip/tiptunnel.c: Add support for uploading a
	file to a relay version of capture and exporting the end
	connection as a pty.

	* tmcd/decls.h, tmcd/common/libsetup.pm: Bump version number since
	the dosubnodelist change is not backwards compatible.

	* tmcd/tmcd.c: Make dosubnodelist and dosubconfig callable even
	when a node isn't allocated.  Add dotiptunnels command that
	returns which serial consoles are to be mounted on a node.  Add
	mote version of subconfig that returns information needed to
	startup the relay version of capture.

	* tmcd/common/bootsubnodes: For motes, startup the relay version
	of capture (XXX stargate specific).

	* tmcd/common/libsetup.pm, tmcd/common/libtmcc.pm,
	tmcd/common/config/rc.config, tmcd/common/config/rc.tiptunnels:
	Client side changes for mounting another nodes serial line.

	* tmcd/common/rc.bootsetup: Always boot the subnodes, even when
	free.  This is used for motes since their capture needs to be up
	for reloading at the time.

	* tmcd/linux/ixpboot: Shuffle some code around so the script
	doesn't fail if the ixp isn't allocated.

	* utils/loghole.in: Add "digest.out" and "report.mail" as global
	logs to be saved in archives and display the "report.mail" file
	when showing a loghole archive.

	* xmlrpc/emulabserver.py.in: Scrub more of the return values to
	get rid of "None"s.

again...

More robot integration and some event system updates.

	* configure, configure.in: Detect rsync for loghole and add
	utils/loghole to the list of template files.

	* db/libdb.pm.in, db/xmlconvert.in: Add virt_node_startloc to the
	list of virtual tables.

	* event/lib/event.h, event/lib/event.c, event/lib/tbevent.py.tail:
	Add event_stop_main function to break out of the event_main()
	loop.  Add timeline to the address tuple.

	* event/sched/GNUmakefile.in, event/sched/error-record.h,
	event/sched/error-record.c, event/sched/event-sched.8,
	event/sched/event-sched.h, event/sched/event-sched.c,
	event/sched/group-agent.h, event/sched/group-agent.c,
	event/sched/listNode.h, event/sched/listNode.c,
	event/sched/local-agent.h, event/sched/local-agent.c,
	event/sched/node-agent.h, event/sched/node-agent.cc,
	event/sched/queue.c, event/sched/rpc.h, event/sched/rpc.cc,
	event/sched/simulator-agent.h, event/sched/simulator-agent.c,
	event/sched/timeline-agent.h, event/sched/timeline-agent.c:
	Updated event scheduler, not completely finished, but well enough
	along for the robots.

	* lib/libtb/GNUmakefile.in, lib/libtb/popenf.h,
	lib/libtb/popenf.c, lib/libtb/systemf.h, lib/libtb/systemf.c: Add
	some handy versions of system/popen that take format arguments.

	* lib/libtb/tbdefs.h, lib/libtb/tbdefs.c: Add some more event and
	object types.

	* tbsetup/assign_wrapper.in: Add the virt_node_startloc building
	to desires string for a node.

	* tbsetup/ptopgen.in: Add a node's location to the feature list.

	* tbsetup/tbreport.in: Display the timeline/sequence an event is a
	part of.

	* tbsetup/ns2ir/GNUmakefile.in: Add timeline, sequence, and
	topography files.

	* tbsetup/ns2ir/node.tcl: Add initial position for nodes and allow
	them to be attached to "topographys".

	* tbsetup/ns2ir/parse-ns.in: Make a hwtype_class array with a
	node_type's class.  Make an 'areas' array that holds the
	buildings where nodes are located.  Make an 'obstacles' table
	with any obstacles in the building.

	* tbsetup/ns2ir/parse.tcl.in: Move named-args function from
	tb_compat.tcl to here.  Add reltime-to-secs function that converts
	time given in a format like "10h2m1s" to a seconds value, used in
	"$ns at" so its easier to write time values.  Add "K", "Kb", and
	"Kbps" as possible units for bandwidth (only the lowercase
	versions were available before).

	* tbsetup/ns2ir/program.tcl: Add "dir" and "timeout" attributes,
	although they don't go anywhere at the moment.

	* tbsetup/ns2ir/sequence.tcl, tbsetup/ns2ir/timeline.tcl,
	tbsetup/ns2ir/topography.tcl: Initial versions.

	* tbsetup/ns2ir/sim.tcl.in: Add support for timelines and
	sequences.  Add 'node-config' method to change the default
	configuration for nodes produced by the Simulator object.  Send an
	initial MODIFY event to any trafgen objects so their configuration
	gets through, even when there are no start/stop events.  Move
	event parsing to the 'make_event' method.

	* utils/loghole.1, utils/loghole.in: Loghole utility, used for
	retrieving logs from experimental nodes and creating archives of
	the logs.

	* xmlrpc/emulabclient.py.in: Escape any strange characters in the
	output field.

	* xmlrpc/emulabserver.py.in: Add virt_node_startloc to the list of
	virtual_tables.  Add emulab.vision_config and
	emulab.obstacle_config methods for getting information pertaining
	to the robots.  Change the OSID listing to include more fields.
	Add a "physical" aspect to experiment.info to get information
	about the physical nodes.  Add parent field to the events in the
	array returned by eventlist.  Add sshdescription to get extra
	information needed to log into a vnode.  Add node.statewait so you
	can wait for nodes to come up.

Change to the SSL version of the event scheduler.

  * db/libdb.py.in, xmlrpc/emulabserver.py.in: Only add the testbed
    library path to sys.path if it is not already there.

  * event/sched/GNUmakefile.in: Make the SSL version of the scheduler
    the default instead of the SSH version and statically link the
    executable.

  * event/sched/event-sched.c: Pass the default SSL port number (3069)
    to RPC_init.

  * event/sched/rpc.cc: Bring the SSL code up to date: read the cert
    from the user's home directory, make the connection persistent,
    and use TBROOT as the request path, so the development version of
    the XML-RPC library is used when appropriate.

  * xmlrpc/sslxmlrpc_server.py.in: Updated to let the user select from
    a set of allowed library paths where the 'emulabserver' module
    should be imported from.  Import the 'emulabserver' module after the
    fork so we always get the latest version of the module.  Twiddled
    the necessary bits to turn on persistent connection support.

Make the hurting stop.  Make sshxmlrpc auto-detect things, fails over
properly, and dump useful information when it is unable to deal with
the peer.

  * xmlrpc/sshxmlrpc.py: Major update.  It now tries to autoconfigure
    itself by scanning the path for "ssh" and "plink.exe" (although I
    haven't actually tried it on windows).  Environment variables can
    now be used to turn on debugging and set the command to use for
    doing the ssh.  Before running ssh, it will check for an agent or
    a passphrase-less key and prints a warning if it finds neither.
    The last five lines read from the server, as well as the standard
    error output, are stored so they can be dumped later; helpful for
    figuring out what is actually being run on the other side.  The
    protocol layer between ssh and xml-rpc will now respond to a
    "probe" header so that clients can figure out who they are talking
    too.  The server side will now properly detect a closed connection
    and not write anything, which means no more annoying "Write to
    stdout failed" messages.  You can now pass additional options to
    ssh and set the identity.  The module can be run standalone, with
    the default action being to probe the peer:

      $ ./sshxmlrpc.py ssh://boss/xmlrpc
      Probe results for: ssh://boss/xmlrpc
        response time=1.49 s
      Response Headers
        date: Wed Oct 27 16:10:58 2004
	content-length: 0
	probe: /usr/testbed/devel/stack/lib/sshxmlrpc.py
	probe-response: EmulabServer

  * xmlrpc/sshxmlrpc_server.py.in: Set the value returned by a "probe"
    to the name of the invoked module.  This way, the other side can
    figure out who they are talking to (e.g. EmulabServer
    vs. experiment vs. fs vs. osid).

  * event/sched/event-sched.c, event/sched/rpc.cc, event/sched/rpc.h,
    xmlrpc/script_wrapper.py.in: Multiple paths (e.g. xmlrpc,
    $prefix/sbin/sshxmlrpc_server.py) are now probed before giving up.
    Force the use of the user's default identity and protocol one.
    For event-sched, a single connection is now made at startup and
    dropped before going into the event loop.

  * event/sched/GNUmakefile.in: Add a dependency for the install
    target and add -I$(OBJDIR) to the CXXFLAGS.

  * install/ports/ulsshxmlrpcpp/Makefile,
    install/ports/ulsshxmlrpcpp/distinfo,
    install/ports/ulsshxmlrpcpp/pkg-descr: Bump version number to 1.1
    and tweak the description.

  * config.h.in, configure, configure.in: Add a "#define TBROOT" that
    has the install prefix.

* The per-experiment event scheduler now runs on ops instead of boss.
  Boss still runs elvind and uses events internally, but the user part
  of the event system has moved.

* Part of the guts of eventsys_control moved to new script, eventsys.proxy,
  which runs on ops and fires off the event scheduler. The only tricky part
  of this is that the scheduler runs as the user, but killing it has to be
  done as root since a different person might swap out the experiment. So,
  the proxy is a perl wrapper invoked from a root ssh from boss, which
  forks, writes the pid file into /var/run/emulab/evsched/$pid_$eid.pid,
  then flips to the user and execs the event scheduler (which is careful
  not to fork). Obviously, if the kill is done as root, the pid file has to
  be stored someplace the user is not allowed to write.

* The event scheduler has been rewritten to use Tim's C++ interface to the
  sshxmlrpc server on boss. Actually, I reorg'ed the scheduler so that it
  can be built either as a mysql client, or as RPC client. Note that it can
  also be built to use the SSL version of the XMLRPC server, but that will
  not go live until I finish the server stuff up. Also some goo for dealing
  with building the scheduler with C++.

* Changes to several makefiles to install the ops binaries over NFS to
  /usr/testbed/opsdir. Makes life easier, but only if boss and ops are
  running the same OS. For now, using static linking on the event scheduler
  until ops upgraded to same rev as boss.

* All of the event clients got little tweaks for dealing with the new CNAME
  for the event system server (event-sever). Will need to build new images
  at some point. Old images and clients will continue to work cause of an
  inetd hack on boss that uses netcat to transparently redirect elvind
  connections to ops.

* Note that eventdebug needs some explaining. In order to make the inetd
  redirect work, elvind cannot be listening on the standard port. So, the
  boss event system uses an alternate port since there are just a few
  subsystems on boss that use the server, and its easy to propogate changes
  on boss. Anyway, the default for eventdebug is to connect to the standard
  port on localhost, which means it will work as expected on ops, but will
  require -b argument on boss.

* Linktest changes were slightly more involved. No longer run linktest on
  boss when called from the experiment swapin path, but ssh over to ops to
  fire it off. This is done as the user of course, and there are some
  tricks to make it possible to kill a running linktest and its ssh when
  experiment swapin is canceled (or from the command line) by forcing
  allocation of a tty. I will probably revisit this at some point, but I
  did not want to spend a bunch of time on linktest.

* The upgrade path detailed in doc/UPDATING is necessarily complicated and
  bound to cause consternation at remote sites doing an upgrade.

possible to:

	gmake client
	sudo gmake client-install

on a FBSD4, FBSD5, RHL7.3, and RHL9.0 client node.

There are still some dependencies that are not explicit and which would
prevent a build/install from working on a "clean" OS.  Two that I know of are:
you must install our version of the elvin libraries and you must install boost.

system programs. Build two versions of the event library, one for non
threaded programs and another (libevent_r.a) for threaded versions.
Remove all that -pthread goo from all of the makefiles, except for the
scheduler (which is threaded); it gets linked as before, but with
-levent_r instead. If you try and use the threaded API without the
proper link, the library will print an error message and quit.

mysql code (similar to whats in tmcd.c) plus some logging function
support.

scheme that we discussed in email. Notifications and subscriptions now
take an "address_tuple" argument (I know, crappy name) that is a
structure that looks like this:

	char		*site;		/* Which Emulab site. God only */
	char		*expt;		/* Project and experiment IDs */
	char		*group;		/* User defined group of nodes */
	char		*host;		/* A specific host */
	char		*objtype;	/* LINK, TRAFGEN, etc ... */
        char		*objname;	/* link0, cbr0, cbr1, etc ... */
        char		*eventtype;	/* START, STOP, UP, DOWN, etc ... */

These can be a specific value, ADDRESSTUPLE_ANY if you are a
subscriber, or ADDRESSTUPLE_ALL if you are a producer. The reason for
the distinction is that you can optimize the match expression with the
extra bit of information, and the above structure can make for a
fairly lengthy match expression, which takes more time of course.
You should use address_tuple_alloc() and address_tuple_free() rather
than allocating them yourself. Note that host above is actually the
ipaddr of control interface. This turns out to be more convenient
since free nodes do not have virtual names.

Also added a new tbgen directly. This directory includes 3 programs in
the making:

tbmevd: Is the Testbed Master Event Daemon, to be run on boss and will
handle TBCONTROL events (reboot, reload, etc). It is just a shell of a
program right now, that takes the events but does not do anything
useful with them. Have not defined what the events are, and what DB
state will be modified.

tbmevc: Is the Testbed Master Event Client (akin to tmcc). It
generates TBCONTROL events which the tbmevd will pick up and do
something useful with. This program is intended to be wrapped by a
perl script that will ask the tmcd for the name of the boss (running
the event daemon).

sample-client: This is a little client to demonstrate how to connect
to the event system and use the address tuple to subscribe to events,
and then how to get information out of notifications.

Note that I have not created a proper build environment yet, so new
programs should probably go in the event dir for now, and link using
the same approach as in tbgen/GNUmakefile.in.

Also several changes for building on FreeBSD. I've left the older
Makefiles in place so Ian can continue to build in his environmant
(Linux?).