before turning them into xmlrpc dates

use/test the event system while logins are turned off.

classes are now used from SocketServer.ForkingMixIn, and its easier to
move the little global state we have into the classes, rather then try
to refresh at each fork.

list of lists instead of a big long string to be parsed (which I did
cause I had forgotten about Tim's C++ RPC library).

shells on boss) to use the rpc server without an agent running.
Using the no-passphrase key, these changes allow us to use the server
from ops in a very restricted manner. This change is temporary, until
I have something better in place. In the meantime, admin people change
their auth keys files on *boss* as such:

command="/usr/testbed/sbin/sshxmlrpc_server.py -ro",from="ops.emulab.net" ... rest of emulab generated key ...

Note the -ro argument; very important!

scheduler running on ops.

Okay, so this is probably not an ideal mechanism for wrapping up what are
essentially DB queries, since there are no xmlrpc-over-ssh bindings for
C. So we have to run an external client (sshxmlrpc_client) to talk to the
server. This is not so bad in itself, but returning the eventlist as a
single response seems like an unnatural approach for returning what is
essentialy a stream of data. Besides, unless we expect people might want to
write their own schedulers (highly unlikely), exporting this stuff via the
XMLRPC server is a silly thing to do.

Instead, some kind of protected DB access proxy might be more appropriate.
I have no idea how this might operate, but its something to think about.

Add waitforactive() to avoid waiting for state='active' on the client.

supporting both a shell script driven interface, plus the newer XMLRPC
interface. This change removes the script driven interface from boss,
replacing it with just the XMLRPC interface. Since we like to maintain
backwards compatability with interfaces we have advertised to users (and
which we know are being used), I have implemented a script wrapper that
exports the same interface, but which converts the operations into XMLRPC
requests to the server. This wrapper is written in python and uses our
locally grown xmlrpc-over-ssh library. Like the current "demonstation"
client, you can take this wrapper to your machine that has python and ssh
installed, and use it there; you do not need to use these services from
just users.emulab.net. Other things to note:

* The wrapper is a single python script that has a "class" for each wrapped
  script. Running the wrapper without any arguments will list all of the
  operations it supports. You can invoke the wrapper with the operation as
  its argument:

    {987} stoller$ script_wrapper.py swapexp --help
    swapexp -e pid,eid in|out
    swapexp pid eid in|out
    where:
         -w   - Wait for experiment to finish swapping
         -e   - Project and Experiment ID
         in   - Swap experiment in  (must currently be swapped out)
        out   - Swap experiment out (must currently be swapped in)

    Wrapper Options:
        --help      Display this help message
        --server    Set the server hostname
        --login     Set the login id (defaults to $USER)
        --debug     Turn on semi-useful debugging

   But more convenient is to create a set of symlinks so that you can just
   invoke the operation by its familiar scriptname. This is what I have
   done on users.emulab.net.

    {987} stoller$ /usr/tesbed/bin/swapexp --help
    swapexp -e pid,eid in|out
    swapexp pid eid in|out


* For those of you talking directly to the RPC server from python, I have
  added a wrapper class so that you can issue requests to any of the
  modules from a single connection. Instead using /xmlrpc/modulename, you
  can use just /xmlrpc, and use method names of the form experiment.swapexp,
  node.reboot, etc.

  Tim this should be useful for the netlab client which I think opens up
  multiple ssh connections?

* I have replaced the paperbag shell with a stripped down xmlrpcbag shell
  that is quite a bit simpler since we no longer allow access to anything
  but the RPC server. No interactive mode, no argument processing, no
  directory changing, etc. My main reason for reworking the bag is to make
  it easier to understand, maintain, and verify that it is secure. The new
  bag also logs all connections to syslog (something we should have done in
  the orginal). I also added some setrlimit calls (core, maxcpu). I also
  thought about niceing the server down, but that would put RPC users at a
  disadvantage relative to web interface users. When we switch the web
  interface to use the XMLRPC backend, we can add this (reniceing from the
  web server would be a pain cause of its scattered implementation).

* Added a wrapper class so that you can invoke methods as
  experiment.swapexp or node.reboot. So instead of invoking as
  /XMLRPC/experiment can calling swapexp, you can call the server as
  /XMLRPC and call experiment.swapexp. This allows you to use a single
  connection to talk to different parts of the API. Note this is standard
  (or is it defacto) syntax in XMLRPC.

* Changed the demonstration client to talk the server this way.

* Changed paperbag to allow this as well; the xmlrpc server is invoked with
  no args, which tells it to export the wrapper interface instead of a
  specific module interface.

* A few more cleanups in the server, more permission checks, etc.

interface (which will have its own frontend to emulate the original
script interface!)

* Refactor the experiment permission checks into a subroutine.

* Cleanup some parameter names that did not make sense.

* Add portstats, readycount, node_admin, eventsys_control, savelogs,

* A couple of little fixes.

instead of just their names.  Enhance experiment.statewait() to accept
multiple states to wait for, or, to wait for any state change.  Add
experiment.thumbnail() method so the topology can be displayed in
NetBuild.

Fix experiment.info() to conditionally add the "idle" field to the
mapping aspect.

client to wait for an experiment to reach a desired state (active,
swapped, etc). Use the optional timeout parameter to terminate the
wait early. See the documentation for a full descriptione of the
interface.

This is implemented using the just added "poll" interface to the
python event interface library

format instead of plain old well formed XML. Note this change does not
affect the parser in any way; the parser continues to spit out its own
brand of XML, and xmlconvert treats that special when it reads it.

The XML I spit out now is better formed and makes more sense when you
look at it. It also helps that its much faster to generate, and the
resulting output is much smaller. For example:

	time /usr/testbed/libexec/xmlconvert testbed jail-2440
	real    2m54.229s
	user    1m7.492s
	sys     1m15.927s

	time /usr/testbed/devel/stoller/libexec/xmlconvert testbed jail-2440
	real    0m14.738s
	user    0m5.755s
	sys     0m5.156s

	/tmp/bar*.xml
	-rw-rw-r--  1 stoller  wheel  15595161 Jun  4 11:16 /tmp/rpc.xml
	-rw-rw-r--  1 stoller  wheel   5285690 Jun  4 11:17 /tmp/plain.xml

Thats for a big experiment. For a small experiment:

	time /usr/testbed/libexec/xmlconvert testbed jail-416
	real    0m9.346s
	user    0m5.083s
	sys     0m2.675s

	time /usr/testbed/devel/stoller/libexec/xmlconvert testbed jail-416
	real    0m2.751s
	user    0m1.377s
	sys     0m1.183s

	ll /tmp/bar*.xml
	-rw-rw-r--  1 stoller  wheel  3135518 Jun  4 11:19 /tmp/rpc.xml
	-rw-rw-r--  1 stoller  wheel  1015745 Jun  4 11:19 /tmp/plain.xml

As you can see, the RPC code is rather non-linear in its performance!

Since this affects Tim and netlab-client, I have added an alternate
XMLRPC routine called virtual_topology_xml() that returns the new format,
since Tim will need to transition to this new format (parsing it into a
Java data structure). The old format is left in the RPC server until that
is done.

For tbres/resweb (75 nodes):

	Old time: 42 seconds.
	New time: 19 seconds.

For testbed/BigLan (600 nodes):

	Old time: 120 seconds.
	New time: 40  seconds.

So, still needs some work, but for the average case its not so bad
anymore.

I also turn on compression to level 5 by default in our sshxmlrpc.py
library, thereby overriding the user's config file for that option.

for the "links" aspect in experiments with no links.

data.  Add hostkeys() to the node module so we can get the SSH host
keys from nodes.

pid and eid, and it returns a giant data structure consisting of all
of the virtual tables.

At some point we want to combine the physical representation into the
structure (for mapped experiments).

Add five new modules:

  imageid, osid - Get listings of image/OS IDs.
  fs - Get directory listings for the emulab file system.
  user - Get information about the user.
  emulab - Get information about emulab as a whole.

Added new method response, RESPONSE_REFUSED, which is returned
whenever emulab is down or otherwise unavailable.

Arguments to methods can now be strings or typed XML-RPC data.  For
example, a boolean parameter will accept (in python) True or the
string "true".

Added osid and status values to the "mapping" aspect of the
"experiment.info" method.  Also added "experiment.constraints" for
getting the idle threshold and "experiment.getlist" for getting the
list of experiments where the user is the head.

Added "node.console" method to get ACL data for connecting to a node's
console.

invoked from the RPC server.

Add "info" method to return stuff that we currently get via the
tbreport script. The difference is that the info is returned as hashed
arrays (or arrays) instead of as a big print statement. Currently just
the node mapping and link mapping functions (of tbreport) are
supported.

At Jay's request, purge pid/eid/gid from documentation and use/allow
"proj", "exp", "group" instead. Note that pid/eid/gid are still
accepted by the server (cause I won't be able to stop using them!).

brought on by too much thinking and not enough hacking.