"arbitrary" script as defined in the stated_triggers table. Currently
using this to invoke the new opsreboot script whenever ISUP comes in
from ops.

The opsreboot script is currently a skeleton. All it does is send
email.  I'll add the rest later (which really won't be much at first;
just getting the event schedulers started).

goes whacky.

node to get ignored.

- add functions to recursively dump hashes and arrays into a string
  suitable for printing as debugging output (great for data structures)

- add three new trigger strings

- add 'use strict', do corresponding cleanup

stated changes:

- move special-cased stuff in handleEvent for PXEBOOTING and BOOTING into
  triggers (PXEBOOTING, BOOTING, and CHECKGENISUP)

- clarify (via comments) the existing kinds of triggers and which ones run
  when, and add a new kind (global "any-mode" triggers). We already had
  per-node mode-specific, per-node any-mode, and global mode-specific
  triggers. Now you can have a trigger that is good for any mode in a
  given state, that can be overridden on a mode-specific basis. This is
  great for PXEBOOTING, BOOTING, and ISUP, since they each have a trigger
  list that should be run regardless of what mode you're in. Now they only
  require 3 entries instead of 3*N that have to be maintained per mode.

     # A note about triggers:
     #
     # "per-node" triggers only affect their specific node in a
     # particular mode/state, and are run first of all. "global"
     # triggers are triggers for a given mode/state that affect all
     # nodes, and are run after any per-node triggers. "Any-mode"
     # triggers are tied to a state, and occur in that state in any
     # mode. The any-mode triggers are over-ridden by global triggers,
     # and if an "Any-mode" trigger for state XYZ exists as well as a
     # global trigger for mode FOOBAR state XYZ, then when I arrive in
     # XYZ any per-node triggers will be run. Then, if I'm in mode
     # FOOBAR, only the global trigger will run. If I'm in any other
     # mode, only the any-mode trigger will run.

     # (our "*" is stored as $TBANYMODE)
     # Per-node triggers have a specific node_id
     # Global triggers have "*" as the node_id
     # Any-mode triggers have "*" as the mode, and can be global or per-node

  The updated table looks like this in the accompanying change to
  database-fill.sql:

+---------+----------+------------+-----------------------+
| node_id | op_mode  | state      | trigger               |
+---------+----------+------------+-----------------------+
| *       | *        | BOOTING    | BOOTING, CHECKGENISUP |
| *       | *        | ISUP       | RESET                 |
| *       | *        | PXEBOOTING | PXEBOOT               |
| *       | RELOAD   | RELOADDONE | RESET, RELOADDONE     |
| *       | ALWAYSUP | SHUTDOWN   | ISUP                  |
+---------+----------+------------+-----------------------+

- I also cleaned up the functions that add, get, and delete triggers.
  Before, the get function didn't include global triggers. Now it does,
  and has an option to just get the per-node triggers. Add and delete are
  still just per-node, of course.

- Also found and fixed some little bugs while I was in there. (global
  triggers not taking a list,

These changes are me getting ready to re-add all the changes I made months
ago in order to do a before-and-after experiment for my thesis. Between
now and the end of next week I'll be working on taking before numbers,
patching stated with the changes, and getting after numbers.

The problems I'm trying to replicate are the problems and slowdowns we
used to get when os_{load,setup} would reboot a node, thinking it had
timed out, when it really didn't know whether it was making progress or
not. The fix includes making os_{load,setup} depend on stated to watch for
progress and timeouts, and do any appropriate retries. Part of that is the
StateWait stuff, that lets programs watch for events easily, and the
node_reboot-with-events stuff that puts stated in control of nodes as they
reboot.

handling PXEWAKUP timeouts, retrying 3 times and then forcing a power
cycle.  Changed BOOTING event action to auto switch in and out of the
special PXEKERNEL state machine that all local nodes use since all
local nodes boot the same pxeboot kernel and talk to bootinfo (as
directed to by dhcp).

probably imperfect, but better then nothing. New option, "-t tag"
allows you to specify an arbitrary tag to match against the stated_tag
of the nodes table. The stated invocation will only operate on nodes
that match the tag, ignoring all events for other nodes. If
unspecified, stated will operate on all nodes with a NULL tag. This is
setup up at the beginning of time (or during a reload) saving the
per-node tag in the $nodes hash. Each time an event arrives, check the
tag in the table, ignoring the event if not a match.

On signaled reload() must also be careful to throw away timeouts from
the queue (and be careful not to set up new timeouts for ignored
nodes).  So, this allows you to set the tag for a node in the DB, and
then HUP stated so that it reloads it tables. That node will now be
ignored by that stated.

Also made some changes to debug mode. In debug mode, don't worry about
the pidfile or the lockfile or checking for other running stated
(which causes my debug version to exit! right away). Also, added a new
-l option to turn of syslog output and just send it all to stdout with
the debug output. -l can be only be used with -d of course.

So what can I do with all this:

	update nodes set stated_tag='lbs' where node_id='pc5';
	sudo kill -HUP `cat /var/run/stated.pid`
	sudo stated -d -l -t lbs

Which tells the main stated to ignore pc5. Then I run a debugging
stated that operates only on pc5. Later when done:

	update nodes set stated_tag=NULL where node_id='pc5';
	sudo kill -HUP `cat /var/run/stated.pid`

Which tells the main stated to operate on pc5 again.

as defined in the defs-* file (e.g. "TBLOGFACIL=local2").  The default is
"local5" which is what we are setup to use so you shouldn't need to mess
with your defs- file!

perl scripts just get this value configured in when configure is run.
C programs get the value in two ways.  For programs that are intimate with
the testbed infrastructure, and include "config.h", they just get it from
that file.  For programs that we sometimes use outside the Emulab build
environment (e.g., frisbee, capture) and that don't include config.h,
the value is set via a "-DLOG_TESTBED=..." in the GNUmakefile build line.
If the value isn't set, it defaults to what it used to be (usually LOG_USER).

Still to do: healthd, hmcd (whose build doesn't seem to be completely
integrated) and plabdaemon.in (since its icky python :-)

model of waiting for state changes. Before we were watching the database
(which means we can only watch for terminal/stable/long-lived states, and
have to poll the db). Now things that are waiting for states to change
become event listeners, and watch the stream of events flow by, and don't
have to do any polling. They can now watch for any state, and even
sequences of states (ie a Shutdown followed by an Isup).

To do this, there is now a cool StateWait.pm library that encapsulates the
functionality needed. To use it, you call initStateWait before you start
the chain of events (ie before you call node reboot). Then do your stuff,
and call waitForState() when you're ready to wait. It can be told to
return periodically with the results so far, and you can cancel waiting
for things. An example program called waitForState is in
testbed/event/stated/ , and can also be used nicely as a command line tool
that wraps up the library functionality.

This also required the introduction of a TBFAILED event that can be sent
when a node isn't going to make it to the state that someone may be
waiting for. Ie if it gets wedged coming up, and stated retries, but
eventually gives up on it, it sends this to let things know that the node
is hozed and won't ever come up.

Another thing that is part of this is that node_reboot moves (back) to the
fully-event-driven model, where users call node reboot, and it does some
checks and sends some events. Then stated calls node_reboot in "real mode"
to actually do the work, and handles doing the appropriate retries until
the node either comes up or is deemed "failed" and stated gives up on it.
This means stated is also the gatekeeper of when you can and cannot reboot
a node. (See mail archives for extensive discussions of the details.)

A big part of the motivation for this was to get uninformed timeouts and
retries out of os_load/os_setup and put them in stated where we can make a
wiser choice. So os_load and os_setup now use this new stuff and don't
have to worry about timing out on nodes and rebooting. Stated makes sure
that they either come up, get retried, or fail to boot. tbrestart also
underwent a similar change.

really-reboot-nodes-that-timeout stuff.

NOTE: Until the timeout/retry stuff is gone from os_load/os_setup, it is
disabled in stated. It will still only send email. But all the stuff is
there and has been tested.

NOTE: Until other things don't depend on the old behavior of node_reboot
(when it returns, all nodes are in SHUTDOWN), the event stuff is disabled.
Real mode is the default, and can be run by anyone.

In short, this commit is new versions of stated and node_reboot that act
almost exactly like the old ones. But I wanted to commit them before I go
on making a bunch more changes, to have a checkpoint that I know works.

is supported:

- stated listens for TBCOMMAND events, and currently handles REBOOT,
  POWEROFF, POWERON, and POWERCYCLE events. It does everything except make
  the actual calls to node_reboot and power. And it accepts batches of
  nodes instead of just single ones.

- Timeouts were added to the db for these commands, with no timeout for
  the power ones (since the node can't hang during those), and a 15 second
  timeout from reboot until the SHUTDOWN state.

- If a rebootimes out, it tries it again, up to 3 times. If it gets to
  three times without working, it sends mail to tbops and turns the
  machine off instead of continuing to reboot it. Right now I haven't
  made it do node_reboot -f or power cycle on retries, but it easily
  could.

- Stuff to be done before they work: make node_reboot send an event
  instead of doing the work, and make a new script that has node_reboot's
  old guts. Note that this requires authentication in our events for these
  commands, and a way to make sure that the command that came in as an
  event was properly authenticated.

- For future growth and expansion, it is set up so it should be relatively
  easy to add other commands that do different things, even if they take
  arbitrary params that aren't nodes or lists of nodes.

1. timeouts for nodes weren't getting reset when they had a mode
ransition, so they were timing out in shutdown after changing modes.
2. It was still going back into a blocking wait, even though a signal had
been recieved, and not quitting back up to the main loop to handle it.

code when taking a signal.

memory leak in one of the timeout queue data structures, more or less.

didn't see in testing. Specifically, why it pegs at 100% CPU after a
while, and why it gets timeouts after it has removed the timeout from the
queue.

1. Change from inefficient timeout search algo that ran once per second to
a highly efficient priority queue method of managing timeouts. Now
instead of checking every node's timestamps, we just look at the head of
the queue, and it is often much less frequent than once a second, since we
know how long we have until the next timeout.

2. Start using a blocking poll for events, so I can sleep for long periods
of time instead of having to wake up at least once a second to check for
timeouts and events. Will set the block timeout for the shortest of: the
time to send out the next batch of queued emails, the next time a timeout
may occur, or when there are no mails waiting and no timeouts possible, 10
minutes. Comes back as soon as an event comes in.

3. Given the above two items, we no longer need a sleep(1) in our main
loop.

One small glitch is in the progress of being fixed. When using blocking
polls, things hang when trying to unregister from the event system. Not a
big deal, just ^C twice to kill it. (May cause it to need two SIGUSR1's
to get it to restart, too.)

In the next update, look for:
 - Really take action on timeouts.
   - keep track of how many times we've retried, and notify if something
     may be wrong with the node.
   - Find out policy on taking action with timeouts.
     - Do it if the expt is in transition or the node is free
     - Probably don't touch if the expt is established.
     - Maybe? in active expt, send (good) email to expt owner on timeouts

Related "coming soon" items:
os_load/os_setup etc.:
 - Add the waitforstate stuff we've talked about
 - make os_load/os_setup use it

allows only SHUTDOWN and ISUP states, and whenever stated sees a SHUTDOWN
for it, there's a trigger that sends an ISUP right afterwards. Thus the
name, ALWAYSUP.

get an updated copyrights message.

on any node on any state, in any specific mode, or without any mode
restriction.

The imediate use of this is the FREENODE trigger. Now RELOADDONE adds
a FREENODE trigger on the ISUP state, if the node is in the reloading
expt. Then next time the node hits ISUP, it gets freed from the
reloading expt.

This fix solves the race where recently freed (and still rebooting)
nodes get grabbed by an expt and get rebooted in a way that may hoze
their FS's.

Also fixed a problem that was making it load the db twice on startup.

state. Before, this caused "Running os Foo but in the wrong opmode!"
messages. Now, if I get to BOOTING and the state I'm coming from was ISUP,
and there's a next_op_mode waiting for me, force the mode transition
without sending any mail.

make sure it got run as root. Update that to allow running as non root if
you're in a devel tree and you're an admin (in your own copy of the db).
This will let flest run it simply by using 'withadminprivs' with the
startup of stated.

 - fix bad indenting to a uniform 4 spaces (before was 2, 4 and 8 mixed)
 - Move ping-for-isup functionality into a separate script
 - Make sure every transition triggered by stated (directly or indirectly)
   sends an event, instead of taking shortcuts.

This called for a new script, eventping, which just pings until the node
is pingable, then sends an ISUP event. Stated runs this in the background
where necessary, and nothing else should run it.

Adding eventping meant modifying configure and the utils makefile, too.

treatment. Fix up the scripts that use them, so that they get
quoted correctly.

as the op_mode it is currently running. If not, force it and send mail.

This fixes the "stuck-in-reloading" phenomenon we occasionally see when
the states get messed up. Now anytime it loads PXEFRISBEE it should force
the mode to RELOAD, and it will stop reloading the first time it hits
RELOADDONE.

events. This may delay handling of other stuff that happens in my main
loop, but not by too much. To prevent skew, everything (including reload
frequency) is done strictly by seconds elapsed, not by iterations or
anything.

I found that even polling for multiple events without sleeping, I could
only handle a little over 1 per second when I was calling inuse/statetime
for additional info on every event. Even though this only happens in the
worst case (every event is wrong), it won't do. So I took that out. I'll
probably end up adding a faster lookup of the info I need (mostly
reservation, and what osid it thinks it is running). That change took it
up to at least 4 per second (as fast as I could send them manually), more
than 4x our previous performance. So we should be able to keep up now.

Also, add the support for "announcements" to testbed ops when I die and
such. (Been in a few days, but this is the first commit of it)

Set mailgap to 15 seconds, and wait a second on startup before sending mail so that the timeouts all come in one message. And don't clean up my stated.pid file if I'm a forked child of the real daemon.

First, fix up the isup generation code. When a node/OS doesn't send its
own isups, but is pingable, we need to fork and ping it, and send ISUP
when it pings. The code was there, but was broken. This fixes it. The one
time that it may cause errant messages is in modes other than MINIMAL.
When we get BOOTING, we check if it needs isup generated. If we have to
ping it, when it pings we send ISUP. This means that if we are really in
NORMAL mode, we might send ISUP before the node sends REBOOTED (or TBSETUP
in NORMALv1), and it would look funny. But that case will be really rare,
since everything that sends REBOOTED or TBSETUP has no reason not to send
ISUP itself.

Second, after mailbombing myself a couple of times, Kirk and I decided I'd
better put some throttling in the notification code that stated uses. So
now it throttles itself and digests the messages if they're sent too close
together. The first message it gets will get sent immediately. If the next
one is long enough after that, it sends it immediately too. If a message
comes too soon after sending one, we queue it up, and send it later
after enough time has passed. Currently it is set to wait 5 seconds
between messages, so it will send up to 12 per second, and wait no more
than 5 seconds before sending a message that is queued up.

(Something similar to this may be a nice thing in the rest of our stuff,
but it was made a lot easier by the fact that stated already had a polling
loop in it. Without that, you'd have to use alarms or some other weird
thing, which would be painful.)