1. 19 Aug, 2004 1 commit
  2. 18 Aug, 2004 1 commit
    • Leigh B. Stoller's avatar
      Minor extension to stated. Add a trigger mechanism for invoking an · 6cf3e936
      Leigh B. Stoller authored
      "arbitrary" script as defined in the stated_triggers table. Currently
      using this to invoke the new opsreboot script whenever ISUP comes in
      from ops.
      The opsreboot script is currently a skeleton. All it does is send
      email.  I'll add the rest later (which really won't be much at first;
      just getting the event schedulers started).
  3. 22 Jul, 2004 1 commit
  4. 11 Feb, 2004 1 commit
  5. 15 Jan, 2004 1 commit
    • Mac Newbold's avatar
      libdb changes: · 78ad260c
      Mac Newbold authored
      - add functions to recursively dump hashes and arrays into a string
        suitable for printing as debugging output (great for data structures)
      - add three new trigger strings
      - add 'use strict', do corresponding cleanup
      stated changes:
      - move special-cased stuff in handleEvent for PXEBOOTING and BOOTING into
      - clarify (via comments) the existing kinds of triggers and which ones run
        when, and add a new kind (global "any-mode" triggers). We already had
        per-node mode-specific, per-node any-mode, and global mode-specific
        triggers. Now you can have a trigger that is good for any mode in a
        given state, that can be overridden on a mode-specific basis. This is
        great for PXEBOOTING, BOOTING, and ISUP, since they each have a trigger
        list that should be run regardless of what mode you're in. Now they only
        require 3 entries instead of 3*N that have to be maintained per mode.
           # A note about triggers:
           # "per-node" triggers only affect their specific node in a
           # particular mode/state, and are run first of all. "global"
           # triggers are triggers for a given mode/state that affect all
           # nodes, and are run after any per-node triggers. "Any-mode"
           # triggers are tied to a state, and occur in that state in any
           # mode. The any-mode triggers are over-ridden by global triggers,
           # and if an "Any-mode" trigger for state XYZ exists as well as a
           # global trigger for mode FOOBAR state XYZ, then when I arrive in
           # XYZ any per-node triggers will be run. Then, if I'm in mode
           # FOOBAR, only the global trigger will run. If I'm in any other
           # mode, only the any-mode trigger will run.
           # (our "*" is stored as $TBANYMODE)
           # Per-node triggers have a specific node_id
           # Global triggers have "*" as the node_id
           # Any-mode triggers have "*" as the mode, and can be global or per-node
        The updated table looks like this in the accompanying change to
      | node_id | op_mode  | state      | trigger               |
      | *       | *        | BOOTING    | BOOTING, CHECKGENISUP |
      | *       | *        | ISUP       | RESET                 |
      | *       | *        | PXEBOOTING | PXEBOOT               |
      | *       | RELOAD   | RELOADDONE | RESET, RELOADDONE     |
      | *       | ALWAYSUP | SHUTDOWN   | ISUP                  |
      - I also cleaned up the functions that add, get, and delete triggers.
        Before, the get function didn't include global triggers. Now it does,
        and has an option to just get the per-node triggers. Add and delete are
        still just per-node, of course.
      - Also found and fixed some little bugs while I was in there. (global
        triggers not taking a list,
      These changes are me getting ready to re-add all the changes I made months
      ago in order to do a before-and-after experiment for my thesis. Between
      now and the end of next week I'll be working on taking before numbers,
      patching stated with the changes, and getting after numbers.
      The problems I'm trying to replicate are the problems and slowdowns we
      used to get when os_{load,setup} would reboot a node, thinking it had
      timed out, when it really didn't know whether it was making progress or
      not. The fix includes making os_{load,setup} depend on stated to watch for
      progress and timeouts, and do any appropriate retries. Part of that is the
      StateWait stuff, that lets programs watch for events easily, and the
      node_reboot-with-events stuff that puts stated in control of nodes as they
  6. 12 Jan, 2004 1 commit
    • Leigh B. Stoller's avatar
      Hmm, this file dropped from previous commit. Added support for · 5378d87c
      Leigh B. Stoller authored
      handling PXEWAKUP timeouts, retrying 3 times and then forcing a power
      cycle.  Changed BOOTING event action to auto switch in and out of the
      special PXEKERNEL state machine that all local nodes use since all
      local nodes boot the same pxeboot kernel and talk to bootinfo (as
      directed to by dhcp).
  7. 07 Jan, 2004 1 commit
    • Leigh B. Stoller's avatar
      A set of debugging changes to allow running multiple stateds. This is · cf61f6f3
      Leigh B. Stoller authored
      probably imperfect, but better then nothing. New option, "-t tag"
      allows you to specify an arbitrary tag to match against the stated_tag
      of the nodes table. The stated invocation will only operate on nodes
      that match the tag, ignoring all events for other nodes. If
      unspecified, stated will operate on all nodes with a NULL tag. This is
      setup up at the beginning of time (or during a reload) saving the
      per-node tag in the $nodes hash. Each time an event arrives, check the
      tag in the table, ignoring the event if not a match.
      On signaled reload() must also be careful to throw away timeouts from
      the queue (and be careful not to set up new timeouts for ignored
      nodes).  So, this allows you to set the tag for a node in the DB, and
      then HUP stated so that it reloads it tables. That node will now be
      ignored by that stated.
      Also made some changes to debug mode. In debug mode, don't worry about
      the pidfile or the lockfile or checking for other running stated
      (which causes my debug version to exit! right away). Also, added a new
      -l option to turn of syslog output and just send it all to stdout with
      the debug output. -l can be only be used with -d of course.
      So what can I do with all this:
      	update nodes set stated_tag='lbs' where node_id='pc5';
      	sudo kill -HUP `cat /var/run/stated.pid`
      	sudo stated -d -l -t lbs
      Which tells the main stated to ignore pc5. Then I run a debugging
      stated that operates only on pc5. Later when done:
      	update nodes set stated_tag=NULL where node_id='pc5';
      	sudo kill -HUP `cat /var/run/stated.pid`
      Which tells the main stated to operate on pc5 again.
  8. 15 Oct, 2003 1 commit
    • Mike Hibler's avatar
      Uniform syslog'ing. Change everything I could find to use a syslog facility · cc6d6fa7
      Mike Hibler authored
      as defined in the defs-* file (e.g. "TBLOGFACIL=local2").  The default is
      "local5" which is what we are setup to use so you shouldn't need to mess
      with your defs- file!
      perl scripts just get this value configured in when configure is run.
      C programs get the value in two ways.  For programs that are intimate with
      the testbed infrastructure, and include "config.h", they just get it from
      that file.  For programs that we sometimes use outside the Emulab build
      environment (e.g., frisbee, capture) and that don't include config.h,
      the value is set via a "-DLOG_TESTBED=..." in the GNUmakefile build line.
      If the value isn't set, it defaults to what it used to be (usually LOG_USER).
      Still to do: healthd, hmcd (whose build doesn't seem to be completely
      integrated) and plabdaemon.in (since its icky python :-)
  9. 13 Oct, 2003 1 commit
  10. 10 Oct, 2003 1 commit
    • Mac Newbold's avatar
      New StateWait changes - the main point of all this is to move to our new · 2b2a306d
      Mac Newbold authored
      model of waiting for state changes. Before we were watching the database
      (which means we can only watch for terminal/stable/long-lived states, and
      have to poll the db). Now things that are waiting for states to change
      become event listeners, and watch the stream of events flow by, and don't
      have to do any polling. They can now watch for any state, and even
      sequences of states (ie a Shutdown followed by an Isup).
      To do this, there is now a cool StateWait.pm library that encapsulates the
      functionality needed. To use it, you call initStateWait before you start
      the chain of events (ie before you call node reboot). Then do your stuff,
      and call waitForState() when you're ready to wait. It can be told to
      return periodically with the results so far, and you can cancel waiting
      for things. An example program called waitForState is in
      testbed/event/stated/ , and can also be used nicely as a command line tool
      that wraps up the library functionality.
      This also required the introduction of a TBFAILED event that can be sent
      when a node isn't going to make it to the state that someone may be
      waiting for. Ie if it gets wedged coming up, and stated retries, but
      eventually gives up on it, it sends this to let things know that the node
      is hozed and won't ever come up.
      Another thing that is part of this is that node_reboot moves (back) to the
      fully-event-driven model, where users call node reboot, and it does some
      checks and sends some events. Then stated calls node_reboot in "real mode"
      to actually do the work, and handles doing the appropriate retries until
      the node either comes up or is deemed "failed" and stated gives up on it.
      This means stated is also the gatekeeper of when you can and cannot reboot
      a node. (See mail archives for extensive discussions of the details.)
      A big part of the motivation for this was to get uninformed timeouts and
      retries out of os_load/os_setup and put them in stated where we can make a
      wiser choice. So os_load and os_setup now use this new stuff and don't
      have to worry about timing out on nodes and rebooting. Stated makes sure
      that they either come up, get retried, or fail to boot. tbrestart also
      underwent a similar change.
  11. 21 Aug, 2003 1 commit
  12. 19 Jun, 2003 1 commit
    • Mac Newbold's avatar
      The new and fully functional rebooting-via-events stuff and the · 1daaa992
      Mac Newbold authored
      really-reboot-nodes-that-timeout stuff.
      NOTE: Until the timeout/retry stuff is gone from os_load/os_setup, it is
      disabled in stated. It will still only send email. But all the stuff is
      there and has been tested.
      NOTE: Until other things don't depend on the old behavior of node_reboot
      (when it returns, all nodes are in SHUTDOWN), the event stuff is disabled.
      Real mode is the default, and can be run by anyone.
      In short, this commit is new versions of stated and node_reboot that act
      almost exactly like the old ones. But I wanted to commit them before I go
      on making a bunch more changes, to have a checkpoint that I know works.
  13. 09 Jun, 2003 1 commit
  14. 06 Jun, 2003 1 commit
    • Mac Newbold's avatar
      First batch of changes for adding TBCOMMAND events. Currently, here's what · 71b82cc4
      Mac Newbold authored
      is supported:
      - stated listens for TBCOMMAND events, and currently handles REBOOT,
        POWEROFF, POWERON, and POWERCYCLE events. It does everything except make
        the actual calls to node_reboot and power. And it accepts batches of
        nodes instead of just single ones.
      - Timeouts were added to the db for these commands, with no timeout for
        the power ones (since the node can't hang during those), and a 15 second
        timeout from reboot until the SHUTDOWN state.
      - If a rebootimes out, it tries it again, up to 3 times. If it gets to
        three times without working, it sends mail to tbops and turns the
        machine off instead of continuing to reboot it. Right now I haven't
        made it do node_reboot -f or power cycle on retries, but it easily
      - Stuff to be done before they work: make node_reboot send an event
        instead of doing the work, and make a new script that has node_reboot's
        old guts. Note that this requires authentication in our events for these
        commands, and a way to make sure that the command that came in as an
        event was properly authenticated.
      - For future growth and expansion, it is set up so it should be relatively
        easy to add other commands that do different things, even if they take
        arbitrary params that aren't nodes or lists of nodes.
  15. 23 May, 2003 1 commit
    • Mac Newbold's avatar
      Fix two problems: · c8848155
      Mac Newbold authored
      1. timeouts for nodes weren't getting reset when they had a mode
      ransition, so they were timing out in shutdown after changing modes.
      2. It was still going back into a blocking wait, even though a signal had
      been recieved, and not quitting back up to the main loop to handle it.
  16. 22 May, 2003 2 commits
  17. 20 May, 2003 2 commits
    • Mac Newbold's avatar
      Back out last set of changes until we figure out some odd behaviors that I · 1934cc59
      Mac Newbold authored
      didn't see in testing. Specifically, why it pegs at 100% CPU after a
      while, and why it gets timeouts after it has removed the timeout from the
    • Mac Newbold's avatar
      Bunch of pretty good-sized changes to stated: · b438d5f5
      Mac Newbold authored
      1. Change from inefficient timeout search algo that ran once per second to
      a highly efficient priority queue method of managing timeouts. Now
      instead of checking every node's timestamps, we just look at the head of
      the queue, and it is often much less frequent than once a second, since we
      know how long we have until the next timeout.
      2. Start using a blocking poll for events, so I can sleep for long periods
      of time instead of having to wake up at least once a second to check for
      timeouts and events. Will set the block timeout for the shortest of: the
      time to send out the next batch of queued emails, the next time a timeout
      may occur, or when there are no mails waiting and no timeouts possible, 10
      minutes. Comes back as soon as an event comes in.
      3. Given the above two items, we no longer need a sleep(1) in our main
      One small glitch is in the progress of being fixed. When using blocking
      polls, things hang when trying to unregister from the event system. Not a
      big deal, just ^C twice to kill it. (May cause it to need two SIGUSR1's
      to get it to restart, too.)
      In the next update, look for:
       - Really take action on timeouts.
         - keep track of how many times we've retried, and notify if something
           may be wrong with the node.
         - Find out policy on taking action with timeouts.
           - Do it if the expt is in transition or the node is free
           - Probably don't touch if the expt is established.
           - Maybe? in active expt, send (good) email to expt owner on timeouts
      Related "coming soon" items:
      os_load/os_setup etc.:
       - Add the waitforstate stuff we've talked about
       - make os_load/os_setup use it
  18. 30 Apr, 2003 1 commit
  19. 28 Apr, 2003 1 commit
  20. 17 Apr, 2003 1 commit
    • Mac Newbold's avatar
      Add generic per-node state triggers to stated. You can put a trigger · 074149f5
      Mac Newbold authored
      on any node on any state, in any specific mode, or without any mode
      The imediate use of this is the FREENODE trigger. Now RELOADDONE adds
      a FREENODE trigger on the ISUP state, if the node is in the reloading
      expt. Then next time the node hits ISUP, it gets freed from the
      reloading expt.
      This fix solves the race where recently freed (and still rebooting)
      nodes get grabbed by an expt and get rebooted in a way that may hoze
      their FS's.
      Also fixed a problem that was making it load the db twice on startup.
  21. 19 Mar, 2003 1 commit
  22. 12 Mar, 2003 1 commit
  23. 08 Mar, 2003 1 commit
  24. 07 Mar, 2003 1 commit
    • Mac Newbold's avatar
      A few changes to stated: · 92fa4ae2
      Mac Newbold authored
       - fix bad indenting to a uniform 4 spaces (before was 2, 4 and 8 mixed)
       - Move ping-for-isup functionality into a separate script
       - Make sure every transition triggered by stated (directly or indirectly)
         sends an event, instead of taking shortcuts.
      This called for a new script, eventping, which just pings until the node
      is pingable, then sends an ISUP event. Stated runs this in the background
      where necessary, and nothing else should run it.
      Adding eventping meant modifying configure and the utils makefile, too.
  25. 26 Feb, 2003 1 commit
  26. 05 Feb, 2003 2 commits
  27. 29 Jan, 2003 3 commits
  28. 07 Jan, 2003 2 commits
  29. 20 Dec, 2002 1 commit
  30. 16 Dec, 2002 1 commit
    • Mac Newbold's avatar
      Fix the 1-event-per-second limitations. Poll until I don't get more · a77a1559
      Mac Newbold authored
      events. This may delay handling of other stuff that happens in my main
      loop, but not by too much. To prevent skew, everything (including reload
      frequency) is done strictly by seconds elapsed, not by iterations or
      I found that even polling for multiple events without sleeping, I could
      only handle a little over 1 per second when I was calling inuse/statetime
      for additional info on every event. Even though this only happens in the
      worst case (every event is wrong), it won't do. So I took that out. I'll
      probably end up adding a faster lookup of the info I need (mostly
      reservation, and what osid it thinks it is running). That change took it
      up to at least 4 per second (as fast as I could send them manually), more
      than 4x our previous performance. So we should be able to keep up now.
      Also, add the support for "announcements" to testbed ops when I die and
      such. (Been in a few days, but this is the first commit of it)
  31. 09 Dec, 2002 1 commit
  32. 03 Dec, 2002 1 commit
  33. 22 Nov, 2002 1 commit
  34. 14 Nov, 2002 1 commit
    • Mac Newbold's avatar
      Lots of changes. · 349db7bf
      Mac Newbold authored
      First, fix up the isup generation code. When a node/OS doesn't send its
      own isups, but is pingable, we need to fork and ping it, and send ISUP
      when it pings. The code was there, but was broken. This fixes it. The one
      time that it may cause errant messages is in modes other than MINIMAL.
      When we get BOOTING, we check if it needs isup generated. If we have to
      ping it, when it pings we send ISUP. This means that if we are really in
      NORMAL mode, we might send ISUP before the node sends REBOOTED (or TBSETUP
      in NORMALv1), and it would look funny. But that case will be really rare,
      since everything that sends REBOOTED or TBSETUP has no reason not to send
      ISUP itself.
      Second, after mailbombing myself a couple of times, Kirk and I decided I'd
      better put some throttling in the notification code that stated uses. So
      now it throttles itself and digests the messages if they're sent too close
      together. The first message it gets will get sent immediately. If the next
      one is long enough after that, it sends it immediately too. If a message
      comes too soon after sending one, we queue it up, and send it later
      after enough time has passed. Currently it is set to wait 5 seconds
      between messages, so it will send up to 12 per second, and wait no more
      than 5 seconds before sending a message that is queued up.
      (Something similar to this may be a nice thing in the rest of our stuff,
      but it was made a lot easier by the fact that stated already had a polling
      loop in it. Without that, you'd have to use alarms or some other weird
      thing, which would be painful.)