1. 29 Jul, 2004 3 commits
    • Leigh B. Stoller's avatar
      Rework TBGetSiteVar() slightly. Add optional second parameter $rptr to · 03403a55
      Leigh B. Stoller authored
      store the result in. When called this new way, the value goes into
      $rptr, and exit status is returned to caller instead. In addition,
      when called this way, all errors are non-fatal; it is up to the caller
      to decide what to do.
    • Leigh B. Stoller's avatar
      Two unrelated bug fixes (with some related cleanups and tweaks) · 9f4edbba
      Leigh B. Stoller authored
      * The first involves swapmod. When a swapmod on an active experiment fails,
        tbswap will reswap the experiment back to the original configuration. The
        problem is that it is reswapping it with the *new* virtual state of the
        experiment in the DB. It is not until later when control returns to
        swapexp that the virtual state is restored. This is plainly wrong, and in
        fact was causing the event scheduler grief cause it was starting up,
        reading the the virtual topo, which was different, wrong, and about to be
        blown away.
        I reorganized the modify section of swapexp so that virtual state is
        restored only when its a swapmod on a swapped experiment. On an active
        experiment, I moved that code down into tbswap, which will now does all
        of the virtual and physical state retore before it does the reswap back
        to the original experiment. Just for kicks, its also done if tbswap
        decides to swap the experiment cause of a fatal error.
        Cleanups: I changed $NoRecover to $CanRecover. My feeble brain cannot
        deal with !$NoRecover. I know, two knots make a wright for most people.
        Renderer: I was annoyed by the fact that we rerun the renderer on a
        failed swapmod. The original reason is that the renderer runs in the
        background and so vis_nodes cannot be saved with the rest of the virtual
        state tables cause the renderer might still be running when the user
        fires off the swapmod. Well, the hell with that. We lock the vis_nodes
        table anyway in the renderer during update, so we are certain to get a
        consistent snapshot. We store the renderer pid in the experiments table,
        so if the renderer was running, just fire off another one; mostly this is
        not going to happen. In addition, tbprerun no longer starts a new
        renderer when doing the swapmod; I start the new renderer later after
        swapmod succeeds. I might end up tweaking this a bit depending on what
        people notice as being different.
      * Termination changes to batchexp and swapexp: I've rearranged the
        termination code using an END block so that any uncontrolled exit from
        either batchexp or swapexp will go through the cleanup code, and
        hopefully insert a stats record, as well as not leave the experiment in
        some inbetween state. I've set the max DB retry count to zero in both
        cases, which means infinite retry. I've also added SIGTERM handlers to
        both so that again, we can kill a hung batch/swap and have it clean up
        things more or less. Note that END blocks are not caught when a signal
        causes the program to die; you have to catch it and then die() so that
        the END block is executed.
        Eventually, we need to clean up the various libraries so that we do not
        use DBQueryFatal(), but rather use DBQueryWarn(), and look for failure.
        Ditto for event system interface.
    • Leigh B. Stoller's avatar
  2. 15 Jul, 2004 2 commits
    • Leigh B. Stoller's avatar
      Couple of minor tweaks to make sure that experiment state events · d1a35ea9
      Leigh B. Stoller authored
      get sent properly; need to call TBdbfork(), and add a couple more
      event sends in libdb.
    • Leigh B. Stoller's avatar
      Overview: Add Event Groups: · ed964507
      Leigh B. Stoller authored
      	set g1 [new EventGroup $ns]
      	$g1 add  $link0 $link1
      	$ns at 60.0 "$g1 down"
      See the new advanced tutorial section on event groups for a better
      Changed tbreport to dump the event groups table when in summary mode.
      At the same time, I changed tbreport to use the recently added
      virt_lans:vnode and ip slots, decprecating virt_nodes:ips in one more
      place. I also changed the web interface to always dump the event and
      event group summaries.
      The parser gets a new file (event.tcl), and the "at" method deals with
      event group events by expanding them inline into individual events
      sent to each member. For some agents, this is unavoidable; traffic
      generators get the initial params in the event, so it is not possible
      to send a single event to all members of the group. Same goes for
      program objects, although program objects do default to the initial
      command now, at least on new images.
      Changed the event scheduler to load the event groups table. The
      current operation is that the scheduler expands events sent to a
      group, into a set of distinct events sent to each member of the
      group. At some point we proably want to optimize this by telling the
      agents (running on the nodes) what groups they are members of.
      Other News: Added a "mustdelay" slot to the virt_lans table so the
      parser can tell assign_wrapper that a link needs to be delayed, say if
      there are events or if the link is red/gred. Previously,
      assign_wrapper tried to figure this out by looking at the event list,
      etc. I have removed that code; see database-migrate for instructions
      on how to initialize this slot in existing experiments. assign_wrapper
      is free to ignore or insert delays anyway, but having the parser do
      this makes more sense.
      I also made some "rename" changes to the parser wrt queues and lans
      and links. Not really necessary, but I got sidetracked (for several
      hours!) trying to understand that rename stuff a little better, and
      now I do.
  3. 12 Jul, 2004 2 commits
  4. 20 May, 2004 1 commit
    • Leigh B. Stoller's avatar
      Add EventFork() to event.pm (perl interface to event system) and to · 116539b6
      Leigh B. Stoller authored
      the tail file of course. Called from TBdbfork() in libdb, EventFork
      resets the event handle so that the child does a reconnect. Note that
      I do not disconnect in the child since I have no idea what that is
      going to do to the parents connection to the elvind, as Elvin makes no
      mention of what to do in the presence of a process that forks after
      connecting to the event server. At the least, this avoids a bunch of
      warnings and errors from vnodesetup!
  5. 09 Apr, 2004 1 commit
  6. 19 Mar, 2004 1 commit
  7. 08 Mar, 2004 1 commit
  8. 11 Feb, 2004 2 commits
  9. 10 Feb, 2004 1 commit
  10. 09 Feb, 2004 1 commit
  11. 05 Feb, 2004 1 commit
  12. 16 Jan, 2004 2 commits
  13. 15 Jan, 2004 2 commits
    • Leigh B. Stoller's avatar
      Back part of Mac's last revision since it breaks the parser. · 56139163
      Leigh B. Stoller authored
      No idea, just making it work.
    • Mac Newbold's avatar
      libdb changes: · 78ad260c
      Mac Newbold authored
      - add functions to recursively dump hashes and arrays into a string
        suitable for printing as debugging output (great for data structures)
      - add three new trigger strings
      - add 'use strict', do corresponding cleanup
      stated changes:
      - move special-cased stuff in handleEvent for PXEBOOTING and BOOTING into
      - clarify (via comments) the existing kinds of triggers and which ones run
        when, and add a new kind (global "any-mode" triggers). We already had
        per-node mode-specific, per-node any-mode, and global mode-specific
        triggers. Now you can have a trigger that is good for any mode in a
        given state, that can be overridden on a mode-specific basis. This is
        great for PXEBOOTING, BOOTING, and ISUP, since they each have a trigger
        list that should be run regardless of what mode you're in. Now they only
        require 3 entries instead of 3*N that have to be maintained per mode.
           # A note about triggers:
           # "per-node" triggers only affect their specific node in a
           # particular mode/state, and are run first of all. "global"
           # triggers are triggers for a given mode/state that affect all
           # nodes, and are run after any per-node triggers. "Any-mode"
           # triggers are tied to a state, and occur in that state in any
           # mode. The any-mode triggers are over-ridden by global triggers,
           # and if an "Any-mode" trigger for state XYZ exists as well as a
           # global trigger for mode FOOBAR state XYZ, then when I arrive in
           # XYZ any per-node triggers will be run. Then, if I'm in mode
           # FOOBAR, only the global trigger will run. If I'm in any other
           # mode, only the any-mode trigger will run.
           # (our "*" is stored as $TBANYMODE)
           # Per-node triggers have a specific node_id
           # Global triggers have "*" as the node_id
           # Any-mode triggers have "*" as the mode, and can be global or per-node
        The updated table looks like this in the accompanying change to
      | node_id | op_mode  | state      | trigger               |
      | *       | *        | BOOTING    | BOOTING, CHECKGENISUP |
      | *       | *        | ISUP       | RESET                 |
      | *       | *        | PXEBOOTING | PXEBOOT               |
      | *       | RELOAD   | RELOADDONE | RESET, RELOADDONE     |
      | *       | ALWAYSUP | SHUTDOWN   | ISUP                  |
      - I also cleaned up the functions that add, get, and delete triggers.
        Before, the get function didn't include global triggers. Now it does,
        and has an option to just get the per-node triggers. Add and delete are
        still just per-node, of course.
      - Also found and fixed some little bugs while I was in there. (global
        triggers not taking a list,
      These changes are me getting ready to re-add all the changes I made months
      ago in order to do a before-and-after experiment for my thesis. Between
      now and the end of next week I'll be working on taking before numbers,
      patching stated with the changes, and getting after numbers.
      The problems I'm trying to replicate are the problems and slowdowns we
      used to get when os_{load,setup} would reboot a node, thinking it had
      timed out, when it really didn't know whether it was making progress or
      not. The fix includes making os_{load,setup} depend on stated to watch for
      progress and timeouts, and do any appropriate retries. Part of that is the
      StateWait stuff, that lets programs watch for events easily, and the
      node_reboot-with-events stuff that puts stated in control of nodes as they
  14. 12 Jan, 2004 1 commit
    • Leigh B. Stoller's avatar
      Add more node state machine constants. · 899be6af
      Leigh B. Stoller authored
      Add constants for the osids describing the FreeBSD and Frisbee MFSs.
      Complete redo of TBBooWhat to match the changes in bootinfo. Look
      there for description of new boot protocol (how TBBooWhat now works).
  15. 25 Dec, 2003 1 commit
  16. 23 Dec, 2003 1 commit
    • Leigh B. Stoller's avatar
      Add new state PXEWAIT to be explained later ... · 22ee2594
      Leigh B. Stoller authored
      Change ptopgen to look at the eventstate of a node; a node is not considered
      free unless it is ISUP or PXEWAIT.
      Add TBAvailablePCs() to libdb, removing the corresponding code from
      assign_wrapper. This routine does the equiv of ptopgen, returning the
      number of PCs that are available for use (looking at eventstate).
      Change TBFreePCs in the web interface accordingly.
      The above changes correspond to an upcoming change in stated.
  17. 20 Dec, 2003 1 commit
  18. 18 Dec, 2003 1 commit
    • Leigh B. Stoller's avatar
      First try at solving the problem of validating user input for the · 8dbead16
      Leigh B. Stoller authored
      zillions of DB fields that we have to set. My solution was to add a
      meta table that describes what is a legal value for each table/slot
      for which we take from user input. The table looks like this right
      now, but is likely to adapt as we get more experience with this
      approach (or it might get tossed if it turns out to be a pain in the
      	CREATE TABLE table_regex (
      	  table_name varchar(64) NOT NULL default '',
      	  column_name varchar(64) NOT NULL default '',
      	  column_type enum('text','int','float') default NULL,
      	  check_type enum('regex','function','redirect') default NULL,
      	  check tinytext NOT NULL,
      	  min int(11) NOT NULL default '0',
      	  max int(11) NOT NULL default '0',
      	  comment tinytext,
      	  UNIQUE KEY table_name (table_name,column_name)
      	) TYPE=MyISAM;
      Entries in this table look like this:
      Which says that the vname slot of the virt_nodes table (which we trust the
      user to give us in some form) is a text field to be checked with the given
      regex (perlre of course), and that the min/max length of the text field is
      1 and 32 chars respectively.
      Now, you wouldn't want to write the same regex over and over, and since we
      use the same fields in many tables (like pid, eid, vname, etc) there is an
      option to redirect to another entry (recursively). So, for "PID" I do this:
      which redirects to:
      And, for many fields you just want to describe generically what could go
      into it. For that I have defined some default fields. For example, a user
      which redirects to:
      and this says that a tinytext (in our little corner of the database
      universe) field can have printable characters (but not a newline), and
      since its a tinytext field, its maxlen is 256 chars.
      You also have integer fields, but these are little more irksome in the
      and you would use this anyplace you do not care about the min/max values
      being something specific in the tinyint range. The range for a float is of
      course stated as an integer, and thats kinda bogus, but we do not have many
      floats, and they generally do not take on specific values anyway.
      A note about the min/max fields and redirecting. If the initial entry has
      non-zero min/max fields, those are the min mac fields used. Otherwise they
      come from the default. So for example, you can do this:
      So, you can redirect to the standard "tinyint" regular expression, but you
      still get to define min/max for the specific field.
      Isn't this is really neat and really obtuse too? Sure, you can say it.
      Anyway, xmlconvert now sends all of its input through these checks (its
      all wrapped up in library calls), and if a slot does not have an entry, it
      throws an error so that we are forced to define entries for new slots as we
      add them.
      In the web page, I have changed all of the public pages (login, join
      project, new project, and a couple of others) to also use these checks.
      As with the perl code, its all wrapped up in a library. Lots more code
      needs to be changed of course, but this is a start.
  19. 16 Dec, 2003 1 commit
  20. 10 Dec, 2003 1 commit
  21. 07 Dec, 2003 1 commit
    • Shashi Guruprasad's avatar
      Minimal NSE related changes on the mainbed so that I can work · 425b4e47
      Shashi Guruprasad authored
      in the dev tree. I'm tired of problems on the mini that wastes
      my time.
      Changes include 2 new tmcd commands: tmcc routelist returns
      the routes for all the vnodes hosted on a pnode. tmcc role
      returns the role of a reserved node, like 'virthost' or
      'simhost.  tmcc ifconfig now reports an RTABID field which
      is calculated in assign wrapper. All the new changes
      in assign wrapper will be checked in after I finish testing.
      All the DB changes are in: simnode_capacity in node_types, rtabid in
      interfaces and veth_interfaces. New NSE event_objtype and NSEEVENT
      event_eventtype. Changed the erole field in the reserved table
      to have 'simhost' instead of 'simnode'. Changed the correspoding
      libdb subroutines.
  22. 18 Nov, 2003 1 commit
  23. 17 Nov, 2003 1 commit
    • Leigh B. Stoller's avatar
      Merge the two state machines (batchstate and state) into a single · 2025e0bd
      Leigh B. Stoller authored
      state machine (state). All of the stuff that was previously handled by
      using batchstate is now embedded into the one state machine. Of
      course, these mostly overlapped, so its not that much of a change,
      except that we also redid the machine, adding more states (for
      example, modify phases are now explicit. To get a picture of the
      actual state machine, on boss:
      		stategraph -o newstates EXPTSTATE
      		gv newstates.ps
      Things to note:
      * The "batchstate" slot of the experiments table is now used solely to
        provide a lock for batch daemon. A secondary change will be to
        change the slot name to something more appropriate, but it can
        happen anytime after this new stuff is installed.
      * I have left expt_locked for now, but another later change will be to remove
        expt_locked, and change it to active_busy or some such new state name in
        the state machine. I have removed most uses of expt_locked, except those
        that were necessary until there is a new state to replace it.
      * These new changes are an implementation of the new state machine,
        but I have not done anything fancy. Most of the code is the same as
        it was before.
      * I suspect that there are races with the batch daemon now, but they
        are going to be rare, and the end result is probably that a
        cancelation is delayed a little bit.
  24. 05 Nov, 2003 1 commit
    • Leigh B. Stoller's avatar
      Frontend and parser portion of two event system changes: · 091a0b62
      Leigh B. Stoller authored
      * Generate a shared secret key for the event system. This key is
        stored into the DB, and passed to the node via tmcd. It is also
        stashed into a file in the experiment directory (can be accessed
        only by the project/group members). The key is used to attach a
        HMAC (hashed message authentication) to each event, which is checked
        by the receivers to ensure that the event is not bogus. More details
        on this later when I commit the event library/client changes.
      * Added "virt_programs" table to store info about each program object
        defined by the user. The intent is to no longer send the command
        string in the event, but to fix it in the DB, and transfer it via
        tmcd. This removes our "remote execution facility" which was always
        a bad idea (we have ssh for that, and that is a lot more secure then
        the event system!).
        Note that for the time being we need to continue send the command in
        the event because of old images, but the new images will now ignore
        that part of the event.
  25. 13 Oct, 2003 1 commit
  26. 10 Oct, 2003 1 commit
    • Mac Newbold's avatar
      New StateWait changes - the main point of all this is to move to our new · 2b2a306d
      Mac Newbold authored
      model of waiting for state changes. Before we were watching the database
      (which means we can only watch for terminal/stable/long-lived states, and
      have to poll the db). Now things that are waiting for states to change
      become event listeners, and watch the stream of events flow by, and don't
      have to do any polling. They can now watch for any state, and even
      sequences of states (ie a Shutdown followed by an Isup).
      To do this, there is now a cool StateWait.pm library that encapsulates the
      functionality needed. To use it, you call initStateWait before you start
      the chain of events (ie before you call node reboot). Then do your stuff,
      and call waitForState() when you're ready to wait. It can be told to
      return periodically with the results so far, and you can cancel waiting
      for things. An example program called waitForState is in
      testbed/event/stated/ , and can also be used nicely as a command line tool
      that wraps up the library functionality.
      This also required the introduction of a TBFAILED event that can be sent
      when a node isn't going to make it to the state that someone may be
      waiting for. Ie if it gets wedged coming up, and stated retries, but
      eventually gives up on it, it sends this to let things know that the node
      is hozed and won't ever come up.
      Another thing that is part of this is that node_reboot moves (back) to the
      fully-event-driven model, where users call node reboot, and it does some
      checks and sends some events. Then stated calls node_reboot in "real mode"
      to actually do the work, and handles doing the appropriate retries until
      the node either comes up or is deemed "failed" and stated gives up on it.
      This means stated is also the gatekeeper of when you can and cannot reboot
      a node. (See mail archives for extensive discussions of the details.)
      A big part of the motivation for this was to get uninformed timeouts and
      retries out of os_load/os_setup and put them in stated where we can make a
      wiser choice. So os_load and os_setup now use this new stuff and don't
      have to worry about timing out on nodes and rebooting. Stated makes sure
      that they either come up, get retried, or fail to boot. tbrestart also
      underwent a similar change.
  27. 09 Oct, 2003 1 commit
    • Leigh B. Stoller's avatar
      Reorg of two aspects of node update. · 2641af4d
      Leigh B. Stoller authored
      * install-rpm, install-tarfile, spewrpmtar.php3, spewrpmtar.in: Pumped
        up even more! The db file we store in /var/db now records both the
        timestamp (of the file, or if remote the install time) and the MD5
        of the file that was installed. Locally, we can get this info when
        accessing the file via NFS (copymode on or off). Remote, we use wget
        to get the file, and so pass the timestamp along in the URL request,
        and let spewrpmtar.in determine if the file has changed. If the
        timestamp it gets is >= to the timestamp of the file, an error code
        of 304 (Not Modifed) is returned. Otherwise the file is returned.
        If the timestamps are different (remote, server sends back an actual
        file), the MD5 of the file is compared against the value stored. If
        they are equal, update the timestamp in the db file to avoid
        repeated MD5s (or server downloads) in the future. If the MD5 is
        different, then reinstall the tarball or rpm, and update the db file
        with the new timestamp and MD5. Presto, we have auto update capability!
        Caveat: I pass along the old MD5 in the URL, but it is currently
        ignored. I do not know if doing the MD5 on the server is a good
        idea, but obviously it is easy to add later. At the moment it
        happens on the node, which means wasted bandwidth when the timestamp
        has changed, but the file has not (probably not something that will
        happen in typical usage).
        Caveat: The timestamp used on remote nodes is the time the tarfile
        is installed (GM time of course). We could arrange to return the
        timestamp of the local file back to the node, but that would mean
        complicating the protocol (or using an http header) and I was not in
        the mood for that. In typical usage, I do not think that people will
        be changing tarfiles and rpms so rapidly that this will make a
        difference, but if it does, we can change it.
      * node_update.in, client side watchdog, and various web pages:
        Deflated node_update, removing all of the older ssh code. We now
        assume that all nodes will auto update on a periodic basis, via the
        watchdog that runs on all client nodes, including plab nodes.
        Changed the permission check to look for new UPDATE permission (used
        to be UPDATEACCOUNT). As before, it requires local_root or better.
        The reason for this is that node_update now implies more than just
        updating the accounts/mounts. The web pages have been changed to
        explain that in addition to mounts/accounts, rpms and tarfiles will
        also be updated. At the moment, this is still tied to a single
        variable (update_accounts) in the nodes table, but as Kirk requested
        at the meeting, it will probably be nice to split these out in the
        Added the ability to node_update a single node in an experiment (in
        addition to all nodes option on the showexp page). This has been
        added to the shownode webpage menu options.
        Changed locking code to use the newer wrapper states, and to move
        the experiment to RUNNING_LOCKED until the update completes. This is
        to prevent mayhem in the rest of the system (which could be dealt
        with, but is not worth the trouble; people have to wait until their
        initiated update is complete, before they can swap out the
        Added "short" mode to shownode routine, equiv to the recently added
        short mode for showexp. I use this on the confirmation page for
        updating a single node, giving the user a couple of pertinent (feel
        good) facts before they comfirm.
  28. 07 Oct, 2003 1 commit
  29. 01 Oct, 2003 1 commit
  30. 30 Sep, 2003 1 commit
    • Leigh B. Stoller's avatar
      Up to now we have had two state variables associated with an experiment, · 4269dad1
      Leigh B. Stoller authored
      plus a lock field. The lock field was a simple "experiment locked, go away"
      slot that is easy to use when you do not care about the actual state that
      an experiment is in, just that it is in "transition" and should not be
      messed with.
      The other two state variables are "state" and "batchstate". The former
      (state) is the original variable that Chris added, and was used by the tb*
      scripts to make sure that the experiment was in the state each particular
      script wanted them to be in. But over time (and with the addition of so
      much wrapper goo around them), "state" has leaked out all over the place to
      determine what operations on an experiment are allowed, and if/when it
      should be displayed in various web pages. There are a set of transition
      states in addition to the usual "active", "swapped", etc like "swapping"
      that make testing state a pain in the butt.
      I added the other state variable ("batchstate") when I did the batch
      system, obviously! It was intended as a wrapper state to control access to
      the batch queue, and to prevent batch experiments from being messed with
      except when it was really okay (for example, its okay to terminate a
      swapped out batch experiment, but not a swapped in batch experiment since
      that would confuse the batch daemon). There are fewer of these states, plus
      one additional state for "modifying" experiments.
      So what I have done is change the system to use "batchstate" for all
      experiments to control entry into the swap system, from the web interface,
      from the command line, and from the batch daemon. The other state variable
      still exists, and will be brutally pushed back under the surface until its
      just a vague memory, used only by the original tb* scripts. This will
      happen over time, and the "batchstate" variable will be renamed once I am
      convinced that this was the right thing to do and that my changes actually
      work as intended.
      Only people who have bothered to read this far will know that I also added
      the ability to cancel experiment swapin in progress. For that I am using
      the "canceled" flag (ah, this one was named properly from the start!), and
      I test that at various times in assign_wrapper and tbswap. A minor downside
      right now is that a canceled swapin looks too much like a failed swapin,
      and so tbops gets email about it. I'll fix that at some point (sometime
      after the boss complains).
      I also cleaned up various bits of code, replacing direct calls to exec
      with calls to the recently improved SUEXEC interface. This removes
      some cruft from each script that calls an external script.
      Cleaned up modifyexp.ph3 quite a bit, reformatting and indenting.
      Also fixed to not run the parser directly! This was very wrong; should
      call nscheck instead. Changed to use "nobody" group instead of group
      flux (made the same change in nscheck).
      There is a script in the sql directory called newstates.pl. It needs
      to be run to initialize the batchstate slot of the experiments table
      for all existing experiments.
  31. 29 Sep, 2003 3 commits