1. 20 Oct, 2005 2 commits
  2. 19 Oct, 2005 2 commits
    • Timothy Stack's avatar
      · bd627836
      Timothy Stack authored
      Some event system changes for linktest and any future things we want to
      run with the event system in the swapin path:
      
      	* event/linktest/linktest_control.in: Let linktest be run while
      	the experiment is activating.
      
      	* event/sched/event-sched.c, event/sched/rpc.h,
      	event/sched/rpc.cc: Don't wait for the experiment to become active
      	before loading the eventlist so any system defined agents are
      	available to use.  Don't start time unless the experiment is
      	already active, let boss do it otherwise.  Send out COMPLETE
      	events so 'tevc' can listen for them.
      
      	* event/sched/timeline-agent.c: Send a complete event even if the
      	timeline is empty.
      
      	* event/tbgen/tevc.c: Add a '-w' option so that tevc can wait for
      	an event that sends back a COMPLETE.
      
      	* tbsetup/tbswap.in: Explicitly send an event to start event time.
      bd627836
    • Leigh B. Stoller's avatar
      As per Jay's request, change the message at the top of the activity log · aeed9520
      Leigh B. Stoller authored
      to include the date and the experiment index.
      aeed9520
  3. 06 Oct, 2005 2 commits
  4. 20 May, 2005 1 commit
    • Leigh B. Stoller's avatar
      Checkpoint some robot changes. · 5f67fe09
      Leigh B. Stoller authored
      * New robot event listener:
      
          * It is intended to be started and stopped from the experiment
            swapin path instead of as a global daemon. It takes the pid/eid
            of the experiment, and will deal with events only for those
            nodes that are allocated to the experiment. We have some long
            range plans of time sharing the robot lab, so I figured we might
            as get a little bit of a start on that.
      
          * Once it fires up, it subscribes to the usual assortment of
            events, just like the loclistener does.
      
          * It then binds a socket on which to listen for connections from
            the web server.
      
          * Then it loops, looking for events and for connections from the
            web server. Connections from the web server are for forwarding
            the event stream in real time to whatever applets are currently
            viewing the robot lab.
      
          * As each event comes in, it is parsed, entered into the DB (nodes
            and location_info table), and fired out (in a textua...
      5f67fe09
  5. 12 May, 2005 1 commit
    • Leigh B. Stoller's avatar
      Checkpoint the rest of my changes to support swapmod of both ElabInElab and · 6eff9de6
      Leigh B. Stoller authored
      Firewalled experiments (see tbsetup/elabinelab.in for the other stuff).
      
      * To support firewalled experiments, needed to add a new virt_firewalls
        table to split the existing firewalls table up, which included both
        virtual and physical stuff. There are the usual frontend changes and a
        few other things scattered around, including tmcd.c.
      
      * The firewall code in tbswap got some beefing up to support adding and
        deleting nodes from the its special control net vlan. Note that I have
        not made any progress on containment of deleted nodes, just as we do not
        do anything now for teardown (unless its paniced, in which case the
        experiment cannot be modified anyway).
      
      * ptopgen and assign_wrapper got some interesting modifications: Unlike
        regular swapmod, we cannot just tear down all the vlans since that would
        interrupt everything inside the inner elab. Instead, leave the vlans as
        is. The problem is that when assign runs, it can just as easily pick
        different interfaces on the same nodes, which would be a royal pain in
        the ass to deal with! So, ptopgen got a new option (-u) that assign
        wrapper uses to tell ptopgen that it should prune out unused interfaces
        from nodes that are already allocated to the experiment. This is, at
        best, as pathetically gross hack, but it makes sure that all the
        interfaces stay the same across swapmods.
      
      * The unrelated revision of elabinelab has a bunch of new code for adding
        and deleting nodes from the inner elab. Mostly it deals with dhcpd (inner
        and outer, waiting for nodes to reboot, etc). It also deals with updating
        the vlans table in the DB, pruning out any nodes (ports) that are deleted
        but for which there are still interfaces in existing vlans. Said ports
        are them moved back to the default vlan with calls to snmpit. Also under
        another revision a a couple of weeks ago are the web interface changes to
        support the newnode MFS inside an inner Emulab.
      
      * swapexp and endexp got some more checks for firewalled and paniced
        experiments, which were missing.
      6eff9de6
  6. 27 Jan, 2005 1 commit
    • Leigh B. Stoller's avatar
      Add admission control checks to both tbswap and to assign_wrapper.in. · b5c775be
      Leigh B. Stoller authored
      There are two different kinds of admission control checks, hence two
      locations.
      
      * From tbswap, easy checks are made (like, too many nodes of a real
        type), or trying to the use robot lab when its closed.
      
      * From assign_wrapper.in. Because of virtual types, and because assign
        is the ultimate descision maker, we need to pass into assign per
        physical type limits, and let it deal with how virtual types map to
        physical types, and whether it exceeds the limits. So, when called
        from assign_wrapper, libadminctrl returns a hash of types and the
        counts the current experiment is limited to. That is put into the
        ptop file where assign deals with it.
      
      If admission control goes sour, then just turn off the sitevar for it:
      'swap/use_admission_control'
      b5c775be
  7. 22 Dec, 2004 1 commit
  8. 16 Dec, 2004 1 commit
    • Leigh B. Stoller's avatar
      The panic button ... · 87dd2e60
      Leigh B. Stoller authored
      * tbsetup/panic.in: New backend script to implement the panic button
        feature. When used, it will cut the severe the connection to the
        firewall node by using snmpit to disable the port. Sets the panic
        bit (and date) in the experiments table, and changes the state of
        the experiment from "active" to "paniced" to ensure that the
        experiment cannot be messed with (swapped out or modified). Sends
        email to tbops when the panic button is pressed.
      
        Used with -r option, reverses the above. State is set back to
        active, the panic bit is cleared, and the port is renabled with
        snmpit.
      
      * tbsetup/tbswap.in: During swapout, a firewalled experiment that has
        been paniced will get a cleaning; The nodes are powered off, then
        the osids for all the nodes are reset (with os_select) so that they
        will boot the MFS, and then the nodes are powered on. Then the
        control network is turned back on, and then I wait for the nodes to
        reboot (this is simply cause we do not record in the DB th...
      87dd2e60
  9. 09 Dec, 2004 1 commit
  10. 07 Dec, 2004 1 commit
  11. 06 Dec, 2004 1 commit
  12. 03 Dec, 2004 1 commit
    • Mike Hibler's avatar
      First past at integration of VLAN firewall: · 719794dc
      Mike Hibler authored
      1. Do all the magic snmpit-speak at the beginning of a swapin.
      
      2. Undo all the magic snmpit-speak right before nfree in a swapout.
         THIS IS CURRENTLY UNSAFE because I don't yet put all the firewalled
         nodes into "purgatory" before dropping the firewall.  That is yet
         to come.
      
      3. Disallow modification of a firewalled experiment til I figure out
         how to do it safely.
      719794dc
  13. 15 Nov, 2004 1 commit
    • Leigh B. Stoller's avatar
      A bunch of ElabInELab changes. · 10b116e0
      Leigh B. Stoller authored
      * snmpit: When ElabInELabis true, use the routines in the new
        snmpit_remote.pm library for setting up and tearing down vlans for an
        experiment. At present, only these two operations are proxied out to
        the outer emulab.
      
      * snmpit_remote.pm: A new little library that uses the XMLPRC server on
        the outer emulab to setup and destroy vlans for an inner experiment.
        This code is used from snmpit (see above).
      
      * snmpit_lib.pm: A couple of minor changes for the server side of the
        proxy operation.
      
      * snmpit.proxy.in: A new perl module that is invoked from the RPC
        server.  This proxy sets up and tears down vlans for an inner elab.
        The basic model is that the container experiment will have lots of
        vlans for various individual experiments running on the inner emulab.
      
      * swapexp: A couple of minor elabinelab hacks.
      
      * tbswap: For elabinelab experiments, reconfig/restart dhcpd when
        tearing down the experiment, and call out to new elabinelab script
        when setting up an el...
      10b116e0
  14. 04 Oct, 2004 1 commit
  15. 09 Aug, 2004 1 commit
    • Leigh B. Stoller's avatar
      Some cleanups and performance improvements: · f604dc33
      Leigh B. Stoller authored
      * Be more selective about what lists are regenerated; we were generating
        way too many lists each time called. When calling from tbswap, use new
        -t option to generate just the active lists. When called from setgroups,
        use -p option to generate lists just for the project. Add update option
        for when user changes email address (and all lists really do need to be
        regenerated).
      
      * Add "diff" processing. Instead of blindly firing each new list over to
        ops with ssh, store a copy of all of the lists in
        /usr/testbed/lists. After we generate the new list, diff it against the
        stored copy. If the same, skip it. Otherwise stash new copy and fire it
        over. This should reduce the wait times by quite a bit since the lists
        rarely change (except for the activity lists of course).
      
      * Add -n (impotent) option for debugging; skips the ssh over to ops.
      
      * Reorg a lot of stuff; it was getting hard to follow.
      f604dc33
  16. 29 Jul, 2004 1 commit
    • Leigh B. Stoller's avatar
      Two unrelated bug fixes (with some related cleanups and tweaks) · 9f4edbba
      Leigh B. Stoller authored
      * The first involves swapmod. When a swapmod on an active experiment fails,
        tbswap will reswap the experiment back to the original configuration. The
        problem is that it is reswapping it with the *new* virtual state of the
        experiment in the DB. It is not until later when control returns to
        swapexp that the virtual state is restored. This is plainly wrong, and in
        fact was causing the event scheduler grief cause it was starting up,
        reading the the virtual topo, which was different, wrong, and about to be
        blown away.
      
        I reorganized the modify section of swapexp so that virtual state is
        restored only when its a swapmod on a swapped experiment. On an active
        experiment, I moved that code down into tbswap, which will now does all
        of the virtual and physical state retore before it does the reswap back
        to the original experiment. Just for kicks, its also done if tbswap
        decides to swap the experiment cause of a fatal error.
      
        Cleanups: I changed $NoRecover to $CanRecover. My feeble brain cannot
        deal with !$NoRecover. I know, two knots make a wright for most people.
      
        Renderer: I was annoyed by the fact that we rerun the renderer on a
        failed swapmod. The original reason is that the renderer runs in the
        background and so vis_nodes cannot be saved with the rest of the virtual
        state tables cause the renderer might still be running when the user
        fires off the swapmod. Well, the hell with that. We lock the vis_nodes
        table anyway in the renderer during update, so we are certain to get a
        consistent snapshot. We store the renderer pid in the experiments table,
        so if the renderer was running, just fire off another one; mostly this is
        not going to happen. In addition, tbprerun no longer starts a new
        renderer when doing the swapmod; I start the new renderer later after
        swapmod succeeds. I might end up tweaking this a bit depending on what
        people notice as being different.
      
      * Termination changes to batchexp and swapexp: I've rearranged the
        termination code using an END block so that any uncontrolled exit from
        either batchexp or swapexp will go through the cleanup code, and
        hopefully insert a stats record, as well as not leave the experiment in
        some inbetween state. I've set the max DB retry count to zero in both
        cases, which means infinite retry. I've also added SIGTERM handlers to
        both so that again, we can kill a hung batch/swap and have it clean up
        things more or less. Note that END blocks are not caught when a signal
        causes the program to die; you have to catch it and then die() so that
        the END block is executed.
      
        Eventually, we need to clean up the various libraries so that we do not
        use DBQueryFatal(), but rather use DBQueryWarn(), and look for failure.
        Ditto for event system interface.
      9f4edbba
  17. 28 May, 2004 1 commit
  18. 26 Apr, 2004 1 commit
    • Leigh B. Stoller's avatar
      Changes to exit status stuff to reflect recent changes by Rob to how · 1c4a613c
      Leigh B. Stoller authored
      assign exits (exit codes).
      
      * in assign_wrapper, no longer return any status from assign to the
        caller. This was pointless. Instead, return 0 on success, 1 on
        controlled error, and -1 on uncontrolled error (die() called
        someplace). Add in CANRECOVER bit whenever the wrapper exits, even
        if uncontrolled, by putting in an END block to catch the die. This
        should prevent certain cases where a swapmod error would be flagged
        as not recoverable.
      
      * Remove most of the assign output processing since we no longer
        return its codes. Still print a portion of it to the log though.
      
      * Change call to fatal() in assign_wrapper; do not pass an exitcode
        since in every case it was the same damn thing!
      
      * Change tbswap to no longer carry assign_wrapper exit code to its
        exit.
      
      * Change the batch daemon to treat all errors as continuable (keep
        batch queued) unless exit code is -1. We will need to revisit this a
        bit perhaps, when Rob adds precheck code.
      1c4a613c
  19. 16 Feb, 2004 1 commit
  20. 10 Feb, 2004 1 commit
  21. 09 Jan, 2004 1 commit
    • Shashi Guruprasad's avatar
      Changes to do auto re-swap of expts with simnodes when an nse on a simhost · 375f87c1
      Shashi Guruprasad authored
      (or more than one simhost) is unable to keep up with real-time. It includes
      changes to assign_wrapper to handle swap modify for simnodes, the simple
      algorithm in nseswap that bumps up the nodeweight of simnodes being hosted
      on a simhost that reports "can't keep up with real-time" (aka nse violation),
      ptopgen and sim.tcl to prefer nodes that already have the FBSD-NSE image.
      Also, changes to other files to send out NSESWAP event.
      
      One unrelated change: We now have per-swapin .top files and assign.log
      files along with .ptop files. This helps in debugging across multiple
      swapins since files remain in the form of
      <pid>-<eid>-<process_id>.{top,ptop} and assign-<pid>-<eid>-<process_id>.log
      Also useful for archiving.
      375f87c1
  22. 08 Jan, 2004 1 commit
  23. 13 Dec, 2003 1 commit
  24. 01 Dec, 2003 1 commit
    • Robert Ricci's avatar
      New scripts: tarfiles_setup, fetchtar.proxy, and webtarfiles_setup . · c0c6547c
      Robert Ricci authored
      The idea is to give us hooks for grabbing experimenters' tarballs (and
      RPMs) from locations other than files on ops. Mainly, to remove
      another dependance on users having shells on ops.
      
      tarfiles_setup supports fetching files from http and ftp URLs right
      now, through wget. It places them into the experiment directory, so
      that they'll go away when the experiment is terminated, and the rest
      of the chain (ie. downloading to clients and os_setup's checks)
      remains unchaged.  It is now tarfiles_setup's job to copy tarballs and
      RPMs from the virt_nodes table to the nodes table for allocated nodes.
      This way, it can translate URLs into the local filenames it
      constructs. It get invoked from tbswap.
      
      Does the actual fetching over on ops, running as the user, with
      fetchtar.proxy.
      
      Should be idempotent, so we should be able to give the user a button
      to run webtarfiles_setup (none exists yet) yet to 'freshen' their
      tarballs. (We'd also have to somehow let the experiment's nodes know
      they need to re-fetch their tarballs.)
      
      One funny side effect of this is that the separator in
      virt_nodes.tarfiles is now ';' instead of ':' like nodes.tarballs,
      since we can now put URLs in the former. Making these consistent is a
      project for another day.
      c0c6547c
  25. 17 Nov, 2003 1 commit
    • Leigh B. Stoller's avatar
      Merge the two state machines (batchstate and state) into a single · 2025e0bd
      Leigh B. Stoller authored
      state machine (state). All of the stuff that was previously handled by
      using batchstate is now embedded into the one state machine. Of
      course, these mostly overlapped, so its not that much of a change,
      except that we also redid the machine, adding more states (for
      example, modify phases are now explicit. To get a picture of the
      actual state machine, on boss:
      
      		stategraph -o newstates EXPTSTATE
      		gv newstates.ps
      
      Things to note:
      
      * The "batchstate" slot of the experiments table is now used solely to
        provide a lock for batch daemon. A secondary change will be to
        change the slot name to something more appropriate, but it can
        happen anytime after this new stuff is installed.
      
      * I have left expt_locked for now, but another later change will be to remove
        expt_locked, and change it to active_busy or some such new state name in
        the state machine. I have removed most uses of expt_locked, except those
        that were necessary until there is a ...
      2025e0bd
  26. 30 Sep, 2003 1 commit
    • Leigh B. Stoller's avatar
      Up to now we have had two state variables associated with an experiment, · 4269dad1
      Leigh B. Stoller authored
      plus a lock field. The lock field was a simple "experiment locked, go away"
      slot that is easy to use when you do not care about the actual state that
      an experiment is in, just that it is in "transition" and should not be
      messed with.
      
      The other two state variables are "state" and "batchstate". The former
      (state) is the original variable that Chris added, and was used by the tb*
      scripts to make sure that the experiment was in the state each particular
      script wanted them to be in. But over time (and with the addition of so
      much wrapper goo around them), "state" has leaked out all over the place to
      determine what operations on an experiment are allowed, and if/when it
      should be displayed in various web pages. There are a set of transition
      states in addition to the usual "active", "swapped", etc like "swapping"
      that make testing state a pain in the butt.
      
      I added the other state variable ("batchstate") when I did the batch
      system, obvious...
      4269dad1
  27. 18 Sep, 2003 2 commits
  28. 27 Aug, 2003 1 commit
  29. 22 Aug, 2003 1 commit
  30. 06 Aug, 2003 1 commit
    • Leigh B. Stoller's avatar
      Clean up temporary files used in modify. The temp dirs were being · 05bd80ff
      Leigh B. Stoller authored
      created in /tmp and left behind. I've moved them to the expwork
      directory instead, and added a routine in the library to clear them
      out.
      
      Clear out the nsfile (stored in /tmp) used in modify. The web page was
      creating a temp file, but never removing it. swapexp now copies the
      nsfile in so that the web page can remove the temporary after the
      script exits. The temp is placed in the expwork directory as well, but
      left behind for debugging.
      
      When swapmod fails, send along the nsfile in the email message.
      05bd80ff
  31. 05 Aug, 2003 1 commit
  32. 30 Jul, 2003 1 commit
  33. 29 Jul, 2003 1 commit
  34. 25 Jul, 2003 1 commit
    • Leigh B. Stoller's avatar
      Commit my version of assign_wrapper as assign_wrapper-new, and change · 62e38deb
      Leigh B. Stoller authored
      tbswap to use this version inside the testbed project only! All other
      projects will see the old version for now; there are just too many
      things to test, and the testsuite gets just a fraction of them. Some
      highlights (which I will expand on later when I commit this version to
      the main version):
      
      * New -t option to create the TOP file, and then exit. The only other
        side effect of this is to update the min/max nodes for the
        experiment in the DB, unles new option -n (impotent mode) is given.
      
      * New -n option to operate in impotent mode; do not allocate nodes and
        do not modify the DB. Okay, so this option is not as great as it
        sounds. I eventually hit the point of diminishing returns, with
        trying to make things work right without DB modification. At some
        point I just throw in the towel and exit. This currently happens after
        interpolating the link results of assign. But, I have found it very
        useful, and could get better with time. Being able to run assign on
        the main DB without sucking up the nodes is nice for debugging.
      
      * Lots of data structure organization, mostly on the virtual topology
        side of assign (you can think of assign as two sections, the part
        that interprets the DB tables and creates the TOP file, and the part
        that reads the results of assign and sets up all the physical stuff
        in the DB). I removed numerous global hashes, and combined them into
        aggregate data structures, such as they are in Perl. My approach for
        this was to read the tables from the DB, and keep them handy,
        extending them as needed with stuff that assign_wrapper generates as
        it proceeds. This has the side effect of cutting down on the number
        of queries as well.
      
        The next task is to do the physical side reorg, but not up for that
        yet.
      62e38deb
  35. 17 Jul, 2003 1 commit
  36. 15 Jul, 2003 1 commit
    • Leigh B. Stoller's avatar
      A set of changes to make swapmod work on jailed nodes (note, swapmod · 92ff875a
      Leigh B. Stoller authored
      does not yet work with remove virtual nodes; that will take even more
      work).
      
      Added a new allocstate called RES_TEARDOWN. assign_wrapper no longer
      deallocates unused nodes, but rather moves them into the new state for
      the wrapper (tbswap) to deal with. Thats cause deleted vnodes need to
      be torn down, since its possible that the node on which they were
      living will not be deallocated (say, if there are other vnodes on
      it). We do not want to be doing that from assign_wrapper, so tbswap
      looks for those nodes.
      
      Made vnode_setup allocstate aware in the same way that os_setup is;
      do not reboot vnodes or try to set up vnodes when they are already in
      the RES_READY state, as they will be when doing a swapmod. In
      addition, if os_setup is going to reboot the underlying physnode, move
      the vnodes on that node into RES_READY too, since there they will
      setup automatically. Might need an interim state here, for correctness.
      92ff875a