1. 26 Aug, 2014 1 commit
    • Kirk Webb's avatar
      Re-arrange batch system front-end to match Web UI behavior · 170b1ddb
      Kirk Webb authored
      * autoswap time and idleswap time defaults are inherited from the
        associated sitevars.
      
       - "general/autoswap_threshold", and "idle/threshold", respectively
      
      * autoswap default state follows associated sitevar ("general/autoswap_mode").
      
      * Any user-requested autoswap duration is checked against the
        "general/autoswap_ max" sitevar.
      
       - Admins can override.
      
      * Any user-requested idleswap duration is checked against the "idle/threshold"
        sitevar.
      
       - Admins can override.
      
      * Disabling swapping (-S option) is prohibited, except by admins.
      
      * Removed all special-case checks for requests coming via the XMLRPC server.
      170b1ddb
  2. 12 Jun, 2014 1 commit
    • Kirk Webb's avatar
      Update the use of realpath across all perl scripts · 3f167217
      Kirk Webb authored
      Change to use the realpath function in the 'Cwd' module instead of
      calling realpath via the shell.  The shell command varies in its
      reaction to a missing final path component.  On some platforms (Linux,
      FBSD10+) realpath reports an error if the final component doesn't exist
      on the filesystem.  On others (FBSD < 10), it does not report an error.
      
      The perl function from 'Cwd' emulates the same behavior as FBSD prior to
      version 10, which is the behavior the scripts expect.
      
      From here on out, instead of using `realpath`, do the following:
      
      use Cwd qw(realpath);
      ..
      ..
      my $realpath = realpath($somepath);
      3f167217
  3. 29 May, 2014 1 commit
    • Mike Hibler's avatar
      Allow /tmp/php* as well as /var/tmp/php*. · e10872da
      Mike Hibler authored
      This shows up on FBSD 10.x based elabs that use one big FS for
      /, /var, /usr since realpath(/var/tmp/...) resolves to /tmp/...
      Originally noticed by Jeff Bowles at NMC, but I didn't make the fix
      til it bit me on the ass in APT!
      e10872da
  4. 19 Sep, 2013 1 commit
  5. 24 Jul, 2013 1 commit
  6. 22 Mar, 2013 1 commit
  7. 08 Mar, 2013 1 commit
  8. 30 Jan, 2013 1 commit
  9. 29 Jan, 2013 1 commit
  10. 27 Nov, 2012 1 commit
    • Leigh Stoller's avatar
      Start saving the virtual and physical state after each swapin/swapmod · 229d2bcc
      Leigh Stoller authored
      so that we can do post-mortem debugging. I'm adding this so I can
      track down the current snmpit failures, but might useful later too.
      
      Sadly, mysql dump files have no table metadata so once the schema
      changes, these files will become less useful if we ever want to load
      them into a running DB. It would be nie to dump them in a format that
      allows this, but will be a bunch more work.
      229d2bcc
  11. 24 Sep, 2012 1 commit
    • Eric Eide's avatar
      Replace license symbols with {{{ }}}-enclosed license blocks. · 6df609a9
      Eric Eide authored
      This commit is intended to makes the license status of Emulab and
      ProtoGENI source files more clear.  It replaces license symbols like
      "EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
      blocks that contain actual license statements.
      
      This change was driven by the fact that today, most people acquire and
      track Emulab and ProtoGENI sources via git.
      
      Before the Emulab source code was kept in git, the Flux Research Group
      at the University of Utah would roll distributions by making tar
      files.  As part of that process, the Flux Group would replace the
      license symbols in the source files with actual license statements.
      
      When the Flux Group moved to git, people outside of the group started
      to see the source files with the "unexpanded" symbols.  This meant
      that people acquired source files without actual license statements in
      them.  All the relevant files had Utah *copyright* statements in them,
      but without the expanded *license* statements, the licensing status of
      the source files was unclear.
      
      This commit is intended to clear up that confusion.
      
      Most Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the Affero GNU General Public License, version 3
      (AGPLv3).
      
      Most Utah-copyrighted files related to ProtoGENI are distributed under
      the terms of the GENI Public License, which is a BSD-like open-source
      license.
      
      Some Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the GNU Lesser General Public License, version 2.1
      (LGPL).
      6df609a9
  12. 30 Aug, 2011 1 commit
  13. 22 Aug, 2011 1 commit
  14. 07 Nov, 2009 1 commit
    • Leigh Stoller's avatar
      Change to infodir (/usr/testbed/expinfo) handling; experiment · 41d34103
      Leigh Stoller authored
      directories are now placed in a project subdirectory, to avoid
      blowing out the max number of subdirs (32K in FreeBSD). Dirs are
      now called $pid/$eid/$idx.
      
      Added sanity checks to batchexp, swapexp, and endexp to watch for the
      case that testbed admin installed the new code but did not run the
      fixup script as instructed in doc/UPDATING.
      41d34103
  15. 03 Jun, 2009 1 commit
    • Kevin Atkinson's avatar
      Add support for returning more detailed information in the case of a · 340f10a4
      Kevin Atkinson authored
      swap failure in the xmlrpc server.  To use it, the extra option
      "extrainfo" needs to be set to true.  This will cause "value" to be a
      structure (in the XML-RPC sence) with useful information instead of
      just the exit value of the script.  The structure will contain at
      least the following fields:
        cause: "temp", "user", etc
        cause_desc
        mesg: more specific error information
        exitval: script return value
        log: activity log
      Note, that value may still be an integer in the case of some other
      failure that is not swap related.
      
      To support this, the "-X" option was added to swapexp and batchexp
      which will output a RPC-XML method response to stdout when possible.
      340f10a4
  16. 28 May, 2009 1 commit
    • Kevin Atkinson's avatar
      Add "-N" option to batchexp/swapexp/endexp to suppress most email to · 649cd68c
      Kevin Atkinson authored
      testbed-ops and the user.  Important email that requires testbed-ops
      attention, such as on a recursive cleanup error, will still be sent.
      In addition mail normally sent to testbed-logs will still be sent.
      
      Also, add "noemail" option to xmlrpc server methods corresponding to
      those commands, and "-N" option to related commands in script_wrapper.
      649cd68c
  17. 06 Jul, 2008 1 commit
  18. 08 May, 2008 1 commit
  19. 28 Nov, 2007 1 commit
  20. 26 Nov, 2007 1 commit
  21. 02 Aug, 2007 1 commit
  22. 27 Jul, 2007 1 commit
    • Leigh Stoller's avatar
      * Add new tables to store NS files (and any files they source) in the · f8333ef2
      Leigh Stoller authored
        DB alongside the resource records. Previously, we stored only the
        nsfiles for current experiments, and purged them when the experiment
        was terminated. The new approach saves them forever using the resource
        record ID. Note that we do not store copies of NS files, but reference
        them indirectly instead so that we can MD5 them and avoid the dups.
      
        I put a "compressed" bit into the table cause at some point we will
        start compressing the data before storing them into the DB. Or maybe
        we bag this and start using GFS!
      
        Also note that this addresses the problem of losing the NS file
        history when using swapmod, since the NS file is overwritten.
      
      * Add a pmapping table stores the nodes (and their types) used by an
        experiment. This data is also saved forever, alongside the resource
        records, so that we can more accurately replay an experiment. As Rob
        points out, the node names can also be used in conjunction with the
        ptop files that are saved, to get a 100% accurate remap of resources.
      f8333ef2
  23. 25 Jul, 2007 1 commit
  24. 09 Jul, 2007 1 commit
    • Leigh Stoller's avatar
      Checkpoint my cvs interface to the workbench. This first cut uses the · 8371fc79
      Leigh Stoller authored
      "rtag" directive to initiate template modify operations. So, to get started
      you do a checkout:
      
        cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo checkout XXXXX
      
      where XXXXX is the part of the guid (10000/1) before the slash. Might try
      and roll all templates into a single project wide repo at some point, to
      avoid the extraneous path stuff, but didn't want to worry that just yet.
      
      Okay, so have a checkout. You can work along the trunk, doing commits. To
      create a new template (a modify of the existing template), you tag the tree
      using rtag:
      
        cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo rtag mytag XXXXX
      
      A template modify is started at the end, and you should probably wait for
      email before continuing. Eventually I will need to add locking of some
      kind, but I have to do the modify in the background, or else I get deadlock
      cause cvs keeps the repo locked, and the modify also needs to access it.
      
      Each time you tag along the trunk, you get a modified template, which in
      the history diagram looks like:
      
        10000/1 --> 10000/2 --> 10000/3 ...
      
      If you want to branch, say at 10000/2 you can create a branch tag using rtag:
      
        cvs -d [cut] rtag -r T10000/2 -b mytag2 XXXXX
      
      You can also use your own tags for -r option, but I also create a TXXXXX/YY
      tag at each template modify, which is easy to remember.
      
      Then update your sandbox to the new branch, commit changes along that
      branch, and then later use rtag again to initiate a template modify
      operation:
      
        cvs update -r mytag2
        cvs commit ...
        cvs -d [cut] rtag -r mytag2 mytag3 XXXXX
      
      And now the history diagram looks like:
      
        10000/1 --> 10000/2 --> 10000/3 ...
                      |
                      |
                      -> 10000/4 ...
      
      You should be able to mix interaction via the web with interaction via the
      cvs interface. I've tested it, although not extensively.
      8371fc79
  25. 19 Jun, 2007 1 commit
    • Leigh Stoller's avatar
      Big update to the stats gathering code ... · 495f6803
      Leigh Stoller authored
      This change attempts to make the stats gathering code more reliable by
      not relying on the testbed_stats records to reconstruct usage
      statistics.  The main source of errors and total confusion in the
      current stats code is that testbed_stats includes all the errors and
      transitions, from which I have to reconstruct what happened in order
      to determine usage by a project or user.
      
      The new stats code still generates the testbed_stats code, but actual
      usage is recorded as it happens, in the experiment_resources table, as
      swapins, swapouts, and swapmods occur. Its also much faster to compute
      the data for the tables in the web interface, not having to scan a
      zillion testbed_stats records in php.
      
      There is a time consuming update to the records that takes place with
      a lot of tables locked.
      495f6803
  26. 28 May, 2007 1 commit
    • Leigh Stoller's avatar
      Implement two requests by Eric: · e12ea1e2
      Leigh Stoller authored
      * The checkout directory now includes just the datastore directory and
        tbdata/nsfile.ns. This removes a bunch of clutter.
      
      * You can now run template_commit in any subdir of a checkout; the
        python wrapper will crawl upwards looking for the .template file.
      
        Note that I removed the option that allowed you to do a commit from
        the template tree in /proj/pid/templates since that directory will
        ultimately go away (or at least hide from view), and cause it
        conflicted with the new option and I didn't want to make things any
        messier for no reason.
      e12ea1e2
  27. 07 May, 2007 1 commit
    • Leigh Stoller's avatar
      Mostly this commit is the switch from SVN archives to ZIP archives. · 55d1bb6e
      Leigh Stoller authored
      Other stuff leaked in too ...
      
      I did separate out a lot of tbsetup/libArchive into db/Archive, and
      whats left in libArchive.pm will eventually move over into the
      Template library.
      
      Note that I have dropped archiving of plain experiments; this is not
      really owrth it outside the workbench context, and it just wastes
      space and makes a lot if stuff painful in the web interface.
      55d1bb6e
  28. 13 Mar, 2007 1 commit
  29. 07 Mar, 2007 1 commit
  30. 06 Mar, 2007 1 commit
  31. 09 Jan, 2007 1 commit
  32. 08 Dec, 2006 1 commit
    • Leigh Stoller's avatar
      As discussed in meetings and email ... this commit changes what is · b898a8cc
      Leigh Stoller authored
      archived.  Rather then a special "archive" directory in the experiment
      directory, we know archive the entire experiment directory.
      
      This change should be backwards compatable, but let me know if not.
      
      Note that the nsdata directory is gone; the nsfile comes from the
      tbdata, but I know place a copy in nsfile.ns so that the name is well
      known.
      b898a8cc
  33. 06 Nov, 2006 1 commit
    • Kevin Atkinson's avatar
      libaudit related changes: · e89ee617
      Kevin Atkinson authored
        - Added "LIBAUDIT_FANCY" option to AuditStart.  When this option is
          used libaudit will send a different email than it normally sends,
          and on error call tblog_find_error() to determine the error.
      
        - Also add audit function AddAuditInfo which adds add additional
          information for libaudit to use in SendAuditMail when AUDIT_FANCY
          is set.
      
        - Modify template_swapin, template_instantiate, and template_create
          to use the new audit functionality.
      
        - Suppressing calling tblog_find_error and sending the error email
          when auditing in swapexp and batchexp
      
      tblog changes:
      
        - Shorten the message sent to the user when the error in unknown.
          Remove all parts about lack of free nodes as it no longer really
          applies as tblog now correctly identified those errors and handles
          them separately.  The message is now just "Please look at the log
          below to see what happened."
      
        - Improve algo. used to determine the other error when canceled.
          Will now work by removing all errors related to the cancel request
          and the essentially rerunning tblog_find_error.  If the cause of
          the error is still canceled, repeat and try again until the cause
          is something other than canceled or no errors are left.
      
        - Refactor tblog_find_error, which involves creating new internal
          functions: tblog_determine_single_error, tblog_store_error,
          tblog_dump_error
      
        - Add section on Primary vs Secondary Errors to the inline POD
          documentation.
      
        - Other minor enhancements and bug fixes.
      e89ee617
  34. 20 Oct, 2006 1 commit
    • Mike Hibler's avatar
      Wow, this should make me look important! · afa5e919
      Mike Hibler authored
      Two-day boondoggle to support "/scratch", an optional large, shared filesystem
      for users.  To do this, I needed to find all the instances where /proj is used
      and behave accordingly.  The boondoggle part was the decision to gather up all
      the hardwired instances of shared directory names ("/proj", "/users", etc.)
      so that they are set in a common place (via unexposed configure variables).
      This is a boondoggle because:
      
      1. I didn't change the client-side scripts.  They need a different mechanism
         (e.g., tmcd) to get the info, configure is the wrong way.
      
      2. Even if I had done #1 it is likely--no, certain--that something would
         fail if you tried to rename "/proj" to be "/mike".  These names are just
         too ingrained.
      
      3. We may not even use "/scratch" as it turns out.
      
      Note, I also didn't fix any of the .html documentation.  Anyway, it is done.
      To maintain my illusion in the future you should:
      
      1. Have perl scripts include "use libtestbed" and use the defined PROJROOT(),
         et.al. functions where possible.  If not possible, make sure they run
         through configure and use @PROJROOT_DIR@, etc.
      
      2. Use the configure method for python, C, php and other languages.
      
      3. There are perl (TBValidUserDir) and php (VALIDUSERPATH) functions which
         you should call to determine if an NS, template parameter, tarball or
         other file are in "an acceptable location."  Use these functions where
         possible.  They know about the optional "scratch" filesystem.  Note that
         the perl function is over-engineered to handles cases that don't occur
         in nature.
      afa5e919
  35. 18 Oct, 2006 1 commit
  36. 04 Oct, 2006 1 commit
  37. 16 Aug, 2006 1 commit
    • Kevin Atkinson's avatar
      - Added tbreport database schema (added three tables), storage for · 9c5d3308
      Kevin Atkinson authored
        tbreport errors & context.
      
      - Modified fatal() in swapexp, batchexp, and tbprerun, and die_noretry()
        in os_setup to pass hash parameter to tblog functions.
      
      - Added tbreport errror & context information for select errors in
        swapexp, tbswap, assign_wrapper2, snmpit_lib, snmpit, batchexp,
        assign_wrapper, os_setup, parse-ns, & tbprerun.
      
      - Added assign error parser in assign_wrapper2.
      
      - Added parse.tcl error parser in parse-ns.
      
      - Added severity constants for tbreport in libtblog_simple.
      
      - Added tbreport() function & context table mappging for reporting
        discrete error types to libtblog.
      9c5d3308
  38. 14 Aug, 2006 1 commit
  39. 03 Aug, 2006 1 commit
    • Leigh Stoller's avatar
      Support for capturing the trace data that is stored in the pcal files · 4ce9c421
      Leigh Stoller authored
      into per-experiment databases on ops. Additional support for reconsituting
      those databases back into temporary databases on ops, for post processing.
      
      * This revision relies on the "snort" port (/usr/ports/security/snort)
        to read the pcap files and load them into a database. The schema is
        probably not ideal, but its better then nothing. See the file
        ops:/usr/local/share/examples/snort/create_mysql for the schema.
      
      * For simplicity, I have hooked into loghole, which already had all
        the code for downloading the trace data. I added some new methods to
        the XMLRPC server for loghole to use, to get the users DB password
        and the name of the per-experiment database. There is a new slot in
        the traces table that indicates that the trace should be snorted to
        its DB. In case you forgot, at the end of a run or when the instance
        is swapped out, loghole is run to download the trace data.
      
      * For reconsituting, there are lots of additions to opsdb_control and
        opsdb_control.proxy to create "temporary" databases and load them
        from a dump file that is stored in the archive. I've added a button
        to the Template Record page, inappropriately called "Analyze" since
        right now all it does is reconsitute the trace data into a DB on
        ops.
      
        Currently, the only indication of what has been done (the name of
        the DBs created on ops) is the log email that the user gets. A
        future project is tell the user this info in the web interface.
      
      * To turn on database capturing of trace data, do this in your NS
        file:
      
      	set link0 ...
      	$link0 trace
      	$link0 trace_snaplen 128
      	$link0 trace_db 1
      
         the increase in snaplen is optional, but a good idea if you want
         snort to undertand more then just ip headers.
      
      * Also some changes to the parser to allow plain experiments to take
        advantage of all this stuff. To simple get yourself a per-experiment
        DB, put this in your NS file:
      
      	tb-set-dpdb 1
      
        however, anytime you turn trace_db on for a link or lan, you
        automatically get a per-experiment DB.
      
      * To capture the trace data to the DB, you can run loghole by hand:
      
      	loghole sync -s
      
        the -s option turns on the "post-process" phase of loghole.
      4ce9c421
  40. 28 Jul, 2006 1 commit
    • Leigh Stoller's avatar
      Add a "Create Template from Instance" ability. Basically, you can · a651da71
      Leigh Stoller authored
      create a new template (well, really a modify) from the current
      swapped in experiment. This allows you to create a template, swap in
      an instance, modify the datastore in the instance (which is a copy
      of the datastore in the template), and then create a new template
      using the datastore and nsfile from the instance. This is a new menu
      item on the showexp page for the instance.
      
      Also in this commit are fixes and improvements to the new navagation
      bar that I recently added.
      a651da71