1. 27 Jul, 2007 1 commit
    • Leigh B. Stoller's avatar
      * Add new tables to store NS files (and any files they source) in the · f8333ef2
      Leigh B. Stoller authored
        DB alongside the resource records. Previously, we stored only the
        nsfiles for current experiments, and purged them when the experiment
        was terminated. The new approach saves them forever using the resource
        record ID. Note that we do not store copies of NS files, but reference
        them indirectly instead so that we can MD5 them and avoid the dups.
      
        I put a "compressed" bit into the table cause at some point we will
        start compressing the data before storing them into the DB. Or maybe
        we bag this and start using GFS!
      
        Also note that this addresses the problem of losing the NS file
        history when using swapmod, since the NS file is overwritten.
      
      * Add a pmapping table stores the nodes (and their types) used by an
        experiment. This data is also saved forever, alongside the resource
        records, so that we can more accurately replay an experiment. As Rob
        points out, the node names can also be used in conjunction with the
        ptop files that are saved, to get a 100% accurate remap of resources.
      f8333ef2
  2. 25 Jul, 2007 1 commit
  3. 19 Jul, 2007 1 commit
  4. 18 Jul, 2007 1 commit
  5. 17 Jul, 2007 2 commits
    • Leigh B. Stoller's avatar
      Add some sanity checking to make sure that the user is tagging the · 4c278840
      Leigh B. Stoller authored
      highest revision on the branch (or trunk) of the magic .template file,
      since doing otherwise would seem to indicate an inconsistency in
      what the user is doing. Why? Templates are modified via a tag
      operation of the .template file, but splitting off can only be done
      via a branch operation. Subsequent mods after a branch have to happen
      along a straight line.
      
      Also allow for tag moving to initiate a template modify since the
      above implied error can be corrected by doing a cvs update, and then a
      retag with the -F option to move the tag.
      4c278840
    • David Johnson's avatar
      Support for accessing the v4 NM via a nm-controller slice account. Now, · 88e61ae3
      David Johnson authored
      for each delegated slice we create, we set the 'delegations' attribute to
      'utah_nmcontrol', the name of our nm-controller slice that has permissions
      to talk to the NM.
      
      Also, a few other fixes.
      88e61ae3
  6. 16 Jul, 2007 4 commits
    • Leigh B. Stoller's avatar
      Generate a info.xml file in the records/run directory that describes · 055b8769
      Leigh B. Stoller authored
      the run. Includes the bindings for the run and the metadata name/value
      pairs. This is the same data that is dumped by template export, so
      at this point template_export has been completely superceded by the
      CVS repo.
      055b8769
    • David Johnson's avatar
      Several things in this commit: · 3bbd843b
      David Johnson authored
        * Prior to this commit, libplab depended on db state to create slivers
          and slices.  Now it can be done using the regular command line tools
          without the metadata in the db.  This makes development and debugging
          much easier and allows us to use the command-line tools even if state has
          been cleared out of the db (i.e., for sliver garbage collection).
        * Add support for sliver start/stop/restart via the v4 NM.
        * Some support for sliver garbage collection.
        * Various other improvements and cleanup.
      3bbd843b
    • Leigh B. Stoller's avatar
      Remove debugging code. · 4a50330d
      Leigh B. Stoller authored
      4a50330d
    • Leigh B. Stoller's avatar
      The CVS repo will now have two subdirs, one called "setup" which is · 1aaa383c
      Leigh B. Stoller authored
      where the current stuff goes (datastore, nsfile, etc.). The other
      subdir is called "records" and will be populated with stuff that we
      currently put into the archive (the contents of the experiment
      directory).
      
      There is a record for each run, currently named XXX,YY where XXX is
      the instance ID and YY is the run number. I plan to add an XML file to
      map these to the symbolic names, but these are not required to be
      unique, which is why the directories are not named with them.
      1aaa383c
  7. 14 Jul, 2007 2 commits
  8. 13 Jul, 2007 2 commits
    • Russ Fish's avatar
      Minor fixups. · f3d49948
      Russ Fish authored
      f3d49948
    • Russ Fish's avatar
      Check that the inner elab db schema isn't out of date wrt the outer one, · b54ae4da
      Russ Fish authored
        since we import a bunch of dumped db state from the outer to the inner db.
        Added/removed columns in the schema misalign row data, causing much chaos.
      
      tbsetup/elabinelab.in - Dump and send the outer db schema with the db state.
        Runs mysqldump just like schemacheck does.
      
      tmcd/common/config/rc.mkelab - Add the schema check using schemadiff.
      
      utils/schemadiff - Add an option to ignore table creates and drops.
      b54ae4da
  9. 10 Jul, 2007 1 commit
    • Leigh B. Stoller's avatar
      Changes to switch from using tag to rtag. Note that this commit requires · 7a2ef6f1
      Leigh B. Stoller authored
      a locally modifed version of CVS that does a couple of extra things. The
      hacks are quite simple and currently on users. Briefly, there is a new
      config file option to specify a program to run when the tag operation is
      complete (similar to a module hook for rtag), and in tag.c I run that hook.
      
      Check out a sandbox as before:
      
        cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo rtag mytag XXXXX
      
      To create a new template (along the trunk), use the tag operation instead:
      
      	cd sandbox
      	cvs tag mtag
      
      and what for the email that says the template is created. You can
      continue to work along the trunk in this manner, although note that
      the .template file is changing (by the backend commit), and you might
      find it less distriacting to do a cvs update each time.
      
      To work along a branch:
      
      	cd sandbox
      	cvs tag -r existing_tag -b newtag1
              cvs update -r newtag
              make changes and commits
      	cvs tag newtag2
      
      When working along branched you will want to do a cvs update to get
      the .template file in sync.
      7a2ef6f1
  10. 09 Jul, 2007 1 commit
    • Leigh B. Stoller's avatar
      Checkpoint my cvs interface to the workbench. This first cut uses the · 8371fc79
      Leigh B. Stoller authored
      "rtag" directive to initiate template modify operations. So, to get started
      you do a checkout:
      
        cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo checkout XXXXX
      
      where XXXXX is the part of the guid (10000/1) before the slash. Might try
      and roll all templates into a single project wide repo at some point, to
      avoid the extraneous path stuff, but didn't want to worry that just yet.
      
      Okay, so have a checkout. You can work along the trunk, doing commits. To
      create a new template (a modify of the existing template), you tag the tree
      using rtag:
      
        cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo rtag mytag XXXXX
      
      A template modify is started at the end, and you should probably wait for
      email before continuing. Eventually I will need to add locking of some
      kind, but I have to do the modify in the background, or else I get deadlock
      cause cvs keeps the repo locked, and the modify also needs to access it.
      
      Each time you tag along the trunk, you get a modified template, which in
      the history diagram looks like:
      
        10000/1 --> 10000/2 --> 10000/3 ...
      
      If you want to branch, say at 10000/2 you can create a branch tag using rtag:
      
        cvs -d [cut] rtag -r T10000/2 -b mytag2 XXXXX
      
      You can also use your own tags for -r option, but I also create a TXXXXX/YY
      tag at each template modify, which is easy to remember.
      
      Then update your sandbox to the new branch, commit changes along that
      branch, and then later use rtag again to initiate a template modify
      operation:
      
        cvs update -r mytag2
        cvs commit ...
        cvs -d [cut] rtag -r mytag2 mytag3 XXXXX
      
      And now the history diagram looks like:
      
        10000/1 --> 10000/2 --> 10000/3 ...
                      |
                      |
                      -> 10000/4 ...
      
      You should be able to mix interaction via the web with interaction via the
      cvs interface. I've tested it, although not extensively.
      8371fc79
  11. 20 Jun, 2007 1 commit
    • Leigh B. Stoller's avatar
      Add summary node utilization stats. The initial values are derived by · 21890006
      Leigh B. Stoller authored
      processing the node_history table, but that is *way* too slow to do on
      the fly (say, from the web interface) cause of the number of records,
      so the summary info is stored in the new node_utilization table. I
      generate the summary info as new entries are added to node_history (in
      SetNodeHistory) but if that becomes too messy, we can just as easily
      shift to processing the table on a nightly basis.
      
      Note that I added a new "inception" date field to the nodes table,
      which will get set on new nodes, but for existing nodes I have to
      derive it from the first entry in the node_history table, or else the
      numbers will not make sense.
      21890006
  12. 19 Jun, 2007 1 commit
    • Leigh B. Stoller's avatar
      Big update to the stats gathering code ... · 495f6803
      Leigh B. Stoller authored
      This change attempts to make the stats gathering code more reliable by
      not relying on the testbed_stats records to reconstruct usage
      statistics.  The main source of errors and total confusion in the
      current stats code is that testbed_stats includes all the errors and
      transitions, from which I have to reconstruct what happened in order
      to determine usage by a project or user.
      
      The new stats code still generates the testbed_stats code, but actual
      usage is recorded as it happens, in the experiment_resources table, as
      swapins, swapouts, and swapmods occur. Its also much faster to compute
      the data for the tables in the web interface, not having to scan a
      zillion testbed_stats records in php.
      
      There is a time consuming update to the records that takes place with
      a lot of tables locked.
      495f6803
  13. 15 Jun, 2007 1 commit
  14. 06 Jun, 2007 1 commit
  15. 30 May, 2007 1 commit
  16. 28 May, 2007 1 commit
    • Leigh B. Stoller's avatar
      Implement two requests by Eric: · e12ea1e2
      Leigh B. Stoller authored
      * The checkout directory now includes just the datastore directory and
        tbdata/nsfile.ns. This removes a bunch of clutter.
      
      * You can now run template_commit in any subdir of a checkout; the
        python wrapper will crawl upwards looking for the .template file.
      
        Note that I removed the option that allowed you to do a commit from
        the template tree in /proj/pid/templates since that directory will
        ultimately go away (or at least hide from view), and cause it
        conflicted with the new option and I didn't want to make things any
        messier for no reason.
      e12ea1e2
  17. 24 May, 2007 2 commits
  18. 23 May, 2007 1 commit
    • Leigh B. Stoller's avatar
      First cut at template checkout and commit from a checkout. The interface · b674bc7b
      Leigh B. Stoller authored
      described is the one exported to ops via the XMLRPC interface. This is
      just playing aroundl no doubt this stuff is going to change.
      
      * template_checkout guid/vers
      
        Checkout a copy of the template to the current working directory.
      
      * template_commit
      
        Modify the previous template checkout, using the nsfile contained in
        the tbdata directory (subdir of the current directory). In other words,
        the current template is modified, creating a new template in the
        current working directory (the current directory refers to the new
        template).
      
        The datastore subdir is imported into the new template, but that is
        the only directory that is imported at present. Might change that.
      
      So this sounds much cooler then it really is. Why?
      
      * This only works from ops.
      
      * The "current directory" must be one of the standard approved directories
        (/proj, /users, /groups).
      
      * Cause, boss reads and writes that directory via NFS, as told to it
        by the xmlrpc client.
      
      At some point in the future it would be nice to support something
      fancier, using a custom transport, but lets see how this goes.
      b674bc7b
  19. 18 May, 2007 1 commit
  20. 17 May, 2007 2 commits
  21. 16 May, 2007 1 commit
  22. 15 May, 2007 4 commits
    • David Johnson's avatar
      1fadb8ca
    • Leigh B. Stoller's avatar
      Symlink trick to deal with the fact that we switched to archiving the · d4bcc494
      Leigh B. Stoller authored
      entire experiment directory, which now that we zip files, was putting
      thinks in the wrong place.
      d4bcc494
    • David Johnson's avatar
      Add comon data to its own table so that we can use it in joins. Add · 13c6cc7e
      David Johnson authored
      nodehist metrics to their own table for the same reason.
      13c6cc7e
    • Leigh B. Stoller's avatar
      Checkpoint changes that have been discussed in the last few weeks: · c4f53202
      Leigh B. Stoller authored
      * Records are now "help open" when a run is stopped. When the next run
        is started, a check is made to see if the files
        (/project/$pid/exp/$eid) have changed, and if so a new version of the
        archive is committed before the next run is started.
      
      * Change the way swapmod is handled within an instance. A new option
        on the ShowExp page called Modify Resources. The intent is to allow
        an instance to be modified without having to start and stop runs,
        which tends to clutter things up, according to our user base. So, if
        you are within a run, that run is reset (reused) after the swapmod is
        finished. You can do this as many times as you like. If you are
        between runs (last operation was a stoprun), do the swapmod and then
        "speculatively" start a new run. Subsequent modifies reuse the that
        run again, as above.
      
        I think this is what Kevin was after ... there are some UI issues
        that may need to be resolved, will wait to hear what people have to
        say.
      
      * Revising a record is now supported. Export, change in place, and
        then use the Revise link on the ShowRun page. Currently this has to
        happen from the export directory on ops, but eventually allow an
        upload (to correspond to downloaded exports)
      
      * Check to see if export already exists, and give warning. Added a
        checkbox that allows user to overwrite the export.
      
      * A bunch of minor UI changes to the various template pages.
      c4f53202
  23. 11 May, 2007 2 commits
  24. 09 May, 2007 1 commit
  25. 08 May, 2007 1 commit
  26. 07 May, 2007 3 commits