1. 10 Jan, 2008 1 commit
  2. 20 Dec, 2007 1 commit
  3. 26 Nov, 2007 2 commits
  4. 06 Nov, 2007 1 commit
    • Leigh B. Stoller's avatar
      This started out as a simple change to turn the datastore into a CVS · c1cff09b
      Leigh B. Stoller authored
      sandbox, and that I did. It falls back to the older archive when
      the template is older then CVS repos.
      
      But along the way I got annoyed with the fact that template instantiation
      does not provide a logfile to the web interface. The reason is that
      the current logfile stuff is very experiment centric; there has to be an
      experiment and an attached logfile. An instance does not have an experiment
      until really late in the game so the code was just not bothering.
      
      Anyway, I've started to generalize the logfile stuff with a new table
      and the approach that a logfile is named by a random key, and if you
      know the key you can look at the logfile in the web (since without an
      experiment it is hard to do permission checks unless we make logfiles
      uid/gid owned, and I did not want to do that.
      c1cff09b
  5. 17 Oct, 2007 1 commit
  6. 21 Aug, 2007 1 commit
    • Leigh B. Stoller's avatar
      Another round of widearea node hacking for CMU. These changes add · 99346dc0
      Leigh B. Stoller authored
      widearea reloading support.
      
      * New slot in the images table to store an access key which remote
        sites must provide in order to download an image (via https).
      
      * tmcd returns a different kind of ADDRESS field from doloadinfo.
        Instead of the multicast stuff, return a URL that points to boss'
        web server. The URL is of the form:
      
         https://www.myemulab.net/spewimage.php?imageid=10013&access_key=abcdef
      
        which as you can see is fully specified; the client does not need
        to know anything else.
      
      * New webpage and backend scripts appropriately called "spewimage"
        which also includes support for the http HEAD request (from wget) to
        avoid downloading images that are already on the node. I just
        learned about this HEAD request stuff today ... but otherwise these
        operate as expected, spewing the image if the access key is provided.
      
      * Changes to rc.frisbee to deal with remote loading. In addition to
        URL support, I also added support for simple paths, the intent being
        that we will probably distribute images offline (say, at night) so
        that when a node reboots it doesn't actually have to wait 60 minutes
        for an image to download. I have not added any server side support
        for this yet though. Maybe later this week.
      
      * Other bits and pieces and fixes to make this work.
      99346dc0
  7. 16 Aug, 2007 1 commit
    • Leigh B. Stoller's avatar
      Add IP and privkey slots (key INDEX slots) to widearea_nodeinfo table. · 0fe9fe1a
      Leigh B. Stoller authored
      This will superceed the widearea_privkeys table that was used for
      RON/Netbed but was never done well.
      
      The privkey is just something we generate and store on the dongle (as
      a simplification, we just create a dongle per node instead of
      generating them on the fly) since there are not very many of these
      widearea (CMU) nodes. We store the IP here for convenience (there is
      an interfaces table entry of course).
      
      The IP is updated when the node tells us it has changed, using the
      privkey to uniquely identify it (for checkin).
      0fe9fe1a
  8. 31 Jul, 2007 1 commit
  9. 27 Jul, 2007 2 commits
    • Leigh B. Stoller's avatar
      Minor fix to previous revision. · 438ccf09
      Leigh B. Stoller authored
      438ccf09
    • Leigh B. Stoller's avatar
      * Add new tables to store NS files (and any files they source) in the · f8333ef2
      Leigh B. Stoller authored
        DB alongside the resource records. Previously, we stored only the
        nsfiles for current experiments, and purged them when the experiment
        was terminated. The new approach saves them forever using the resource
        record ID. Note that we do not store copies of NS files, but reference
        them indirectly instead so that we can MD5 them and avoid the dups.
      
        I put a "compressed" bit into the table cause at some point we will
        start compressing the data before storing them into the DB. Or maybe
        we bag this and start using GFS!
      
        Also note that this addresses the problem of losing the NS file
        history when using swapmod, since the NS file is overwritten.
      
      * Add a pmapping table stores the nodes (and their types) used by an
        experiment. This data is also saved forever, alongside the resource
        records, so that we can more accurately replay an experiment. As Rob
        points out, the node names can also be used in conjunction with the
        ptop files that are saved, to get a 100% accurate remap of resources.
      f8333ef2
  10. 26 Jul, 2007 1 commit
  11. 12 Jul, 2007 1 commit
  12. 10 Jul, 2007 1 commit
  13. 20 Jun, 2007 1 commit
    • Leigh B. Stoller's avatar
      Add summary node utilization stats. The initial values are derived by · 21890006
      Leigh B. Stoller authored
      processing the node_history table, but that is *way* too slow to do on
      the fly (say, from the web interface) cause of the number of records,
      so the summary info is stored in the new node_utilization table. I
      generate the summary info as new entries are added to node_history (in
      SetNodeHistory) but if that becomes too messy, we can just as easily
      shift to processing the table on a nightly basis.
      
      Note that I added a new "inception" date field to the nodes table,
      which will get set on new nodes, but for existing nodes I have to
      derive it from the first entry in the node_history table, or else the
      numbers will not make sense.
      21890006
  14. 19 Jun, 2007 1 commit
    • Leigh B. Stoller's avatar
      Big update to the stats gathering code ... · 495f6803
      Leigh B. Stoller authored
      This change attempts to make the stats gathering code more reliable by
      not relying on the testbed_stats records to reconstruct usage
      statistics.  The main source of errors and total confusion in the
      current stats code is that testbed_stats includes all the errors and
      transitions, from which I have to reconstruct what happened in order
      to determine usage by a project or user.
      
      The new stats code still generates the testbed_stats code, but actual
      usage is recorded as it happens, in the experiment_resources table, as
      swapins, swapouts, and swapmods occur. Its also much faster to compute
      the data for the tables in the web interface, not having to scan a
      zillion testbed_stats records in php.
      
      There is a time consuming update to the records that takes place with
      a lot of tables locked.
      495f6803
  15. 31 May, 2007 1 commit
  16. 22 May, 2007 1 commit
  17. 07 May, 2007 1 commit
    • Leigh B. Stoller's avatar
      Mostly this commit is the switch from SVN archives to ZIP archives. · 55d1bb6e
      Leigh B. Stoller authored
      Other stuff leaked in too ...
      
      I did separate out a lot of tbsetup/libArchive into db/Archive, and
      whats left in libArchive.pm will eventually move over into the
      Template library.
      
      Note that I have dropped archiving of plain experiments; this is not
      really owrth it outside the workbench context, and it just wastes
      space and makes a lot if stuff painful in the web interface.
      55d1bb6e
  18. 01 May, 2007 1 commit
  19. 17 Apr, 2007 5 commits
  20. 13 Mar, 2007 2 commits
  21. 08 Mar, 2007 1 commit
  22. 06 Mar, 2007 2 commits
  23. 02 Mar, 2007 1 commit
  24. 21 Feb, 2007 1 commit
  25. 20 Feb, 2007 1 commit
  26. 16 Feb, 2007 1 commit
  27. 09 Jan, 2007 1 commit
  28. 20 Dec, 2006 1 commit
  29. 30 Nov, 2006 1 commit
  30. 27 Nov, 2006 1 commit
  31. 03 Nov, 2006 1 commit
    • Leigh B. Stoller's avatar
      Big set of changes intended to solve a couple of problems with long · ff9061d4
      Leigh B. Stoller authored
      term archiving of firstclass objects like users, projects, and of
      course templates.
      
      * Projects, Users, and Groups are now uniquely identified inside the
        DB by a index value that will not be reused. If necessary, this
        could easily be a globally unique identifier, but without federation
        there is no reason to do that yet.
      
      * Currently, pid, gid, and uid still need to be locally unique until
        all of the changes are in place (which is going to take a fairly
        long time since the entire system operates in terms of those, except
        for the few places that I had to change to get the ball rolling).
      
      * We currently archive deleted users to the deleted_users table (their
        user_stats are kept forever since they are indexed by the new index
        column). Eventually do the same with projects (not sure about
        groups) but since we rarely if ever delete a project, there is no
        rush on this one.
      
      * At the same time, I have started a large reorg of the code, to move
        all of the user, group, project code into modules, both in php and
        perl, turning them into first class "objects" (as far as that goes
        in php and perl). Eventually, the number of query statements
        scattered around the code will be manageable, or so I hope.
      
      * Another related part of this reorg is to make it easier to move the
        new user/project/group code in the perl backend so that it can be
        made available via the xmlrpc interface (without duplication of the
        code).
      ff9061d4
  32. 26 Oct, 2006 1 commit
    • Kevin Atkinson's avatar
      Various tblog changes: · 95a3a6a7
      Kevin Atkinson authored
        Make an attempt to discover what the error was before an swap-* was
        canceled, if any.  Both the main error (canceled) and the other error
        are stored in the error table.  To support this a new column in the
        error table is added "rank".  The primary error has a rank 0 while the
        other error has a rank 1.
      
        Make an attempt to determine when an error is a "me too" error or the
        real cause of the problem.  "Me too" errors are errors which are
        generally reported when the callie script determined that the caller
        script fails.  The caller script should have reported the error, but
        in some cases the error didn't make it into the database.  Thus if a
        "me too" is reported as the cause of a "swap-*" more info is needed to
        determine the true cause.  When a "me too" error is reported it is
        followed by a "..." on it's own line.  It is also recorded in the
        errors table under the new column "need_more_info".
      
        Add inferred column to the errors table.  This is the same value as
        the inferred variable in tblog_find_error.
      
        Add revision column to errors table to make it easy to tell which
        algorithm was used to determine the error.
      95a3a6a7