1. 14 Jul, 2007 1 commit
  2. 20 Jun, 2007 1 commit
  3. 19 Jun, 2007 2 commits
    • Leigh B. Stoller's avatar
      Minor fixes to how I handle case where there is a last resource record · 720bb591
      Leigh B. Stoller authored
      pointer. This indicates a previous failure that really got hosed and
      left the reources table and/or stats table in a bad state which must
      be fixed by hand (for now).
      720bb591
    • Leigh B. Stoller's avatar
      Big update to the stats gathering code ... · 495f6803
      Leigh B. Stoller authored
      This change attempts to make the stats gathering code more reliable by
      not relying on the testbed_stats records to reconstruct usage
      statistics.  The main source of errors and total confusion in the
      current stats code is that testbed_stats includes all the errors and
      transitions, from which I have to reconstruct what happened in order
      to determine usage by a project or user.
      
      The new stats code still generates the testbed_stats code, but actual
      usage is recorded as it happens, in the experiment_resources table, as
      swapins, swapouts, and swapmods occur. Its also much faster to compute
      the data for the tables in the web interface, not having to scan a
      zillion testbed_stats records in php.
      
      There is a time consuming update to the records that takes place with
      a lot of tables locked.
      495f6803
  4. 31 May, 2007 2 commits
  5. 15 May, 2007 1 commit
    • Leigh B. Stoller's avatar
      Checkpoint changes that have been discussed in the last few weeks: · c4f53202
      Leigh B. Stoller authored
      * Records are now "help open" when a run is stopped. When the next run
        is started, a check is made to see if the files
        (/project/$pid/exp/$eid) have changed, and if so a new version of the
        archive is committed before the next run is started.
      
      * Change the way swapmod is handled within an instance. A new option
        on the ShowExp page called Modify Resources. The intent is to allow
        an instance to be modified without having to start and stop runs,
        which tends to clutter things up, according to our user base. So, if
        you are within a run, that run is reset (reused) after the swapmod is
        finished. You can do this as many times as you like. If you are
        between runs (last operation was a stoprun), do the swapmod and then
        "speculatively" start a new run. Subsequent modifies reuse the that
        run again, as above.
      
        I think this is what Kevin was after ... there are some UI issues
        that may need to be resolved, will wait to hear what people have to
        say.
      
      * Revising a record is now supported. Export, change in place, and
        then use the Revise link on the ShowRun page. Currently this has to
        happen from the export directory on ops, but eventually allow an
        upload (to correspond to downloaded exports)
      
      * Check to see if export already exists, and give warning. Added a
        checkbox that allows user to overwrite the export.
      
      * A bunch of minor UI changes to the various template pages.
      c4f53202
  6. 07 May, 2007 1 commit
    • Leigh B. Stoller's avatar
      Mostly this commit is the switch from SVN archives to ZIP archives. · 55d1bb6e
      Leigh B. Stoller authored
      Other stuff leaked in too ...
      
      I did separate out a lot of tbsetup/libArchive into db/Archive, and
      whats left in libArchive.pm will eventually move over into the
      Template library.
      
      Note that I have dropped archiving of plain experiments; this is not
      really owrth it outside the workbench context, and it just wastes
      space and makes a lot if stuff painful in the web interface.
      55d1bb6e
  7. 26 Apr, 2007 1 commit
  8. 25 Apr, 2007 1 commit
  9. 17 Apr, 2007 1 commit
  10. 08 Mar, 2007 1 commit
  11. 06 Mar, 2007 1 commit
  12. 14 Feb, 2007 1 commit
  13. 18 Jan, 2007 1 commit
    • Leigh B. Stoller's avatar
      Change how the archive stuff is presented in the template instance and · 808f43ec
      Leigh B. Stoller authored
      run display pages.
      
      * Add new code to archive_control and libarchive to create tags (in
        optional subdirs) of the tags directory. These tags are simply tags
        on the current trunk at the time.
      
      * Change to template instantiate and start/top run code to create a
        "runs" directory. Within the runs directory are subdirs named for
        each runid, and within this directory are start and stop run tags.
        Eventually add some other tags when doing swapmods, but do not have
        that figured out yet.
      
      * Change the viewcvs code to add an "embedded" view of an archive.
        This is just an alternate view that has no header/footer goo, but
        only the actual svn contents. The main reason for this change is
        that the header and footers take up a huge amount of space and
        contribute nothing. Smaller means its easier to add to the instance
        and run show pages.
      
      * Add the above mentioned embedded archive view to the instance and
        run show pages. On the instance page you see the directory of all
        runids, and on the show run page you see the subdir for that run.
      808f43ec
  14. 09 Jan, 2007 1 commit
  15. 09 Nov, 2006 1 commit
  16. 03 Oct, 2006 1 commit
  17. 02 Oct, 2006 1 commit
  18. 29 Sep, 2006 2 commits
  19. 28 Sep, 2006 1 commit
  20. 27 Sep, 2006 1 commit
  21. 26 Sep, 2006 1 commit
  22. 20 Sep, 2006 1 commit
    • Leigh B. Stoller's avatar
      By popular demand, you can now force a swap modify to be done when · b9161642
      Leigh B. Stoller authored
      doing a Start Run. On the web page, there is a new checkbox, and
      on ops, template_startrun takes a new -m option.
      
      Caveat: You cannot specify a new NS file, yet. The original file is
      reparsed, and the idea is that a change in the template parameters
      will result in a change to the topology. I will add the ability to
      specify a new NS file in the next revision of this change.
      
      If you really really want to change the NS file, go to
      /proj/$pid/exp/$eid/archive/nsdata and edit nsfile.ns ...
      
      In addtion, DATASTORE is now defined while parsing the NS file. This
      turned to be quite the headache!
      b9161642
  23. 19 Sep, 2006 1 commit
  24. 14 Sep, 2006 1 commit
    • Leigh B. Stoller's avatar
      Rework the event handling for the program agent so that both the · aead4580
      Leigh B. Stoller authored
      reload and halt events send proper completion events. This is required
      for stoprun and startrun to work correctly. On stoprun the logs are
      not collected until the programs have stopped, and on startrun we do
      not want to proceed until all the agents have reloaded their
      environments.
      aead4580
  25. 10 Sep, 2006 1 commit
    • Leigh B. Stoller's avatar
      The bulk of this commit adds the ability to run the program agent on ops · e8bb6bca
      Leigh B. Stoller authored
      so that users can schedule program events to run there. For example:
      
      	set myprog [new Program $ns]
      	$myprog set node "ops"
      	$myprog set command "/usr/bin/env >& /tmp/foo"
      
      	$ns at 10 "$myprog start"
      or
      	tevc -e pid/eid now myprog start
      
      Since the program agent cannot talk to tmcd from ops, there are new
      routines to create the config files that the program agent uses, in
      the expertment tbdata directory.
      
      I also rewrote the eventsys.proxy script that starts the event
      scheduler on ops; I rolled the startup of the program agent into this
      script, via new -a option which is passed over from boss when an ops
      program agent is detected in the virt topology. This keep the number
      of new processes on ops to a small number.
      
      Also part of the above rewrite is that we now catch when event
      scheduler (or the program agent) exits abnormally, sending email to
      tbops and the swapper of the experiment. We have been seeing abnormal
      exits of the scheduler and it would good to detect and see if we can
      figure out what is going wrong.
      
      Other small bug fixes in experiment run.
      e8bb6bca
  26. 07 Sep, 2006 1 commit
    • Leigh B. Stoller's avatar
      Some changes to how log files are handled; this too way too long to · c01f7b3e
      Leigh B. Stoller authored
      do!
      
      The original operation was to save up every log file forever in the
      work directory, and copy that out to both the user directory and the
      info directory (long term archive). When I cleaned /proj on ops
      yesterday of all this old cruft, I recoved 17GB of disk space. Yow!
      
      So, the new operation is:
      
      * Only files that end in .log are copied to the user directory. No
        longer copying out .top, .ptop, and a couple of other logs; 99% of
        users never look at these things. We still have them available to us
        though, on boss.
      
      * At the beginning of each swap operation, clean out the work
        directory of all the old log files. These are named a variety of
        ways, so I use some pattern patches to do this.
      
      * Jigger the names a little so that we do not name things in the form
        "$$.log", to avoid copying out different named files to the user
        directory each time; instead link the .log file to the real output
        file so that it gets overwritten each time, while still getting the
        per-swap files for long term storage.
      c01f7b3e
  27. 05 Sep, 2006 1 commit
  28. 25 Aug, 2006 1 commit
    • Leigh B. Stoller's avatar
      Add support for dynamic registration of ports on experimental nodes so · 73102ef8
      Leigh B. Stoller authored
      that clients and servers can avoid using hardwired ports on those
      experimental nodes. I have added the following tmcd operation:
      
      	tmcc portregister <service> [<port>]
      
      where we assume its the control network IP (from the DB), and the pid/eid
      of the node the experiment belongs to. The given port is entered into
      the port_registration table for the experiment, using the service as the
      tag. Supplying port=0 clears the registration from the table.
      
      When called like:
      
      	tmcc portregister <service>
      
      we return the registered port, or nothing.
      
      I hacked up a little C library module in libtb so that there is something
      that looks like a C interface to this:
      
       	int
       	PortRegister(char *service, int port);
      
       	int
       	PortLookup(char *service, char *hostname, int namelen, int *port);
      
      The above routines call out to tmcc of course.
      
      Lastly, I changed the sync server and client to use the new port
      registration, via the library calls above.
      
      There are other emulab services that need to be changed as well, but
      they can be done on an as needed basis.
      73102ef8
  29. 03 Aug, 2006 1 commit
    • Leigh B. Stoller's avatar
      Support for capturing the trace data that is stored in the pcal files · 4ce9c421
      Leigh B. Stoller authored
      into per-experiment databases on ops. Additional support for reconsituting
      those databases back into temporary databases on ops, for post processing.
      
      * This revision relies on the "snort" port (/usr/ports/security/snort)
        to read the pcap files and load them into a database. The schema is
        probably not ideal, but its better then nothing. See the file
        ops:/usr/local/share/examples/snort/create_mysql for the schema.
      
      * For simplicity, I have hooked into loghole, which already had all
        the code for downloading the trace data. I added some new methods to
        the XMLRPC server for loghole to use, to get the users DB password
        and the name of the per-experiment database. There is a new slot in
        the traces table that indicates that the trace should be snorted to
        its DB. In case you forgot, at the end of a run or when the instance
        is swapped out, loghole is run to download the trace data.
      
      * For reconsituting, there are lots of additions to opsdb_control and
        opsdb_control.proxy to create "temporary" databases and load them
        from a dump file that is stored in the archive. I've added a button
        to the Template Record page, inappropriately called "Analyze" since
        right now all it does is reconsitute the trace data into a DB on
        ops.
      
        Currently, the only indication of what has been done (the name of
        the DBs created on ops) is the log email that the user gets. A
        future project is tell the user this info in the web interface.
      
      * To turn on database capturing of trace data, do this in your NS
        file:
      
      	set link0 ...
      	$link0 trace
      	$link0 trace_snaplen 128
      	$link0 trace_db 1
      
         the increase in snaplen is optional, but a good idea if you want
         snort to undertand more then just ip headers.
      
      * Also some changes to the parser to allow plain experiments to take
        advantage of all this stuff. To simple get yourself a per-experiment
        DB, put this in your NS file:
      
      	tb-set-dpdb 1
      
        however, anytime you turn trace_db on for a link or lan, you
        automatically get a per-experiment DB.
      
      * To capture the trace data to the DB, you can run loghole by hand:
      
      	loghole sync -s
      
        the -s option turns on the "post-process" phase of loghole.
      4ce9c421
  30. 21 Jul, 2006 1 commit
  31. 18 Jul, 2006 1 commit
    • Leigh B. Stoller's avatar
      Changes necessary for moving most of the stuff in the node_types · 624a0364
      Leigh B. Stoller authored
      table, into a new table called node_type_attributes, which is intended
      to be a more extensible way of describing nodes.
      
      The only things left in the node_types table will be type,class and the
      various isXXX boolean flags, since we use those in numerous joins all over
      the system (ie: when discriminating amongst nodes).
      
      For the most part, all of that other stuff is rarely used, or used in
      contexts where the information is needed, but not for type descrimination.
      Still, it made for a lot of queries to change!
      
      Along the way I added a NodeType library module that represents the type
      info as a perl object. I also beefed up the existing Node module, and
      started using it in more places. I also added an Interfaces module, but I
      have not done much with that yet.
      
      I have not yet removed all the slots from the node_types table; I plan to
      run the new code for a few days and then remove the slots.
      
      Example using the new NodeType object:
      
      	use NodeType;
      
      	my $typeinfo = NodeType->Lookup($type);
      
              if ($typeinfo->control_interface(\$control_iface) ||
                  !$control_iface) {
        	    warn "No control interface for $type is defined in the DB!\n";
              }
      
      or using the Node:
      
      	use Node;
      
              my $nodeobject = Node->Lookup($node_id);
              my $imageable  = $nodeobject->NodeTypeInfo()->imageable();
      or
              my $rebootable = $nodeobject->isrebootable();
      or
              $nodeobject->NodeTypeAttribute("control_interface", \$control_iface);
      
      Lots of way to accomplish the same thing, but the main point is that the
      Node is able to override the NodeType (if it wants to), which I think is
      necessary for flexibly describing one/two of a kind things like switches, etc.
      624a0364
  32. 07 Jul, 2006 1 commit
  33. 05 Jul, 2006 1 commit
  34. 03 Jul, 2006 1 commit
  35. 30 Jun, 2006 1 commit