1. 09 Feb, 2010 1 commit
  2. 21 Dec, 2009 1 commit
    • Leigh Stoller's avatar
      New approach to dealing with nodes that fail to boot is os_setup, and · 5cf6aad2
      Leigh Stoller authored
      land in hwdown.
      
      Currently, if a node fails to boot in os_setup and the node is running
      a system image, it is moved into hwdown. 99% of the time this is
      wasted work; the node did not fail for hardware reasons, but for some
      other reason that is transient.
      
      The new approach is to move the node into another holding experiment,
      emulab-ops/hwcheckup. The daemon watches that experiment, and nodes
      that land in it are freshly reloaded with the default image and
      rebooted. If the node reboots okay after reload, it is released back
      into the free pool. If it fails any part of the reload/reboot, it is
      officially moved into hwdown.
      
      Another possible use; if you have a suspect node, you go wiggle some
      hardware, and instead of releasing it into the free pool, you move it
      into hwcheckup, to see if it reloads/reboots. If not, it lands in
      hwdown again. Then you break out the hammer.
      
      Most of the changes in Node.pm, libdb.pm, and os_setup are
      organizational changes to make the code cleaner.
      5cf6aad2
  3. 23 Jul, 2009 1 commit
  4. 11 Jun, 2009 1 commit
  5. 20 Apr, 2009 1 commit
  6. 03 Apr, 2009 1 commit
  7. 02 Mar, 2009 1 commit
    • Leigh Stoller's avatar
      A bunch of changes for a "standalone" clearinghouse. Presently this · 60f04310
      Leigh Stoller authored
      its really a hugely stripped down Emulab boss install, using a very
      short version of install/boss-install to get a few things into place.
      
      I refactored a few things in both the protogeni code and the Emulab
      code, and whacked a bunch of makefiles and configure stuff. The result
      is that we only need to install about 10-12 files from the Emulab
      code, plus the protogeni code. Quite manageable, if you don't mind
      that it requires FreeBSD 6.X ... Still, I think it satisfies the
      requirement that we have a packaged clearinghouse that can be run
      standalone from a running Emulab site.
      60f04310
  8. 07 Jan, 2009 1 commit
    • Leigh Stoller's avatar
      Change the way that elabinelab communicates with the outer emulab. · 0a004176
      Leigh Stoller authored
      Instead of passing along a set of arguments that needs to be turned
      into command line arguments to the proxy, pass along an xmldoc
      representing the arguments. This xmldoc is passed through the rpc
      server to the snmpit proxy, where it reconstructs the arguments. This
      avoids all that cruftiness in the rpc server (also error prone) and
      make it easier to add new remote methods, say for supporting eventual
      elabibelab firewalls. Note that there are currently two versions of
      the proxy and two methods in the rpc server, so that existing emulabs
      will work.
      
      I also added support for port enable/disable/etc from the inner elab.
      There is also the beginning of firewall support. I pass the stack
      module argument along, but currently not actually doing control stack
      operations from the proxy. Needs more work.
      
      Oh, I copied the cvs repo file for the original proxy so that we do
      not lose the cvs history.
      
      See corresponding commitlog for snmpit for description of other
      changes.
      0a004176
  9. 05 Aug, 2008 1 commit
  10. 18 May, 2008 1 commit
  11. 10 Jul, 2007 1 commit
    • Leigh Stoller's avatar
      Changes to switch from using tag to rtag. Note that this commit requires · 7a2ef6f1
      Leigh Stoller authored
      a locally modifed version of CVS that does a couple of extra things. The
      hacks are quite simple and currently on users. Briefly, there is a new
      config file option to specify a program to run when the tag operation is
      complete (similar to a module hook for rtag), and in tag.c I run that hook.
      
      Check out a sandbox as before:
      
        cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo rtag mytag XXXXX
      
      To create a new template (along the trunk), use the tag operation instead:
      
      	cd sandbox
      	cvs tag mtag
      
      and what for the email that says the template is created. You can
      continue to work along the trunk in this manner, although note that
      the .template file is changing (by the backend commit), and you might
      find it less distriacting to do a cvs update each time.
      
      To work along a branch:
      
      	cd sandbox
      	cvs tag -r existing_tag -b newtag1
              cvs update -r newtag
              make changes and commits
      	cvs tag newtag2
      
      When working along branched you will want to do a cvs update to get
      the .template file in sync.
      7a2ef6f1
  12. 09 Jul, 2007 1 commit
    • Leigh Stoller's avatar
      Checkpoint my cvs interface to the workbench. This first cut uses the · 8371fc79
      Leigh Stoller authored
      "rtag" directive to initiate template modify operations. So, to get started
      you do a checkout:
      
        cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo checkout XXXXX
      
      where XXXXX is the part of the guid (10000/1) before the slash. Might try
      and roll all templates into a single project wide repo at some point, to
      avoid the extraneous path stuff, but didn't want to worry that just yet.
      
      Okay, so have a checkout. You can work along the trunk, doing commits. To
      create a new template (a modify of the existing template), you tag the tree
      using rtag:
      
        cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo rtag mytag XXXXX
      
      A template modify is started at the end, and you should probably wait for
      email before continuing. Eventually I will need to add locking of some
      kind, but I have to do the modify in the background, or else I get deadlock
      cause cvs keeps the repo locked, and the modify also needs to access it.
      
      Each time you tag along the trunk, you get a modified template, which in
      the history diagram looks like:
      
        10000/1 --> 10000/2 --> 10000/3 ...
      
      If you want to branch, say at 10000/2 you can create a branch tag using rtag:
      
        cvs -d [cut] rtag -r T10000/2 -b mytag2 XXXXX
      
      You can also use your own tags for -r option, but I also create a TXXXXX/YY
      tag at each template modify, which is easy to remember.
      
      Then update your sandbox to the new branch, commit changes along that
      branch, and then later use rtag again to initiate a template modify
      operation:
      
        cvs update -r mytag2
        cvs commit ...
        cvs -d [cut] rtag -r mytag2 mytag3 XXXXX
      
      And now the history diagram looks like:
      
        10000/1 --> 10000/2 --> 10000/3 ...
                      |
                      |
                      -> 10000/4 ...
      
      You should be able to mix interaction via the web with interaction via the
      cvs interface. I've tested it, although not extensively.
      8371fc79
  13. 30 May, 2007 1 commit
  14. 23 May, 2007 1 commit
    • Leigh Stoller's avatar
      First cut at template checkout and commit from a checkout. The interface · b674bc7b
      Leigh Stoller authored
      described is the one exported to ops via the XMLRPC interface. This is
      just playing aroundl no doubt this stuff is going to change.
      
      * template_checkout guid/vers
      
        Checkout a copy of the template to the current working directory.
      
      * template_commit
      
        Modify the previous template checkout, using the nsfile contained in
        the tbdata directory (subdir of the current directory). In other words,
        the current template is modified, creating a new template in the
        current working directory (the current directory refers to the new
        template).
      
        The datastore subdir is imported into the new template, but that is
        the only directory that is imported at present. Might change that.
      
      So this sounds much cooler then it really is. Why?
      
      * This only works from ops.
      
      * The "current directory" must be one of the standard approved directories
        (/proj, /users, /groups).
      
      * Cause, boss reads and writes that directory via NFS, as told to it
        by the xmlrpc client.
      
      At some point in the future it would be nice to support something
      fancier, using a custom transport, but lets see how this goes.
      b674bc7b
  15. 17 May, 2007 1 commit
  16. 15 May, 2007 1 commit
    • Leigh Stoller's avatar
      Checkpoint changes that have been discussed in the last few weeks: · c4f53202
      Leigh Stoller authored
      * Records are now "help open" when a run is stopped. When the next run
        is started, a check is made to see if the files
        (/project/$pid/exp/$eid) have changed, and if so a new version of the
        archive is committed before the next run is started.
      
      * Change the way swapmod is handled within an instance. A new option
        on the ShowExp page called Modify Resources. The intent is to allow
        an instance to be modified without having to start and stop runs,
        which tends to clutter things up, according to our user base. So, if
        you are within a run, that run is reset (reused) after the swapmod is
        finished. You can do this as many times as you like. If you are
        between runs (last operation was a stoprun), do the swapmod and then
        "speculatively" start a new run. Subsequent modifies reuse the that
        run again, as above.
      
        I think this is what Kevin was after ... there are some UI issues
        that may need to be resolved, will wait to hear what people have to
        say.
      
      * Revising a record is now supported. Export, change in place, and
        then use the Revise link on the ShowRun page. Currently this has to
        happen from the export directory on ops, but eventually allow an
        upload (to correspond to downloaded exports)
      
      * Check to see if export already exists, and give warning. Added a
        checkbox that allows user to overwrite the export.
      
      * A bunch of minor UI changes to the various template pages.
      c4f53202
  17. 02 Mar, 2007 1 commit
    • David Johnson's avatar
      Adds rmcp support (for new wifi pcs) to the power command. For now, you · d31ab2bd
      David Johnson authored
      have to re-run the swig-wrappers target in tools/rmanage/GNUmakefile to
      generate the wrapper and perl module; this must of course be done when
      changes are made to the rmcp libs.
      
        * GNUmakefile.in, configure, configure.in: add tools/rmanage
        * tbsetup/GNUmakefile.in, tbsetup/power*.in: add rmcp to power command
        * tools/GNUmakefile.in: add rmanage
        * tools/rmanage/*.c,*.h: bugfixes, swig helper methods, etc.
        * tools/rmanage/rmcp.i: swig import control file
        * tools/rmanage/rmcp.pm,rmcp_wrap.c: rmcp wrapper/module generated by swig
      d31ab2bd
  18. 19 Jan, 2007 1 commit
  19. 18 Jan, 2007 4 commits
  20. 25 Oct, 2006 1 commit
    • Leigh Stoller's avatar
      Makefile Whacking! Try to deal with the problem caused by the delay · 7590f9c5
      Leigh Stoller authored
      between when something is installed and when post-install runs. Short
      of a global lock (which we probably need anyway someday), my solution
      is this. In your makefiles, add these variables before the line that
      has the include of $(TESTBED_SRCDIR)/GNUmakerules:
      
      	SETUID_BIN_SCRIPTS   =
      	SETUID_SBIN_SCRIPTS  =
      
      I have added three new rules to GNUmakerules that look like this:
      
      	$(addprefix $(SBINDIR)/, $(SETUID_SBIN_SCRIPTS)): $(SBINDIR)/%: %
      		echo "Installing (setuid) $<"
      		-mkdir -p $(INSTALL_SBINDIR)
      		$(SUDO) $(INSTALL) -o root -m 4755 $< $@
      
      Yep, your eyes ain't lying to you; use sudo to run the target so that
      install does the right thing (which is that the old file is not
      replaced until the new one has the proper attributes on it).
      
      Note that post-install is still needed for the initial install, but
      should no longer be needed for day to day installs since all that other
      stuff post-install does is mkdir/chmod on directories.
      7590f9c5
  21. 24 Oct, 2006 1 commit
  22. 18 Oct, 2006 1 commit
  23. 16 Oct, 2006 1 commit
  24. 08 Aug, 2006 1 commit
  25. 03 Aug, 2006 1 commit
    • Leigh Stoller's avatar
      Support for capturing the trace data that is stored in the pcal files · 4ce9c421
      Leigh Stoller authored
      into per-experiment databases on ops. Additional support for reconsituting
      those databases back into temporary databases on ops, for post processing.
      
      * This revision relies on the "snort" port (/usr/ports/security/snort)
        to read the pcap files and load them into a database. The schema is
        probably not ideal, but its better then nothing. See the file
        ops:/usr/local/share/examples/snort/create_mysql for the schema.
      
      * For simplicity, I have hooked into loghole, which already had all
        the code for downloading the trace data. I added some new methods to
        the XMLRPC server for loghole to use, to get the users DB password
        and the name of the per-experiment database. There is a new slot in
        the traces table that indicates that the trace should be snorted to
        its DB. In case you forgot, at the end of a run or when the instance
        is swapped out, loghole is run to download the trace data.
      
      * For reconsituting, there are lots of additions to opsdb_control and
        opsdb_control.proxy to create "temporary" databases and load them
        from a dump file that is stored in the archive. I've added a button
        to the Template Record page, inappropriately called "Analyze" since
        right now all it does is reconsitute the trace data into a DB on
        ops.
      
        Currently, the only indication of what has been done (the name of
        the DBs created on ops) is the log email that the user gets. A
        future project is tell the user this info in the web interface.
      
      * To turn on database capturing of trace data, do this in your NS
        file:
      
      	set link0 ...
      	$link0 trace
      	$link0 trace_snaplen 128
      	$link0 trace_db 1
      
         the increase in snaplen is optional, but a good idea if you want
         snort to undertand more then just ip headers.
      
      * Also some changes to the parser to allow plain experiments to take
        advantage of all this stuff. To simple get yourself a per-experiment
        DB, put this in your NS file:
      
      	tb-set-dpdb 1
      
        however, anytime you turn trace_db on for a link or lan, you
        automatically get a per-experiment DB.
      
      * To capture the trace data to the DB, you can run loghole by hand:
      
      	loghole sync -s
      
        the -s option turns on the "post-process" phase of loghole.
      4ce9c421
  26. 28 Jul, 2006 1 commit
    • Leigh Stoller's avatar
      Add a "Create Template from Instance" ability. Basically, you can · a651da71
      Leigh Stoller authored
      create a new template (well, really a modify) from the current
      swapped in experiment. This allows you to create a template, swap in
      an instance, modify the datastore in the instance (which is a copy
      of the datastore in the template), and then create a new template
      using the datastore and nsfile from the instance. This is a new menu
      item on the showexp page for the instance.
      
      Also in this commit are fixes and improvements to the new navagation
      bar that I recently added.
      a651da71
  27. 26 Jul, 2006 1 commit
  28. 11 Jul, 2006 1 commit
  29. 21 Jun, 2006 1 commit
  30. 30 May, 2006 1 commit
    • Leigh Stoller's avatar
      Add an export option to the record listing. A new button on the Template · 2cfe4630
      Leigh Stoller authored
      Record page lets you export the contents of the archive that corresponds
      to that record, along with an XML file that describes the various DB bits
      for the template and instance.
      
      This is just a first cut so that Mike can start playing around. Subject to
      change, I'm sure.
      
      The archive is dumped to /proj/$pid/exports/$guid/$vers/$exptidx, which
      is basically the last commit of the instance when it was terminated.
      
      The xml file is called export.xml and is placed in the top level directory
      of the above directory. The file is created with XML::Simple, and a typical
      XML file might look like:
      
      <instance>
        <bindings>
          <name>NodeCount</name>
          <description>Number of nodes!</description>
          <value>1</value>
        </bindings>
        <bindings>
          <name>OS</name>
          <description></description>
          <value>RHL90-STD</value>
        </bindings>
        <bindings>
          <name>ScriptArgs</name>
          <description></description>
          <value>-b</value>
        </bindings>
        <eid>NewOne-V2</eid>
        <guid>10149/2</guid>
        <metadata>
          <name>M1</name>
          <guid>10162/1</guid>
          <value>Some metadata</value>
        </metadata>
        <pid>testbed</pid>
        <runs>
          <name>1</name>
          <archive_tag>T20060526-082533-172_endexp</archive_tag>
          <description></description>
          <exptidx>110</exptidx>
          <idx>1</idx>
          <runid>NewOne-V2</runid>
          <start_time>2006-05-26 08:23:02</start_time>
          <stop_time>2006-05-26 08:25:16</stop_time>
        </runs>
        <uid>stoller</uid>
      </instance>
      2cfe4630
  31. 15 May, 2006 1 commit
    • Mike Hibler's avatar
      Initial "Inner Plab" support. In your NS file, you declare one node: · 9512772e
      Mike Hibler authored
      tb-set-node-plab-role $plc plc
      
      to make it the PLC node.  Then any number of other nodes are declared as:
      
      tb-set-node-plab-role $plab1 node
      
      to make them inner plab nodes.  Unlike elabinelab, there is no magic
      "tb-plab-in-elab" command which implies the topology, you put all the
      plab nodes in a LAN or whatever yourself.  This may or may not be a good idea.
      
      Anyway, these NS commands set DB state in virt_nodes and reserved much like
      elabinelab.  During swapin, the dhcpd.conf file is rewritten so that
      inner plab nodes have their "filename" set to "pxelinux.0" and their
      "next-server" set to the designated PLC node.  The PLC node will then be
      loaded/booted before anything is done to the inner-plab nodes.  After
      it comes up, the inner plab nodes are rebooted and declared as up.
      There is a new tmcd command "eplabconfig" (suggestions for a new name
      welcom!), which returns info like:
      
          NAME=plc ROLE=plc IP=155.98.36.3 MAC=00d0b713f57d
          NAME=plab1 ROLE=node IP=155.98.36.10 MAC=0002b3877a4f
          NAME=plab2 ROLE=node IP=155.98.36.34 MAC=00d0b7141057
      
      to just the PLC node (returns nothing to any other node).
      
      The implications of this setup are:
      
       * The PLC node must act as a TFTP server as we have discussed in the past.
         The TMCC info above is hopefully enough to configure pxelinux, if not
         we can change it.
      
       * The PLC node is responsible for loading the disks of inner plab nodes.
         This is implied by the setup, where we change the dhcpd.conf file before
         doing anything to the inner nodes.  Thus, once the inner nodes are
         rebooted, they will be talking pxelinux with PLC, and not to boss.
         This step is dubious, as we could no doubt load the disks faster than
         whatever plab uses can.  But it simplified the setup (and is more
         realistic!).  The alternative, which is something that might be useful
         anyway, is to introduce a "state" after which nodes have been reloaded
         but before they are rebooted.  With that, we can reload the plab nodes
         and then change the dhcpd.conf file so when they reboot they start
         talking to the PLC.
      9512772e
  32. 12 May, 2006 1 commit
    • Leigh Stoller's avatar
      Redo the entire template library. I've been meaning to use perl · 78503406
      Leigh Stoller authored
      "object" and this was a good opportunity to see if they are useful and
      easy enough to use. Yep they are; the code is much cleaner with many
      fewer utility functions to get at stuff. I recommend this approach
      from now on.
      
      The problem is the php side, which ends up duplicating some stuff, but
      in the old style. This is not so bad for the template code since I
      have made it a point not to do anything but display functions in php;
      all modifications are handled in the backend.
      78503406
  33. 05 May, 2006 1 commit
  34. 30 Mar, 2006 1 commit
  35. 28 Mar, 2006 1 commit
  36. 22 Feb, 2006 1 commit
  37. 07 Feb, 2006 1 commit