1. 18 Oct, 2006 1 commit
  2. 12 Oct, 2006 1 commit
    • Leigh Stoller's avatar
      By popular demand, give user a choice of where to get the next set of · bb996961
      Leigh Stoller authored
      (initial) parameters for a new run. Three choices right now; from the
      template itself, from the instance, or from the previous run. On the
      web interface this is presented as three buttons. On ops, it is the
      the -y option, which takes one of template,instance,lastrun as its
      argument (you can of course combine the -y option with an XML file to
      override specific params).
      
      At present, there is no default. Lets give it a chance to sink in
      before I pick something that will annoy 50% of the people 75% of the
      time.
      bb996961
  3. 06 Jun, 2006 1 commit
    • Leigh Stoller's avatar
      A set of cosmetic changes. Actually one consmetic changes in a lot of · ecab2742
      Leigh Stoller authored
      places. I refactored the "busy" logo stuff that is done in various
      scripts in various formats into a couple of functions. I have not
      changed all the code that does the busy thing, but I got a bunch this
      time around. In menu.php I added a couple of functions (that spit out
      little bits of javascript).
      
      	STARTBUSY($msg)
      
      	Display a centered message (ie: "Starting Swapout"), then a
      	plead to be patient, and then the "busy" icon (spinning thing
      	that my Mac does too).
      
      
      	STOPBUSY();
      
      	Clear the busy icon, and replace "Working ..." with "Done"
      
      
      	CLEARBUSY();
      
      	Clear the busy icon and delete "Working ..."
      
      
      	PAGEREPLACE($url)
      
      	Delay one second and then replace the current page with the
      	new page.
      ecab2742
  4. 26 May, 2006 1 commit
    • Leigh Stoller's avatar
      Fix another Eric nit: · 1605bf1e
      Leigh Stoller authored
        + On the swap-in status page (showing the swap-in progress), there's a link
          to the template page but no link to the instance page.  Conversely, on the
          swap-out page ("Terminate Instance"), there's a link to the instance but
          not to the template.
      1605bf1e
  5. 16 May, 2006 1 commit
  6. 05 May, 2006 1 commit
  7. 31 Mar, 2006 1 commit
    • Leigh Stoller's avatar
      More experiment run stuff. Also the NS file parsing support that was · 74bf8d1d
      Leigh Stoller authored
      needed (reparsing of NS file with current bindings).
      
      Mostly I decided that I made a huge mistake in using the exptidx as the
      main cross index. I started to change that, which took a bunch of time.
      Still more tables to fix up, but I think all of this code is gonna need
      a major reorg once I am no longer under the gun to get it done.
      74bf8d1d
  8. 28 Mar, 2006 1 commit
  9. 18 Dec, 2005 1 commit
    • Leigh Stoller's avatar
      Move the bulk of what was in showlogfile, into showlogfile_sup, so · 41f4d6a5
      Leigh Stoller authored
      that it can be "included" into other scripts; showlogfile, beginexp,
      swapexp, modifyexp, endexp.
      
      Each of these scripts now appends the showlog directly to the page,
      instead of providing a link to a different page.
      
      Also changed the showlog code to place the output into an inner iframe
      so that it can be sized to fit in the window, without the outer stuff
      (like the menus) getting scrolled away.
      41f4d6a5
  10. 26 Oct, 2005 1 commit
  11. 17 Aug, 2005 1 commit
    • Leigh Stoller's avatar
      The Emulab Knowledge Base! · 6f08c442
      Leigh Stoller authored
      Okay, I implemented a primitive Knowledge Base! The current contents are
      *all* the existing FAQ entries, which I entered manually. Here are the
      details.
      
      * My reason for doing this is that we need something very simple. The wiki
        is too much of a barrier, and its search capabilities are pathetic.
      
      * The search page for the Knowledge Base is:
      
      	https://www.emulab.net/kb-search.php3
      
        Fairly primitive keyword search. Turns out that mysql 4.0 has a bunch for
        really good text searching functions built in, but we run 3.23 ... so I
        had to roll it myself. So, its a simple keyword (space or comma
        separated) search, no regular expressions.
      
      * Each DB record has a "faq_entry" flag, so creating the current FAQ on the
        fly from the DB is easy. See:
      
      	https://www.emulab.net/kb-faq.php3
      
      * In reddot mode, you can add new KB entries:
      
      	https://www.emulab.net/kb-manage.php3
      
        The form is fairly obvious but here are details anyway:
      
          Section Name: Choose an existing title, or make up a new one.
          Title:        The title of the KB (or FAQ) entry.
          Faq Entry:    Check this box if the new entry should show up in the FAQ.
          X Ref Tag:    A token so you can refer to other KB entries by name,
                        instead of by its index. Within the KB entry you would
                        write: <a href=kb-show.php3?xref_tag=sometag>
          Body:         Whatever you like. I took the existing FAQ entries and
                        stuck them with no changes except for the xref_tag
                        mentioned about (since some entries referenced other
                        entries).
      
      * Once you click on sumbit, you will see the entry as it will appear to
        users, along with a submenu to Modify/Delete/Add entries. You can modify
        the current entry from that menu. Mere users do not see this menu, only
        when in reddot mode.
      
      * The intent here is that we can generate new entries really easy, right
        from email if you like (with appropriate <pre> or <xmp> tags around it).
      
      * I have added sql/knowlbase-create.sql and a makefile target to
        generate that file when creating a distribution. I also added a section
        to install/boss-install to insert the entries into the new DB.
      
      * I hooked the search function into the existing Search Documentation link.
        We know search both the Knowledge Base *and* the Documentation on doc
        searches. This probably needs a little more work to get right.
      
      * I changed a lot of faq links to be more consistent and to reference
        the proper xref_tags (#swapping instead of #UTT-34).
      6f08c442
  12. 12 Jan, 2005 1 commit
  13. 10 Sep, 2004 1 commit
    • Leigh Stoller's avatar
      Small change to suexec code. This change has the potential for creating · 7e731fba
      Leigh Stoller authored
      unanticipated breakage. If that happens, just need to back out the
      changes under the "suexec-stuff" tag. However, the better solution will
      probably be to fix the PHP scripts that break by adding the proper
      groups in the call to suexec (in the web page, see below) or by fixing
      the backend Perl script that breaks.
      
      This fix is primarily to address the problem of some users being in more
      groups (cause of subgroups) then the max number of groups allowed
      (NGROUPS).  The groups that really mattered (say, for creating an
      experiment in a subgroup) could be left out cause they were at the end
      of the list.
      
      * suexec.c: Change how groups are handled. Instead of taking a single
        gid argument (the gid to setgid as), now takes a comma separated list
        of groups. Further, instead of doing a setgroups to the user's entire
        group list as specified in the groups file (getgroups), setgroups to
        just the groups listed on the command line, plus the user's primary
        group from the password file (this is to prevent potential breakage
        with accessing files from the users homedir, although might not really
        be necessary).
      
        This change is somewhat rational in the sense that in our case, suexec
        is not being used to run arbitrary user code (CGIs), but only to run
        specific scripts that we say should be run. The environment for
        running those scripts can be more tightly controlled then it would
        otherwise need to be if running some random CGI the user has in his
        public html directory.
      
      * www: Change the gid argument to SUEXEC() in a number of scripts so
        that the project and subgroup are explicitly given to suexec, as
        described above. For example, in beginexp:
      
      	SUEXEC(gid, "$pid,$unix_gid", ....);
      
        Aside: note that project names (pid) are always one to one with their
        unix group name, but subgroup names are not, and *always* have to be
        looked up in the DB, hence the "unix_gid" argument.
      
        Script breakage should require nothing more then adding the proper
        group to the list as above.
      7e731fba
  14. 13 May, 2004 1 commit
  15. 17 Nov, 2003 1 commit
    • Leigh Stoller's avatar
      Merge the two state machines (batchstate and state) into a single · 2025e0bd
      Leigh Stoller authored
      state machine (state). All of the stuff that was previously handled by
      using batchstate is now embedded into the one state machine. Of
      course, these mostly overlapped, so its not that much of a change,
      except that we also redid the machine, adding more states (for
      example, modify phases are now explicit. To get a picture of the
      actual state machine, on boss:
      
      		stategraph -o newstates EXPTSTATE
      		gv newstates.ps
      
      Things to note:
      
      * The "batchstate" slot of the experiments table is now used solely to
        provide a lock for batch daemon. A secondary change will be to
        change the slot name to something more appropriate, but it can
        happen anytime after this new stuff is installed.
      
      * I have left expt_locked for now, but another later change will be to remove
        expt_locked, and change it to active_busy or some such new state name in
        the state machine. I have removed most uses of expt_locked, except those
        that were necessary until there is a new state to replace it.
      
      * These new changes are an implementation of the new state machine,
        but I have not done anything fancy. Most of the code is the same as
        it was before.
      
      * I suspect that there are races with the batch daemon now, but they
        are going to be rare, and the end result is probably that a
        cancelation is delayed a little bit.
      2025e0bd
  16. 09 Oct, 2003 1 commit
    • Leigh Stoller's avatar
      Reorg of two aspects of node update. · 2641af4d
      Leigh Stoller authored
      * install-rpm, install-tarfile, spewrpmtar.php3, spewrpmtar.in: Pumped
        up even more! The db file we store in /var/db now records both the
        timestamp (of the file, or if remote the install time) and the MD5
        of the file that was installed. Locally, we can get this info when
        accessing the file via NFS (copymode on or off). Remote, we use wget
        to get the file, and so pass the timestamp along in the URL request,
        and let spewrpmtar.in determine if the file has changed. If the
        timestamp it gets is >= to the timestamp of the file, an error code
        of 304 (Not Modifed) is returned. Otherwise the file is returned.
      
        If the timestamps are different (remote, server sends back an actual
        file), the MD5 of the file is compared against the value stored. If
        they are equal, update the timestamp in the db file to avoid
        repeated MD5s (or server downloads) in the future. If the MD5 is
        different, then reinstall the tarball or rpm, and update the db file
        with the new timestamp and MD5. Presto, we have auto update capability!
      
        Caveat: I pass along the old MD5 in the URL, but it is currently
        ignored. I do not know if doing the MD5 on the server is a good
        idea, but obviously it is easy to add later. At the moment it
        happens on the node, which means wasted bandwidth when the timestamp
        has changed, but the file has not (probably not something that will
        happen in typical usage).
      
        Caveat: The timestamp used on remote nodes is the time the tarfile
        is installed (GM time of course). We could arrange to return the
        timestamp of the local file back to the node, but that would mean
        complicating the protocol (or using an http header) and I was not in
        the mood for that. In typical usage, I do not think that people will
        be changing tarfiles and rpms so rapidly that this will make a
        difference, but if it does, we can change it.
      
      * node_update.in, client side watchdog, and various web pages:
        Deflated node_update, removing all of the older ssh code. We now
        assume that all nodes will auto update on a periodic basis, via the
        watchdog that runs on all client nodes, including plab nodes.
      
        Changed the permission check to look for new UPDATE permission (used
        to be UPDATEACCOUNT). As before, it requires local_root or better.
        The reason for this is that node_update now implies more than just
        updating the accounts/mounts. The web pages have been changed to
        explain that in addition to mounts/accounts, rpms and tarfiles will
        also be updated. At the moment, this is still tied to a single
        variable (update_accounts) in the nodes table, but as Kirk requested
        at the meeting, it will probably be nice to split these out in the
        future.
      
        Added the ability to node_update a single node in an experiment (in
        addition to all nodes option on the showexp page). This has been
        added to the shownode webpage menu options.
      
        Changed locking code to use the newer wrapper states, and to move
        the experiment to RUNNING_LOCKED until the update completes. This is
        to prevent mayhem in the rest of the system (which could be dealt
        with, but is not worth the trouble; people have to wait until their
        initiated update is complete, before they can swap out the
        experiment).
      
        Added "short" mode to shownode routine, equiv to the recently added
        short mode for showexp. I use this on the confirmation page for
        updating a single node, giving the user a couple of pertinent (feel
        good) facts before they comfirm.
      2641af4d
  17. 30 Sep, 2003 1 commit
    • Leigh Stoller's avatar
      Up to now we have had two state variables associated with an experiment, · 4269dad1
      Leigh Stoller authored
      plus a lock field. The lock field was a simple "experiment locked, go away"
      slot that is easy to use when you do not care about the actual state that
      an experiment is in, just that it is in "transition" and should not be
      messed with.
      
      The other two state variables are "state" and "batchstate". The former
      (state) is the original variable that Chris added, and was used by the tb*
      scripts to make sure that the experiment was in the state each particular
      script wanted them to be in. But over time (and with the addition of so
      much wrapper goo around them), "state" has leaked out all over the place to
      determine what operations on an experiment are allowed, and if/when it
      should be displayed in various web pages. There are a set of transition
      states in addition to the usual "active", "swapped", etc like "swapping"
      that make testing state a pain in the butt.
      
      I added the other state variable ("batchstate") when I did the batch
      system, obviously! It was intended as a wrapper state to control access to
      the batch queue, and to prevent batch experiments from being messed with
      except when it was really okay (for example, its okay to terminate a
      swapped out batch experiment, but not a swapped in batch experiment since
      that would confuse the batch daemon). There are fewer of these states, plus
      one additional state for "modifying" experiments.
      
      So what I have done is change the system to use "batchstate" for all
      experiments to control entry into the swap system, from the web interface,
      from the command line, and from the batch daemon. The other state variable
      still exists, and will be brutally pushed back under the surface until its
      just a vague memory, used only by the original tb* scripts. This will
      happen over time, and the "batchstate" variable will be renamed once I am
      convinced that this was the right thing to do and that my changes actually
      work as intended.
      
      Only people who have bothered to read this far will know that I also added
      the ability to cancel experiment swapin in progress. For that I am using
      the "canceled" flag (ah, this one was named properly from the start!), and
      I test that at various times in assign_wrapper and tbswap. A minor downside
      right now is that a canceled swapin looks too much like a failed swapin,
      and so tbops gets email about it. I'll fix that at some point (sometime
      after the boss complains).
      
      I also cleaned up various bits of code, replacing direct calls to exec
      with calls to the recently improved SUEXEC interface. This removes
      some cruft from each script that calls an external script.
      
      Cleaned up modifyexp.ph3 quite a bit, reformatting and indenting.
      Also fixed to not run the parser directly! This was very wrong; should
      call nscheck instead. Changed to use "nobody" group instead of group
      flux (made the same change in nscheck).
      
      There is a script in the sql directory called newstates.pl. It needs
      to be run to initialize the batchstate slot of the experiments table
      for all existing experiments.
      4269dad1
  18. 30 Jul, 2003 1 commit
  19. 29 Jul, 2003 1 commit
    • Leigh Stoller's avatar
      Some cleanup on the batch mode stuff. Make it more explicit in the · 29b820b1
      Leigh Stoller authored
      showexp page that its a batch experiment, by the menu options. Same
      deal in the swapexp output, plus some other minor cleanup. The only
      bug I found while trying to figure out the batchmode problem reported
      this morning by the FileMover people, is that the cancelflag is not
      cleared after swaping a running batch experiment out, so even after
      reinjecting it into the queue, it will not run. Still, that does seem
      to be what the FileMover people reported.
      29b820b1
  20. 17 Jul, 2003 1 commit
  21. 28 May, 2003 1 commit
  22. 24 May, 2003 1 commit
    • Mac Newbold's avatar
      Round of changes related to idleswapping and autoswapping. The web and · 02aaf8e4
      Mac Newbold authored
      back end scripts now support 3 different kind of forced swaps:
      
      1. Idle-Swap : this is ths same one we had before. Email message to them
      says it was swapped "because it was idle for too long"
      
      2. Auto-Swap : A new one, typically for user-requested timed swapouts.
      Email says it was swapped "because it was swapped in too long"
      
      3. Force swap: Generic one, for "none of the above" cases. Just says
      Experiment "has been forcibly swapped out by Testbed Operations."
      
      The force swap option on the web now lets you choose which of these three
      you want. Only "Idle-Swap" counts as an idleswap in the stats. Soon
      idleswap and autoswap will be used by idlemail when it does automatic
      swapping.
      02aaf8e4
  23. 22 May, 2003 2 commits
    • Leigh Stoller's avatar
      Reorg the batch system slightly as per Eric's request that batch mode · da97ba35
      Leigh Stoller authored
      experiments look more like regular experiments. Batch mode experiments
      can now be preloaded and swapped. When preloaded, they go into a
      "Pause" state. Swapping a batch mode experiment in puts them into the
      "posted" state so the batch daemon will see them. Swapping out a
      batchmode experiment does the expected; it puts them back into the
      Pause state. Terminating a batch mode experiment does the expected;
      its gone. When a batch mode experiment finishes normally, it goes back
      into the pause state, which allows batches to be reinjected as many
      times as Eric likes.
      da97ba35
    • Mac Newbold's avatar
      Show a big warning if someone tries to forceswap an unswappable expt. · f527cd91
      Mac Newbold authored
      Also do some prep work to carry $idleswap and $autoswap through this
      script like $force is. (For calling the back end with different params,
      when it supports it.)
      f527cd91
  24. 16 Apr, 2003 1 commit
    • Leigh Stoller's avatar
      Add support for idleswapping an experiment as the creator of the · ff5a57de
      Leigh Stoller authored
      experiment, rather than as an administrator, which presents group
      permission problems when the experiment is in a subgroup (requires two
      additional group, whereas suexec adds only one group). That aside, the
      correct approach is to run the swap as the creator. To do that, must
      flip to the user (from the admin person) in the backend using the new
      idleswap script, and then run the normal swapexp. Add new option to
      swapexp (-i) which changes the email slightly to make it clear that
      the experiment was idleswapped, and so that the From: is tbops not the
      user (again, to make it more clear).
      ff5a57de
  25. 18 Dec, 2002 1 commit
    • Leigh Stoller's avatar
      New "restart" or perhaps better if named "replay" mode to swapexp. · d651dd42
      Leigh Stoller authored
      Attempts to replay an experiment by rebooting all the nodes, clearing
      the various startup bits (ready, startstatus, bootstatus, portstats),
      and then restarting the event system. I am dubious that this is a
      workable solution because of the asynchronous nature of the testbed
      (nodes happily cruise from TBRESET to ISUP and beyond without
      stopping), and so its hard to truly replicate the initial lack of
      state that a freshly swapped in experiment has. Still, people
      requested it and I cheerfully provided it cause thats what I do;
      service with a smile and not a wit of complaint. Is anyone reading
      this?
      d651dd42
  26. 16 Sep, 2002 1 commit
  27. 25 Jul, 2002 1 commit
  28. 07 Jul, 2002 1 commit
  29. 16 Jun, 2002 1 commit
    • Leigh Stoller's avatar
      Some fixes to the spewlogfile stuff so that you do not get the · d9c3dd68
      Leigh Stoller authored
      transition error when you click too fast after creating it. Instead of
      looking at experiment state, use the logile slot of the experiments
      table, and make sure its cleared/set properly in start/swap experiment
      scripts.
      
      Also added a spew option to the swap page so you can watch experiments
      swap in/out.
      d9c3dd68
  30. 19 Apr, 2002 1 commit
  31. 14 Jan, 2002 1 commit
  32. 08 Nov, 2001 1 commit
  33. 06 Nov, 2001 1 commit
  34. 24 Oct, 2001 1 commit
    • Leigh Stoller's avatar
      Add swappable and priority bits to experiment creation form. Not used, · 28c1968f
      Leigh Stoller authored
      but simply entered into the DB record for the experiment until we know
      what to do with them. Add to batchexp script arguments, since all that
      stuff is done outside the web interface. Add a swapexp perl script to
      swap an an experiment in/out form the command line. Add web links on
      the Experiment Information page to do this from the web interface. A
      bunch of locking changes. Previously expt_terminating in the
      experiment record prevented multiple calls to terminate an experiment,
      but now we have a more general locking problem with
      start,swapin,swapout, and terminate, so change expt_terminating to
      expt_locked (still a datetime field) and add locking to all of
      startexp, swapexp, and endexp. Note that batch experiments cannot be
      swapped yet because of locking issues still to be resolved. Minor
      cleanup in tbreport to make email message look better.
      28c1968f
  35. 17 Oct, 2001 1 commit
    • Leigh Stoller's avatar
      Rework of the batch experiment code. Unified it with the immediate · 4d420b21
      Leigh Stoller authored
      experiment code. No longer uses another table. Rather, the experiment
      record contains a couple of extra fields for the batch system. Also
      combined some of the backend code (no longer a killbatch script).
      Also added scriptable experiments; the batchexp program in the bin
      directory can start an experiment from the command line, and in fact
      is used from the web page for both batch experiments and immediate
      experiments (-i option). All of the DB code that was in the web
      interfaces was moved to batchexp.
      4d420b21
  36. 16 Oct, 2001 1 commit
  37. 20 Jun, 2001 1 commit
  38. 04 May, 2001 1 commit
  39. 18 Apr, 2001 1 commit