1. 10 Jan, 2007 1 commit
  2. 09 Jan, 2007 1 commit
  3. 19 Dec, 2006 1 commit
  4. 18 Dec, 2006 1 commit
  5. 14 Dec, 2006 1 commit
  6. 05 Dec, 2006 2 commits
  7. 01 Dec, 2006 2 commits
  8. 21 Nov, 2006 2 commits
  9. 20 Nov, 2006 1 commit
  10. 09 Nov, 2006 1 commit
  11. 03 Nov, 2006 1 commit
    • Leigh Stoller's avatar
      Big set of changes intended to solve a couple of problems with long · ff9061d4
      Leigh Stoller authored
      term archiving of firstclass objects like users, projects, and of
      course templates.
      
      * Projects, Users, and Groups are now uniquely identified inside the
        DB by a index value that will not be reused. If necessary, this
        could easily be a globally unique identifier, but without federation
        there is no reason to do that yet.
      
      * Currently, pid, gid, and uid still need to be locally unique until
        all of the changes are in place (which is going to take a fairly
        long time since the entire system operates in terms of those, except
        for the few places that I had to change to get the ball rolling).
      
      * We currently archive deleted users to the deleted_users table (their
        user_stats are kept forever since they are indexed by the new index
        column). Eventually do the same with projects (not sure about
        groups) but since we rarely if ever delete a project, there is no
        rush on this one.
      
      * At the same time, I have started a large reorg of the code, to move
        all of the user, group, project code into modules, both in php and
        perl, turning them into first class "objects" (as far as that goes
        in php and perl). Eventually, the number of query statements
        scattered around the code will be manageable, or so I hope.
      
      * Another related part of this reorg is to make it easier to move the
        new user/project/group code in the perl backend so that it can be
        made available via the xmlrpc interface (without duplication of the
        code).
      ff9061d4
  12. 27 Oct, 2006 1 commit
  13. 25 Oct, 2006 1 commit
    • Leigh Stoller's avatar
      Makefile Whacking! Try to deal with the problem caused by the delay · 7590f9c5
      Leigh Stoller authored
      between when something is installed and when post-install runs. Short
      of a global lock (which we probably need anyway someday), my solution
      is this. In your makefiles, add these variables before the line that
      has the include of $(TESTBED_SRCDIR)/GNUmakerules:
      
      	SETUID_BIN_SCRIPTS   =
      	SETUID_SBIN_SCRIPTS  =
      
      I have added three new rules to GNUmakerules that look like this:
      
      	$(addprefix $(SBINDIR)/, $(SETUID_SBIN_SCRIPTS)): $(SBINDIR)/%: %
      		echo "Installing (setuid) $<"
      		-mkdir -p $(INSTALL_SBINDIR)
      		$(SUDO) $(INSTALL) -o root -m 4755 $< $@
      
      Yep, your eyes ain't lying to you; use sudo to run the target so that
      install does the right thing (which is that the old file is not
      replaced until the new one has the proper attributes on it).
      
      Note that post-install is still needed for the initial install, but
      should no longer be needed for day to day installs since all that other
      stuff post-install does is mkdir/chmod on directories.
      7590f9c5
  14. 24 Oct, 2006 3 commits
  15. 20 Oct, 2006 2 commits
    • Leigh Stoller's avatar
      Add compression option to sync option of loghole. When turned on, any file · 4d4a27e1
      Leigh Stoller authored
      greater the 512K is automatically compressed with gzip. Might need to
      make this number bigger; we shall see.
      
      If you run emacs, put this in your .emacs file.
      
      	(load "jka-compr")
      	(jka-compr-install)
      
      and any time you visit a file that ends in one of the standard compression
      extensions, emacs will automatically do the uncompress for you on the data
      in the buffer (not the actual disk file of course). Very convenient.
      
      You can also get your browser to do the same, but I leave that as an
      exercise for the reader.
      4d4a27e1
    • Mike Hibler's avatar
      Wow, this should make me look important! · afa5e919
      Mike Hibler authored
      Two-day boondoggle to support "/scratch", an optional large, shared filesystem
      for users.  To do this, I needed to find all the instances where /proj is used
      and behave accordingly.  The boondoggle part was the decision to gather up all
      the hardwired instances of shared directory names ("/proj", "/users", etc.)
      so that they are set in a common place (via unexposed configure variables).
      This is a boondoggle because:
      
      1. I didn't change the client-side scripts.  They need a different mechanism
         (e.g., tmcd) to get the info, configure is the wrong way.
      
      2. Even if I had done #1 it is likely--no, certain--that something would
         fail if you tried to rename "/proj" to be "/mike".  These names are just
         too ingrained.
      
      3. We may not even use "/scratch" as it turns out.
      
      Note, I also didn't fix any of the .html documentation.  Anyway, it is done.
      To maintain my illusion in the future you should:
      
      1. Have perl scripts include "use libtestbed" and use the defined PROJROOT(),
         et.al. functions where possible.  If not possible, make sure they run
         through configure and use @PROJROOT_DIR@, etc.
      
      2. Use the configure method for python, C, php and other languages.
      
      3. There are perl (TBValidUserDir) and php (VALIDUSERPATH) functions which
         you should call to determine if an NS, template parameter, tarball or
         other file are in "an acceptable location."  Use these functions where
         possible.  They know about the optional "scratch" filesystem.  Note that
         the perl function is over-engineered to handles cases that don't occur
         in nature.
      afa5e919
  16. 18 Oct, 2006 1 commit
  17. 05 Oct, 2006 1 commit
    • Leigh Stoller's avatar
      More work on "recording" template events. · e9607a77
      Leigh Stoller authored
      * New version of template_record just for ops, since so much is
        different about ops, not bothering to maintain a single version.
      
      * Various fixes to how the recorded events are stored and reconstituted.
        The big fix is to wrap them in a sequence to that they get fired
        properly (waiting for completion of previous event in recording).
      
      * New buttons to Pause and Continue event time, which is used when
        adding recorded events. This allows users to pause time while they
        "think" so when an event is recorded, the thinking time is not actually
        in the timeline. Eventually hope to figure this out automatically, but
        that will take some real, uh, thinking.
      
      * Add a new event editor (linked off the template page) that allows
        you to delete and change the recordings. Note that you can only edit
        the events at the template level; you cannot edit the events of an
        instance (swapped in experiment), and you can only edit the recorded
        events, not any other events. Not sure its useful to be able to do
        either of these yet, but probably not too hard to add at some point.
      e9607a77
  18. 03 Oct, 2006 1 commit
  19. 02 Oct, 2006 1 commit
  20. 29 Sep, 2006 1 commit
    • Leigh Stoller's avatar
      Minor changes to per-experiment DB stuff. · dc8e62a3
      Leigh Stoller authored
      * Create a per-experiment DB user for the per-experiment DB; the user
        name is equal to the DB name.
      
      * Add a dpdbpassword field to the experiments table; this is the
        randomly generated password for the DB user mentioned above.
      
      * For Templates, use the above user/password in the environment,
        instead of the swapper uid/password.
      
      * Add experiment dbname/dbpassword to the Show Experiment page.
      dc8e62a3
  21. 19 Sep, 2006 3 commits
  22. 14 Sep, 2006 1 commit
  23. 13 Sep, 2006 2 commits
  24. 12 Sep, 2006 1 commit
  25. 05 Sep, 2006 2 commits
    • Leigh Stoller's avatar
      A bunch of template changes resulting from meetings last week. · 087dbfff
      Leigh Stoller authored
      * Add XMLRPC interface for template swapin,stoprun,startrun,swapout and
        add the appropriate wrappers to the script_wrapper on ops.
      
      * Allow parameter descriptions in NS files. This is probably not in its
        final form since its a bit confusing as to what has priority; something
        in the NS file or a metadata item. Anyway, you can do this in your NS
        file:
      
      	$ns define-template-parameter GUID "0/0" "The GUID to be analyzed"
      
        The rules are currently that the NS file description has priority and
        is copied to child templates, unless the user has modified a description
        via the web interface, in which case the NS file description is ignored.
        I know, sounds awful, but for the most part people are going to use the
        NS file anyway.
      
      * Add "clear" option when starting a new experiment run; the per
        experiment DB at the logholes are cleared. Note that this is *not* the
        default behaviour; you have to either check the checkbox on the web form
        or use the -c option to the script wrapper, or clear=yes if talking
        directly to the XMLRPC server.
      
      * Fix up how email is generated for template_swapin and template_create,
        so that Kevin can debug tblog/tbreport stuff, but also so that we maintain
        mail logs as before. I have made some improvements to libaudit so as to
        centralize the mail goo, and avoid duplicating all that stuff.
      
      * Minor fixes to the program agent so that the new environment strings are
        sent before the program agent exits and reloads them!
      
      * Other minor little things.
      087dbfff
    • Leigh Stoller's avatar
      Add --delete option to sync directive, to schedule a clean for the · ef320fa8
      Leigh Stoller authored
      next sync (similar to how it works with the archive directive).
      ef320fa8
  26. 14 Aug, 2006 1 commit
    • Leigh Stoller's avatar
      Checkpoint my dynamic event stuff, crude as it is. The idea for this first · 9d021a07
      Leigh Stoller authored
      draft is that the user will at the end of an experiment run, log into one
      of his nodes and perform some analysis which is intended to be repeated at
      the end of the next run, and in future instantiations of the template.
      
      A new table called experiment_template_events holds the dynamic events for
      the template. Right now I am supporting just program events, but it will be
      easy to support arbitrary events later. As an absurd example:
      
      	node6> /usr/local/bin/template_analyze ~/data_analyze arg arg ...
      
      The user is currently responsible for making sure the output goes into a
      file in the archive. I plan to make the template_analyze wrapper handle
      that automatically later, but for now what you really want is to invoke a
      script that encapsulates that, redirecting output to $ARCHIVE (this
      variable is installed in the environment template_analyze.
      
      The wrapper script will save the current time, and then run the program.
      If the program terminates with a zero exit status, it will ssh over to ops
      and invoke an xmlrpc routine to tell boss to add a program event to both
      the eventlist for the current instance, and to the template_eventlist for
      future instances. The time of the event is the relative start time that was
      saved above (remember, each experiment run replays the event stream from
      time zero).
      
      For the future, we want to allow this to be done on ops as well, but
      that will take more infrastructure, to run "program agents" on ops.
      
      It would be nice to install the ssl xmlrpc client side on our images so
      that we do not have to ssh to ops to invoke the client.
      9d021a07
  27. 10 Aug, 2006 2 commits
  28. 09 Aug, 2006 1 commit
  29. 08 Aug, 2006 1 commit