1. 09 Jun, 2010 1 commit
  2. 04 May, 2010 1 commit
  3. 15 Apr, 2010 1 commit
  4. 10 Nov, 2009 1 commit
  5. 22 Sep, 2009 1 commit
  6. 02 Mar, 2009 2 commits
    • Leigh B. Stoller's avatar
      A bunch of changes for a "standalone" clearinghouse. Presently this · 60f04310
      Leigh B. Stoller authored
      its really a hugely stripped down Emulab boss install, using a very
      short version of install/boss-install to get a few things into place.
      
      I refactored a few things in both the protogeni code and the Emulab
      code, and whacked a bunch of makefiles and configure stuff. The result
      is that we only need to install about 10-12 files from the Emulab
      code, plus the protogeni code. Quite manageable, if you don't mind
      that it requires FreeBSD 6.X ... Still, I think it satisfies the
      requirement that we have a packaged clearinghouse that can be run
      standalone from a running Emulab site.
      60f04310
    • Leigh B. Stoller's avatar
  7. 06 Feb, 2009 1 commit
    • Leigh B. Stoller's avatar
      Start at an attempt to bring some sanity to the virt topology tables. · 68b193a5
      Leigh B. Stoller authored
      So far this wraps the virtual tables in objects, and I have changed
      xmlconvert already, but that needs more testing. Eventually I want to
      provide some simple generators like addnode and addlink so that I can
      use form the Protogeni code to build up a virtual topology from an
      rspec. Not sure how that will go.
      68b193a5
  8. 23 Jan, 2009 1 commit
  9. 21 Jan, 2009 1 commit
  10. 12 Jan, 2009 1 commit
  11. 07 May, 2008 1 commit
  12. 24 Apr, 2008 2 commits
  13. 11 Feb, 2008 1 commit
    • Leigh B. Stoller's avatar
      Initial checkpoint of a script that will check the stats tables · 8ab3e170
      Leigh B. Stoller authored
      nightly, and make repairs for the common things I see happening.  The
      current version repairs most of the errors that have crept in since
      the Epoch, and then rebuilds the summary stats table (user_stats,
      group_stats, project_stats). My intent is to add a "daily_stats" table
      as well since scanning the testbed_stats and experiment_resources
      table to generate range data is getting pretty slow as more records
      enter the system. We can also use a daily_stats table to generate
      graphs on the fly as Jay requested a few weeks ago.
      
      Not done yet, hope to return to it later this week.
      8ab3e170
  14. 14 Jan, 2008 1 commit
  15. 06 Nov, 2007 1 commit
    • Leigh B. Stoller's avatar
      This started out as a simple change to turn the datastore into a CVS · c1cff09b
      Leigh B. Stoller authored
      sandbox, and that I did. It falls back to the older archive when
      the template is older then CVS repos.
      
      But along the way I got annoyed with the fact that template instantiation
      does not provide a logfile to the web interface. The reason is that
      the current logfile stuff is very experiment centric; there has to be an
      experiment and an attached logfile. An instance does not have an experiment
      until really late in the game so the code was just not bothering.
      
      Anyway, I've started to generalize the logfile stuff with a new table
      and the approach that a logfile is named by a random key, and if you
      know the key you can look at the logfile in the web (since without an
      experiment it is hard to do permission checks unless we make logfiles
      uid/gid owned, and I did not want to do that.
      c1cff09b
  16. 11 Jul, 2007 1 commit
  17. 30 May, 2007 1 commit
  18. 07 May, 2007 1 commit
    • Leigh B. Stoller's avatar
      Mostly this commit is the switch from SVN archives to ZIP archives. · 55d1bb6e
      Leigh B. Stoller authored
      Other stuff leaked in too ...
      
      I did separate out a lot of tbsetup/libArchive into db/Archive, and
      whats left in libArchive.pm will eventually move over into the
      Template library.
      
      Note that I have dropped archiving of plain experiments; this is not
      really owrth it outside the workbench context, and it just wastes
      space and makes a lot if stuff painful in the web interface.
      55d1bb6e
  19. 13 Mar, 2007 1 commit
  20. 16 Jan, 2007 2 commits
  21. 27 Nov, 2006 1 commit
    • Leigh B. Stoller's avatar
      Call this commit "Snow in Corvallis" ... · 4998b2d7
      Leigh B. Stoller authored
      The major functional change in this revision is converting from user
      selected UIDs to system selected UIDs. This is controlled by the
      variable $USERSELECTUIDS in defs/defs.php3.in which is now set to
      zero, so system selected UIDs is the default.
      
      The algo for creating the uid is to take the email address, strip the
      @whatever from it, squeeze out dots and dashes and underlines, and
      make sure any +foo tokens are removed. Then make sure it is unique by
      taking the first 5 characters and then adding a 3 digit number,
      derived by checking the DB to see what exists.
      
      Since we will want to (more often) change the UID selected, there is a
      new admin only menu option on the Show User page. It calls the backend
      script to do the work (sbin/changeuid).
      
      The login page now defaults to storing and showing the email address
      for login, rather then the UID. It will still accept either one though
      (has for a long time).
      
      Along the way I also reorg'ed a number of pages to use the new user,
      group, and project classes and moved some common functionality into
      the class defs.
      
      Also changed the way addpubkey is called, to avoid some confusion.
      4998b2d7
  22. 03 Nov, 2006 1 commit
    • Leigh B. Stoller's avatar
      Big set of changes intended to solve a couple of problems with long · ff9061d4
      Leigh B. Stoller authored
      term archiving of firstclass objects like users, projects, and of
      course templates.
      
      * Projects, Users, and Groups are now uniquely identified inside the
        DB by a index value that will not be reused. If necessary, this
        could easily be a globally unique identifier, but without federation
        there is no reason to do that yet.
      
      * Currently, pid, gid, and uid still need to be locally unique until
        all of the changes are in place (which is going to take a fairly
        long time since the entire system operates in terms of those, except
        for the few places that I had to change to get the ball rolling).
      
      * We currently archive deleted users to the deleted_users table (their
        user_stats are kept forever since they are indexed by the new index
        column). Eventually do the same with projects (not sure about
        groups) but since we rarely if ever delete a project, there is no
        rush on this one.
      
      * At the same time, I have started a large reorg of the code, to move
        all of the user, group, project code into modules, both in php and
        perl, turning them into first class "objects" (as far as that goes
        in php and perl). Eventually, the number of query statements
        scattered around the code will be manageable, or so I hope.
      
      * Another related part of this reorg is to make it easier to move the
        new user/project/group code in the perl backend so that it can be
        made available via the xmlrpc interface (without duplication of the
        code).
      ff9061d4
  23. 25 Oct, 2006 1 commit
    • Leigh B. Stoller's avatar
      Makefile Whacking! Try to deal with the problem caused by the delay · 7590f9c5
      Leigh B. Stoller authored
      between when something is installed and when post-install runs. Short
      of a global lock (which we probably need anyway someday), my solution
      is this. In your makefiles, add these variables before the line that
      has the include of $(TESTBED_SRCDIR)/GNUmakerules:
      
      	SETUID_BIN_SCRIPTS   =
      	SETUID_SBIN_SCRIPTS  =
      
      I have added three new rules to GNUmakerules that look like this:
      
      	$(addprefix $(SBINDIR)/, $(SETUID_SBIN_SCRIPTS)): $(SBINDIR)/%: %
      		echo "Installing (setuid) $<"
      		-mkdir -p $(INSTALL_SBINDIR)
      		$(SUDO) $(INSTALL) -o root -m 4755 $< $@
      
      Yep, your eyes ain't lying to you; use sudo to run the target so that
      install does the right thing (which is that the old file is not
      replaced until the new one has the proper attributes on it).
      
      Note that post-install is still needed for the initial install, but
      should no longer be needed for day to day installs since all that other
      stuff post-install does is mkdir/chmod on directories.
      7590f9c5
  24. 18 Jul, 2006 1 commit
    • Leigh B. Stoller's avatar
      Changes necessary for moving most of the stuff in the node_types · 624a0364
      Leigh B. Stoller authored
      table, into a new table called node_type_attributes, which is intended
      to be a more extensible way of describing nodes.
      
      The only things left in the node_types table will be type,class and the
      various isXXX boolean flags, since we use those in numerous joins all over
      the system (ie: when discriminating amongst nodes).
      
      For the most part, all of that other stuff is rarely used, or used in
      contexts where the information is needed, but not for type descrimination.
      Still, it made for a lot of queries to change!
      
      Along the way I added a NodeType library module that represents the type
      info as a perl object. I also beefed up the existing Node module, and
      started using it in more places. I also added an Interfaces module, but I
      have not done much with that yet.
      
      I have not yet removed all the slots from the node_types table; I plan to
      run the new code for a few days and then remove the slots.
      
      Example using the new NodeType object:
      
      	use NodeType;
      
      	my $typeinfo = NodeType->Lookup($type);
      
              if ($typeinfo->control_interface(\$control_iface) ||
                  !$control_iface) {
        	    warn "No control interface for $type is defined in the DB!\n";
              }
      
      or using the Node:
      
      	use Node;
      
              my $nodeobject = Node->Lookup($node_id);
              my $imageable  = $nodeobject->NodeTypeInfo()->imageable();
      or
              my $rebootable = $nodeobject->isrebootable();
      or
              $nodeobject->NodeTypeAttribute("control_interface", \$control_iface);
      
      Lots of way to accomplish the same thing, but the main point is that the
      Node is able to override the NodeType (if it wants to), which I think is
      necessary for flexibly describing one/two of a kind things like switches, etc.
      624a0364
  25. 03 Jul, 2006 1 commit
  26. 21 Jun, 2006 1 commit
    • Leigh B. Stoller's avatar
      Munge the schemacheck code to deal with all the oddities of the way · 53c29cfa
      Leigh B. Stoller authored
      mysql 5.0 dumps the schema. What a pain in the ass.
      
      Note that "timestamp" is basically impossible since its radically
      different between 3.X and 5.X, which would break schemacheck on 3.X
      based Emulabs. Since there are only three of them in the schema, I
      changed schemadiff to not look too hard at them.
      53c29cfa
  27. 23 Jan, 2006 1 commit
  28. 01 Dec, 2005 1 commit
    • Leigh B. Stoller's avatar
      Set up aging of the log table. However, this script also does some · fff1d16b
      Leigh B. Stoller authored
      special stuff at Utah (MAINSITE=1).
      
      * Before aging out, copy the entries to another DB on boss so that we
        have it forever!
      
      * As an extra special hack of the century, copy those entries over to
        ops and store in a DB over there for Kevin to look at. This code
        will go away at some point.
      fff1d16b
  29. 20 Jul, 2005 1 commit
  30. 26 Apr, 2005 1 commit
    • Leigh B. Stoller's avatar
      A watchdog daemon to try and catch (and recover from) the periodic · c47cefa1
      Leigh B. Stoller authored
      mysqld hangs that cause the entire system to grind to a halt. The
      basic theory of operation is like this:
      
      * Once a minute fork a child (protected by a 60 second timeout) to
        connect to the DB and issue a simple query. If the child can access
        the DB okay, it exits with a zero status.
      
      * If the alarm fires, the child is killed. This indicates that mysqld
        is no longer responding in a reasonable amount of time (60 seconds).
        We shift into trying to restart mysqld:
      
           * Send mysqld a TERM. Wait for 30 seconds.
      
           * Try query again; typically, the situation will not have changed one
             bit, but I do it anyway.
      
           * If mysqld was running, send it a kill -9. Wait for 15 seconds.
      
           * Start mysqld. Wait for 5 seconds.
      
           * Try query again. If query succeeds, we are done, and no one
             will have to deal with it Sunday morning at 6am (thanks Tim).
      
           * If query still fails, send email and give up trying to do fix
             anything. The daemon continues to query the DB once a minute;
             once the query succeeds (cause a human fixed things up), the
             daemon goes back into its normal mode (attempt to fix things
             next time it fails).
      
      So, the problem is what happens when someone kills off mysqld for some
      other reason. It may be that this daemon should only try to restart
      mysqld if and only if, it actually killed a running mysqld. Comments?
      c47cefa1
  31. 24 Jan, 2005 1 commit
    • Leigh B. Stoller's avatar
      Bottom line on this commit: Do not update the nodetypeXpid_permissions · 775ca147
      Leigh B. Stoller authored
      table by hand anymore! Update the group_policies table and then run
      the script to update the permissions table (sbin/update_permissions).
      
      Details:
      
      My original thought when I started this was that I would be able to
      replace the existing nodetypeXpid_permissions table with this new
      stuff. Well, it turns out that this was not a good thing to do, for a
      couple of reasons:
      
        * Engineering: We access the nodetypeXpid_permissions table from three
          different languages, and no way I wanted to rewrite this library in
          in python and php!
      
        * Performance: We access the nodetypeXpid_permissions from the web
          interface, on every single page load. In fact, we access it twice if
          if you count the FreePCs() count that we put at the top of the menu.
          Going through this library on each page load would be a serious drag.
      
      So, rather then actually get rid of the nodetypeXpid_permissions table, I
      decided to keep it as a "cache" of permissions stored in the group
      policies table. Each time you update the policy tables, we need to run
      the update_permissions script which will call into this library (see the
      TBUpdateNodeTypeXpidPermissions() routine) to reconstruct the permissions
      table. I have whacked the grantnodetype script to do exactly that.
      
      Note that we could proably do the same thing for users by creating an
      equivalent nodetypeXuid_permissions table, mapping users to types they
      are allowed to use. That would be a lot rows, but the amount of data in
      the table is small. That would give us very fine grained control of what
      we show people in the web interface. Not sure it is worth it though.
      
      I also added some instructions to previous commit in database-migrate.txt
      on populating the new group_policies table from the existing
      permissions table.
      775ca147
  32. 18 Jan, 2005 1 commit
    • Leigh B. Stoller's avatar
      Here is a checkpoint of the admission control stuff I have been working on. · 54f55585
      Leigh B. Stoller authored
      The last part is the stuff to hook it in from assign_wrapper, and some
      additional support in assign that Rob is adding for me. This comment is
      from the top of new file db/libadminctrl.pm.in and describes everything in
      detail.
      
      # Admission control policies. These are the ones I could think of, although
      # not all of these are implemented.
      #
      #  * Number of experiments per type/class (only one expt using robots).
      #
      #  * Number of experiments per project
      #  * Number of experiments per subgroup
      #  * Number of experiments per user
      #
      #  * Number of nodes per project      (nodes really means pc testnodes)
      #  * Number of nodes per subgroup
      #  * Number of nodes per user
      #
      #  * Number of nodes of a class per project
      #  * Number of nodes of a class per group
      #  * Number of nodes of a class per user
      #
      #  * Number of nodes of a type per project
      #  * Number of nodes of a type per group
      #  * Number of nodes of a type per user
      #
      #  * Number of nodes with attribute(s) per project
      #  * Number of nodes with attribute(s) per group
      #  * Number of nodes with attribute(s) per user
      #
      # So we have group (pid/gid) policies and user policies. These are stored
      # into two different tables, group_policies and user_policies, indexed in
      # the obvious manner. Each row of the table defines a count (experiments,
      # nodes, etc) and a type of thing being counted (experiments, nodes, types,
      # classes, etc). When we test for admission, we look for each matching row
      # and test each condition. All conditions must pass. No conditions means a
      # pass. There is also some "auxdata" which holds extra information needed
      # for the policy (say, the type of node being restricted).
      #
      #      uid:     a uid
      #   policy:     'experiments', 'nodes', 'type', 'class', 'attribute'
      #    count:     a number
      #  auxdata:     a string (optional)
      #
      # Example: A user policy of ('mike', 'nodes', 10) says that poor mike is
      # not allowed to have more 10 nodes at a time, while ('mike', 'type',
      # '10', 'pc850') says that mike cannot allocate more than 10 pc850s.
      #
      # The group_policies table:
      #
      #      pid:     a pid
      #      gid:     a gid
      #   policy:     'experiments', 'nodes', 'type', 'class', 'attribute'
      #    count:     a number
      #  auxdata:     a string (optional)
      #
      # Example: A project policy of ('testbed', 'testbed', 'experiments', 10)
      # says that the testbed project may not have more then 10 experiments
      # swapped in at a time, while ('testbed', 'TG1', 'nodes', 10) says that the
      # TG1 subgroup of the testbed project may not use more than 10 nodes at
      # time.
      #
      # In addition to group and user policies (which are policies that apply to
      # specific users/projects/subgroups), we also need policies that apply to
      # all users/projects/subgroups (ie: do not want to specify a particular
      # restriction for every user!). To indicate such a policy, we use a special
      # tag in the tables (for the user or pid/gid):
      #
      #      '+'  -  The policy applies to all users (or project/groups).
      #
      # Example: ('+','experiments',10) says that no user may have more then 10
      # experiments swapped in at a time. The rule overrides anything more
      # specific (say a particular user is restricted to 20 experiments; the above
      # rule overrides that and the user (all users) is restricted to 10.
      #
      # Sometimes, you want one of these special rules to apply to everyone, but
      # *allow* it to be overridden by a more specific rule. For that we use:
      #
      #      '-'  -  The policy applies to all users (or project/groups),
      #              but can be overridden by a more specific rule.
      #
      # Example: The rules:
      #
      #	('-','type',0, 'garcia')
      #       ('testbed', 'testbed', 'type', 10, 'garcia')
      #
      # says that no one is allowed to allocate garcias, unless there is specific
      # rule that allows it; in this case the testbed project can allocate them.
      #
      # There are other global policies we would like to enforce. For example,
      # "only one experiment can be using the robot testbed." Encoding this kind
      # of policy is harder, and leads down a path that can get arbitrarily
      # complex. Tha path leads to ruination, and so we want to avoid it at
      # all costs.
      #
      # Instead we define a simple global policies table that applies to all
      # experiments currently active on the testbed:
      #
      #   policy:     'nodes', 'type', 'class', 'attribute'
      #     test:     'max', others I cannot think of right now ...
      #    count:     a number
      #  auxdata:     a string
      #
      # Example: A global policy of ('nodes', 'max', 10, '') say that the maximum
      # number of nodes that may be allocated across the testbed is 10. Thats not
      # a very realistic policy of course, but ('type', 'max', 1, 'garcia') says
      # that a max of one garcia can be allocated across the testbed, which
      # effectively means only one experiment will be able to use them at once.
      # This is of course very weak, but I want to step back and give it some
      # more thought before I redo this part.
      #
      # Is that clear? Hope so, cause it gets more complicated. Some admission
      # control tests can be done early in the swap phase, before we really do
      # anything (before assign_wrapper). Others (type and class) tests cannot
      # be done here; only assign can figure out how an experiment is going to map
      # to physical nodes (remember virtual types too), and in that case we need
      # to tell assign what the "constraints" are and let it figure out what is
      # possible.
      #
      # So, in addition to the simple checks we can do, we also generate an array
      # to return to assign_wrapper with the maximum counts of each node type and
      # class that is limited by the policies. assign_wrapper will dump those
      # values into the ptop file so that assign can enforce those maximum values
      # regardless of what hardware is actually available to use. As per discussion
      # with Rob, that will look like:
      #
      #	set-type-limit <type> <limit>
      #
      # and assign will spit out a new type of violation that assign_wrapper will
      # parse.
      #
      # NOTES:
      #
      #  1) Admission control is skipped in admin mode; returns okay.
      #  2) Admission control is skipped when the pid is emulab-ops; returns okay.
      #  3) When calculating current usage, nodes reserved to emulab-ops are
      #     ignored.
      #  4) The sitevar "swap/use_admission_control" controls the use of admission
      #     control; defaults to 1 (on).
      #  5) The current policies can be viewed in the web interface. See
      #     https://www.emulab.net/showpolicies.php3
      #  6) The global policy stuff is weak. I plan to step back and think about it
      #     some more before redoing it, but it will tide us over for now.
      #
      54f55585
  33. 03 Jan, 2005 1 commit
  34. 15 Nov, 2004 1 commit
  35. 14 Nov, 2004 1 commit
  36. 28 Oct, 2004 1 commit
  37. 29 Sep, 2004 1 commit