1. 27 Nov, 2006 1 commit
    • Leigh B. Stoller's avatar
      Call this commit "Snow in Corvallis" ... · 4998b2d7
      Leigh B. Stoller authored
      The major functional change in this revision is converting from user
      selected UIDs to system selected UIDs. This is controlled by the
      variable $USERSELECTUIDS in defs/defs.php3.in which is now set to
      zero, so system selected UIDs is the default.
      
      The algo for creating the uid is to take the email address, strip the
      @whatever from it, squeeze out dots and dashes and underlines, and
      make sure any +foo tokens are removed. Then make sure it is unique by
      taking the first 5 characters and then adding a 3 digit number,
      derived by checking the DB to see what exists.
      
      Since we will want to (more often) change the UID selected, there is a
      new admin only menu option on the Show User page. It calls the backend
      script to do the work (sbin/changeuid).
      
      The login page now defaults to storing and showing the email address
      for login, rather then the UID. It will still accept either one though
      (has for a long time).
      
      Along the way I also reorg'ed a number of pages to use the new user,
      group, and project classes and moved some common functionality into
      the class defs.
      
      Also changed the way addpubkey is called, to avoid some confusion.
      4998b2d7
  2. 03 Nov, 2006 1 commit
    • Leigh B. Stoller's avatar
      Big set of changes intended to solve a couple of problems with long · ff9061d4
      Leigh B. Stoller authored
      term archiving of firstclass objects like users, projects, and of
      course templates.
      
      * Projects, Users, and Groups are now uniquely identified inside the
        DB by a index value that will not be reused. If necessary, this
        could easily be a globally unique identifier, but without federation
        there is no reason to do that yet.
      
      * Currently, pid, gid, and uid still need to be locally unique until
        all of the changes are in place (which is going to take a fairly
        long time since the entire system operates in terms of those, except
        for the few places that I had to change to get the ball rolling).
      
      * We currently archive deleted users to the deleted_users table (their
        user_stats are kept forever since they are indexed by the new index
        column). Eventually do the same with projects (not sure about
        groups) but since we rarely if ever delete a project, there is no
        rush on this one.
      
      * At the same time, I have started a large reorg of the code, to move
        all of the user, group, project code into modules, both in php and
        perl, turning them into first class "objects" (as far as that goes
        in php and perl). Eventually, the number of query statements
        scattered around the code will be manageable, or so I hope.
      
      * Another related part of this reorg is to make it easier to move the
        new user/project/group code in the perl backend so that it can be
        made available via the xmlrpc interface (without duplication of the
        code).
      ff9061d4
  3. 25 Oct, 2006 1 commit
    • Leigh B. Stoller's avatar
      Makefile Whacking! Try to deal with the problem caused by the delay · 7590f9c5
      Leigh B. Stoller authored
      between when something is installed and when post-install runs. Short
      of a global lock (which we probably need anyway someday), my solution
      is this. In your makefiles, add these variables before the line that
      has the include of $(TESTBED_SRCDIR)/GNUmakerules:
      
      	SETUID_BIN_SCRIPTS   =
      	SETUID_SBIN_SCRIPTS  =
      
      I have added three new rules to GNUmakerules that look like this:
      
      	$(addprefix $(SBINDIR)/, $(SETUID_SBIN_SCRIPTS)): $(SBINDIR)/%: %
      		echo "Installing (setuid) $<"
      		-mkdir -p $(INSTALL_SBINDIR)
      		$(SUDO) $(INSTALL) -o root -m 4755 $< $@
      
      Yep, your eyes ain't lying to you; use sudo to run the target so that
      install does the right thing (which is that the old file is not
      replaced until the new one has the proper attributes on it).
      
      Note that post-install is still needed for the initial install, but
      should no longer be needed for day to day installs since all that other
      stuff post-install does is mkdir/chmod on directories.
      7590f9c5
  4. 18 Jul, 2006 1 commit
    • Leigh B. Stoller's avatar
      Changes necessary for moving most of the stuff in the node_types · 624a0364
      Leigh B. Stoller authored
      table, into a new table called node_type_attributes, which is intended
      to be a more extensible way of describing nodes.
      
      The only things left in the node_types table will be type,class and the
      various isXXX boolean flags, since we use those in numerous joins all over
      the system (ie: when discriminating amongst nodes).
      
      For the most part, all of that other stuff is rarely used, or used in
      contexts where the information is needed, but not for type descrimination.
      Still, it made for a lot of queries to change!
      
      Along the way I added a NodeType library module that represents the type
      info as a perl object. I also beefed up the existing Node module, and
      started using it in more places. I also added an Interfaces module, but I
      have not done much with that yet.
      
      I have not yet removed all the slots from the node_types table; I plan to
      run the new code for a few days and then remove the slots.
      
      Example using the new NodeType object:
      
      	use NodeType;
      
      	my $typeinfo = NodeType->Lookup($type);
      
              if ($typeinfo->control_interface(\$control_iface) ||
                  !$control_iface) {
        	    warn "No control interface for $type is defined in the DB!\n";
              }
      
      or using the Node:
      
      	use Node;
      
              my $nodeobject = Node->Lookup($node_id);
              my $imageable  = $nodeobject->NodeTypeInfo()->imageable();
      or
              my $rebootable = $nodeobject->isrebootable();
      or
              $nodeobject->NodeTypeAttribute("control_interface", \$control_iface);
      
      Lots of way to accomplish the same thing, but the main point is that the
      Node is able to override the NodeType (if it wants to), which I think is
      necessary for flexibly describing one/two of a kind things like switches, etc.
      624a0364
  5. 03 Jul, 2006 1 commit
  6. 21 Jun, 2006 1 commit
    • Leigh B. Stoller's avatar
      Munge the schemacheck code to deal with all the oddities of the way · 53c29cfa
      Leigh B. Stoller authored
      mysql 5.0 dumps the schema. What a pain in the ass.
      
      Note that "timestamp" is basically impossible since its radically
      different between 3.X and 5.X, which would break schemacheck on 3.X
      based Emulabs. Since there are only three of them in the schema, I
      changed schemadiff to not look too hard at them.
      53c29cfa
  7. 23 Jan, 2006 1 commit
  8. 01 Dec, 2005 1 commit
    • Leigh B. Stoller's avatar
      Set up aging of the log table. However, this script also does some · fff1d16b
      Leigh B. Stoller authored
      special stuff at Utah (MAINSITE=1).
      
      * Before aging out, copy the entries to another DB on boss so that we
        have it forever!
      
      * As an extra special hack of the century, copy those entries over to
        ops and store in a DB over there for Kevin to look at. This code
        will go away at some point.
      fff1d16b
  9. 20 Jul, 2005 1 commit
  10. 26 Apr, 2005 1 commit
    • Leigh B. Stoller's avatar
      A watchdog daemon to try and catch (and recover from) the periodic · c47cefa1
      Leigh B. Stoller authored
      mysqld hangs that cause the entire system to grind to a halt. The
      basic theory of operation is like this:
      
      * Once a minute fork a child (protected by a 60 second timeout) to
        connect to the DB and issue a simple query. If the child can access
        the DB okay, it exits with a zero status.
      
      * If the alarm fires, the child is killed. This indicates that mysqld
        is no longer responding in a reasonable amount of time (60 seconds).
        We shift into trying to restart mysqld:
      
           * Send mysqld a TERM. Wait for 30 seconds.
      
           * Try query again; typically, the situation will not have changed one
             bit, but I do it anyway.
      
           * If mysqld was running, send it a kill -9. Wait for 15 seconds.
      
           * Start mysqld. Wait for 5 seconds.
      
           * Try query again. If query succeeds, we are done, and no one
             will have to deal with it Sunday morning at 6am (thanks Tim).
      
           * If query still fails, send email and give up trying to do fix
             anything. The daemon continues to query the DB once a minute;
             once the query succeeds (cause a human fixed things up), the
             daemon goes back into its normal mode (attempt to fix things
             next time it fails).
      
      So, the problem is what happens when someone kills off mysqld for some
      other reason. It may be that this daemon should only try to restart
      mysqld if and only if, it actually killed a running mysqld. Comments?
      c47cefa1
  11. 24 Jan, 2005 1 commit
    • Leigh B. Stoller's avatar
      Bottom line on this commit: Do not update the nodetypeXpid_permissions · 775ca147
      Leigh B. Stoller authored
      table by hand anymore! Update the group_policies table and then run
      the script to update the permissions table (sbin/update_permissions).
      
      Details:
      
      My original thought when I started this was that I would be able to
      replace the existing nodetypeXpid_permissions table with this new
      stuff. Well, it turns out that this was not a good thing to do, for a
      couple of reasons:
      
        * Engineering: We access the nodetypeXpid_permissions table from three
          different languages, and no way I wanted to rewrite this library in
          in python and php!
      
        * Performance: We access the nodetypeXpid_permissions from the web
          interface, on every single page load. In fact, we access it twice if
          if you count the FreePCs() count that we put at the top of the menu.
          Going through this library on each page load would be a serious drag.
      
      So, rather then actually get rid of the nodetypeXpid_permissions table, I
      decided to keep it as a "cache" of permissions stored in the group
      policies table. Each time you update the policy tables, we need to run
      the update_permissions script which will call into this library (see the
      TBUpdateNodeTypeXpidPermissions() routine) to reconstruct the permissions
      table. I have whacked the grantnodetype script to do exactly that.
      
      Note that we could proably do the same thing for users by creating an
      equivalent nodetypeXuid_permissions table, mapping users to types they
      are allowed to use. That would be a lot rows, but the amount of data in
      the table is small. That would give us very fine grained control of what
      we show people in the web interface. Not sure it is worth it though.
      
      I also added some instructions to previous commit in database-migrate.txt
      on populating the new group_policies table from the existing
      permissions table.
      775ca147
  12. 18 Jan, 2005 1 commit
    • Leigh B. Stoller's avatar
      Here is a checkpoint of the admission control stuff I have been working on. · 54f55585
      Leigh B. Stoller authored
      The last part is the stuff to hook it in from assign_wrapper, and some
      additional support in assign that Rob is adding for me. This comment is
      from the top of new file db/libadminctrl.pm.in and describes everything in
      detail.
      
      # Admission control policies. These are the ones I could think of, although
      # not all of these are implemented.
      #
      #  * Number of experiments per type/class (only one expt using robots).
      #
      #  * Number of experiments per project
      #  * Number of experiments per subgroup
      #  * Number of experiments per user
      #
      #  * Number of nodes per project      (nodes really means pc testnodes)
      #  * Number of nodes per subgroup
      #  * Number of nodes per user
      #
      #  * Number of nodes of a class per project
      #  * Number of nodes of a class per group
      #  * Number of nodes of a class per user
      #
      #  * Number of nodes of a type per project
      #  * Number of nodes of a type per group
      #  * Number of nodes of a type per user
      #
      #  * Number of nodes with attribute(s) per project
      #  * Number of nodes with attribute(s) per group
      #  * Number of nodes with attribute(s) per user
      #
      # So we have group (pid/gid) policies and user policies. These are stored
      # into two different tables, group_policies and user_policies, indexed in
      # the obvious manner. Each row of the table defines a count (experiments,
      # nodes, etc) and a type of thing being counted (experiments, nodes, types,
      # classes, etc). When we test for admission, we look for each matching row
      # and test each condition. All conditions must pass. No conditions means a
      # pass. There is also some "auxdata" which holds extra information needed
      # for the policy (say, the type of node being restricted).
      #
      #      uid:     a uid
      #   policy:     'experiments', 'nodes', 'type', 'class', 'attribute'
      #    count:     a number
      #  auxdata:     a string (optional)
      #
      # Example: A user policy of ('mike', 'nodes', 10) says that poor mike is
      # not allowed to have more 10 nodes at a time, while ('mike', 'type',
      # '10', 'pc850') says that mike cannot allocate more than 10 pc850s.
      #
      # The group_policies table:
      #
      #      pid:     a pid
      #      gid:     a gid
      #   policy:     'experiments', 'nodes', 'type', 'class', 'attribute'
      #    count:     a number
      #  auxdata:     a string (optional)
      #
      # Example: A project policy of ('testbed', 'testbed', 'experiments', 10)
      # says that the testbed project may not have more then 10 experiments
      # swapped in at a time, while ('testbed', 'TG1', 'nodes', 10) says that the
      # TG1 subgroup of the testbed project may not use more than 10 nodes at
      # time.
      #
      # In addition to group and user policies (which are policies that apply to
      # specific users/projects/subgroups), we also need policies that apply to
      # all users/projects/subgroups (ie: do not want to specify a particular
      # restriction for every user!). To indicate such a policy, we use a special
      # tag in the tables (for the user or pid/gid):
      #
      #      '+'  -  The policy applies to all users (or project/groups).
      #
      # Example: ('+','experiments',10) says that no user may have more then 10
      # experiments swapped in at a time. The rule overrides anything more
      # specific (say a particular user is restricted to 20 experiments; the above
      # rule overrides that and the user (all users) is restricted to 10.
      #
      # Sometimes, you want one of these special rules to apply to everyone, but
      # *allow* it to be overridden by a more specific rule. For that we use:
      #
      #      '-'  -  The policy applies to all users (or project/groups),
      #              but can be overridden by a more specific rule.
      #
      # Example: The rules:
      #
      #	('-','type',0, 'garcia')
      #       ('testbed', 'testbed', 'type', 10, 'garcia')
      #
      # says that no one is allowed to allocate garcias, unless there is specific
      # rule that allows it; in this case the testbed project can allocate them.
      #
      # There are other global policies we would like to enforce. For example,
      # "only one experiment can be using the robot testbed." Encoding this kind
      # of policy is harder, and leads down a path that can get arbitrarily
      # complex. Tha path leads to ruination, and so we want to avoid it at
      # all costs.
      #
      # Instead we define a simple global policies table that applies to all
      # experiments currently active on the testbed:
      #
      #   policy:     'nodes', 'type', 'class', 'attribute'
      #     test:     'max', others I cannot think of right now ...
      #    count:     a number
      #  auxdata:     a string
      #
      # Example: A global policy of ('nodes', 'max', 10, '') say that the maximum
      # number of nodes that may be allocated across the testbed is 10. Thats not
      # a very realistic policy of course, but ('type', 'max', 1, 'garcia') says
      # that a max of one garcia can be allocated across the testbed, which
      # effectively means only one experiment will be able to use them at once.
      # This is of course very weak, but I want to step back and give it some
      # more thought before I redo this part.
      #
      # Is that clear? Hope so, cause it gets more complicated. Some admission
      # control tests can be done early in the swap phase, before we really do
      # anything (before assign_wrapper). Others (type and class) tests cannot
      # be done here; only assign can figure out how an experiment is going to map
      # to physical nodes (remember virtual types too), and in that case we need
      # to tell assign what the "constraints" are and let it figure out what is
      # possible.
      #
      # So, in addition to the simple checks we can do, we also generate an array
      # to return to assign_wrapper with the maximum counts of each node type and
      # class that is limited by the policies. assign_wrapper will dump those
      # values into the ptop file so that assign can enforce those maximum values
      # regardless of what hardware is actually available to use. As per discussion
      # with Rob, that will look like:
      #
      #	set-type-limit <type> <limit>
      #
      # and assign will spit out a new type of violation that assign_wrapper will
      # parse.
      #
      # NOTES:
      #
      #  1) Admission control is skipped in admin mode; returns okay.
      #  2) Admission control is skipped when the pid is emulab-ops; returns okay.
      #  3) When calculating current usage, nodes reserved to emulab-ops are
      #     ignored.
      #  4) The sitevar "swap/use_admission_control" controls the use of admission
      #     control; defaults to 1 (on).
      #  5) The current policies can be viewed in the web interface. See
      #     https://www.emulab.net/showpolicies.php3
      #  6) The global policy stuff is weak. I plan to step back and think about it
      #     some more before redoing it, but it will tide us over for now.
      #
      54f55585
  13. 03 Jan, 2005 1 commit
  14. 15 Nov, 2004 1 commit
  15. 14 Nov, 2004 1 commit
  16. 28 Oct, 2004 1 commit
  17. 29 Sep, 2004 1 commit
  18. 09 Aug, 2004 1 commit
  19. 23 Jun, 2004 1 commit
    • Leigh B. Stoller's avatar
      Switch to "dynamic" creation of certain virtual nodes (jail/pcvm and · 4d5620c2
      Leigh B. Stoller authored
      vserver/pcplab). Rather then 10000s of fixed node entries in the DB,
      create node entries on the fly as needed (assign_wrapper), and then
      delete them when they are no longer used (nfree). When new nodes check
      in, no longer create all those nodes table entries (utils/newnode.in
      and tbsetup/plab/libplab.py.in).
      
      I've added a new library: db/Node.pm which is something I started a
      while back, and decided to commit, along with the support for creating
      and deleting virtual nodes. CreateVnodes() creates a new set of nodes,
      choosing non-conflicting names in the DB, and then immediately
      reserves them to the pid/eid specified. DeleteVnodes takes a list of
      vnodes and deletes them from nodes,reserved,etc. This library does a
      few other things which I am going to be playing with, so you might
      want to go read the comment at the top of the file. Feel free to speak
      up. CreateVnodes() is from called assign_wrapper when a node type has
      the "isdynamic" property. Otherwise does the existing avail/nalloc
      stuff. DeleteVnodes() is called from nfree when the node type has the
      isdynamic property.
      
      I've added a script (sql/delvnodes.pl) to run after updating the DB
      and software. All free pcvm and pcplab virtual nodes are deleted from
      the DB; reserved ones will get deleted whenever their experiment ends.
      I've noted all of this in doc/UPDATING, including setting the
      isdynamic property on pcvm in the node_types table.
      
      I've left tbsetup/plab/libplab.py.in to create a single pcplab node
      for the management sliver (still called -20). We can worry about this
      later.
      
      All this for modelnet?
      4d5620c2
  20. 05 May, 2004 1 commit
    • Leigh B. Stoller's avatar
      Add new script to change the uid of a user, since this seems to be · 9f73d582
      Leigh B. Stoller authored
      something we are continually being asked to do! This script will only
      operate in UNAPPROVED users; once a user is approved it becomes too
      difficult to deal with cause of existing directories which might be
      exported to active experiments. We can probably deal with this, but we
      rarely do this to an approved user, so its not a big deal.
      
      Obvious: This script can get out of sync with the DB, but of course.
      9f73d582
  21. 16 Feb, 2004 1 commit
  22. 11 Dec, 2003 1 commit
  23. 10 Oct, 2003 1 commit
  24. 26 Jun, 2003 1 commit
    • Leigh B. Stoller's avatar
      New script. Convert experiment to a "well formed" XML, and back again. · f7fc7ff3
      Leigh B. Stoller authored
      The conversion from XML to an experiment representation updates the
      DB, although the experiment has to exist, and only certain fields can
      be updated. For the rest of the virtual toplogy, the old toplogy is
      deleted, and the new one installed. Use with caution. This is intended
      to be hooked to the parser, and later to the vis tool.
      f7fc7ff3
  25. 28 Apr, 2003 1 commit
  26. 11 Apr, 2003 1 commit
    • Chad Barb's avatar
      · a0248828
      Chad Barb authored
      Added site variables ('sitevars').
      These are stored in the sitevariables database table.
      Each one has a name, a description (NOT OPTIONAL!), a
      default value, and a current value.
      If the current value is NULL, the default value is used.
      
      Soon, a mechanism will be added to the install process to
      ensure all needed site variables exist before installing;
      more on that when it is committed.
      
       - Added 'editsitevars.php3' page, accessable to admins
         via the 'Edit Site Variables' menu option.
      
       - Added 'setsitevar' script,
         an interface for listing, viewing in detail, and setting
         site variables.
      
       - Web interface now uses 'web/nologins' and 'web/message'
         instead of one-off database tables.
      
      NOTE that setting a variable to the default value and
      setting a variable to a value which is string-identical
      are NOT the same thing.
      (This doesn't matter yet, but when we push default values out to
       remote sites as part of our install, it will.)
      a0248828
  27. 08 Apr, 2003 1 commit
    • Mac Newbold's avatar
      Remove idlecheck. Add idletimes and idlemail (and webidlemail). · fd4eaa8d
      Mac Newbold authored
      (webidlecheck was removed in a separate commit.)
      
      idletimes is the new commandline tool for checking idleness. It shows a
      table like this:
      
      PID          EID          Nodes  HrsIdle     Last Activity     Act. type
      ------------------------------------------------------------------------
      AVQ          Tunnel           4   15.18   2003-04-07 21:18:20  tty
      pces         toy              1    3.92   2003-04-08 08:34:02  tty
      Spinglass    fawaz            1   13.76   2003-04-07 22:43:48  tty
      TempleSCTP   FTP-REAL         1   26.40   2003-04-07 10:05:18  tty,net
      testbed      bvclass          4   24.58   2003-04-07 11:54:13  net
      testbed      ltest            1   14.37   2003-04-07 22:07:02  tty
      testbed      ron-image        1  501.88   2003-03-18 13:36:28  cpu,ext,tty,net
      testbed      Tone             1  501.88   2003-03-18 13:36:30  cpu,ext,tty,net
      
      By default it only shows things that are over an idleness threshold. It
      also has a mode to show all expts. Also can show idleness/activity by node
      instead of by expt, which has a slightly different table. (s/Nodes/NodeID/)
      
      idlemail isn't finished yet, but I don't want it to hold up idletimes and
      the other changes any longer. It doesn't get used by anything yet, of
      course.
      fd4eaa8d
  28. 08 Jan, 2003 1 commit
  29. 07 Jan, 2003 1 commit
    • Robert Ricci's avatar
      New script: readycount · 616601b5
      Robert Ricci authored
      Simple command-line interface to the ready bits. Its primary
      purposes are:
      
      * Manually report ready for nodes that can't do it themselves
      * Get a list of which nodes are ready, so that you can figure out
        which one(s) aren't reporting in
      * Clear ready bits so you can use them again without restarting the
        experiment
      * Make it possible to poll ready bits on boss/ops
      616601b5
  30. 15 Aug, 2002 1 commit
  31. 25 Jul, 2002 1 commit
  32. 04 Jul, 2002 1 commit
  33. 24 Jun, 2002 1 commit
  34. 28 May, 2002 1 commit
  35. 24 May, 2002 1 commit
  36. 16 May, 2002 1 commit
    • Robert Ricci's avatar
      New script: grabron · 8bc28a0b
      Robert Ricci authored
      This script grabs updated latency from Dave Andersen't database
      of RON information, and sticks it into the widearea_recent table.
      8bc28a0b
  37. 11 Feb, 2002 1 commit
  38. 29 Jan, 2002 1 commit
    • Robert Ricci's avatar
      New script: interswitch · da928f5a
      Robert Ricci authored
      A simple little script to find links/lans that cross between switches,
      and print them out (including which switches they use, and how many
      members they have on each switch.)
      da928f5a
  39. 25 Jan, 2002 1 commit
    • Robert Ricci's avatar
      New script: dbcheck . Beginngs of a database consistency checker. · 441dfb4a
      Robert Ricci authored
      Right now, it loads foreign key information from the foreign_keys
      table of the database, and prints out information on rows that fail
      the consistency checks.
      
      The plan is that it will eventually check more things, such as the
      existence of files references in the database.
      441dfb4a
  40. 04 Jan, 2002 1 commit
    • Robert Ricci's avatar
      New script: unixgroups . Pretty simple - just a convenient way to manage the · 469dacdb
      Robert Ricci authored
      unixgroup_membershit table from the command line. Runs the appropriate
      commands to make changes in the 'real world' after the database has been
      updated. From the usage message:
      
      Usage: unixgroups <-h | -p | < <-a | -r> uid gid...> >
      -h            This message
      -p            Print group information
      -a uid gid... Add a user to one (or more) groups
      -r uid gid... Remove a user from one (or more) groups
      469dacdb