1. 10 Oct, 2011 1 commit
    • Leigh B Stoller's avatar
      Add support for sharing images between projects. New table called · 646b64f6
      Leigh B Stoller authored
      image_permissions stores access info for images. You can share an
      image with a user or a group (project), and you can specify write
      access to allow updating the image in place. Note that write access
      does not allow the descriptor to be modified, only the image itself.
      Well, that is how it will be after Mike changes mfrisbeed.
      
      The front end script to modify permissions is grantimage:
      
      	boss> grantimage -u stoller -w tbres,myimage
      	boss> grantimage -u stoller -w tbres,myimage
      
      which grants write access to stoller. Or:
      
      	boss> grantimage -g testbed,testbed tbres,myimage
      
      which grants access to the testbed project. Notice that you can
      specify subgroups this way.
      
      	boss> grantimage -l tbres,myimage
      
      will give you a list of current permissions. To revoke, just add -r
      option:
      
      	boss> grantimage -g testbed,testbed -r tbres,myimage
      
      Who is allowed to grant access to an image? 1) An adminstrator of
      course, 2) the image creator, and 3) any group_root in the group that
      the image belongs to. Being granted access to use an image does not
      confer permission to grant access to others.
      
      One last task; while the web interface displays the permissions, there
      is no web interface to modify the permissions; users will still have
      to ask us for now.
      646b64f6
  2. 08 Oct, 2011 2 commits
    • Mike Hibler's avatar
      Revert "Adjust the set of unix gids used for a download server." · baba7478
      Mike Hibler authored
      This reverts commit fc89eb38.
      
      Checked in a bunch of crap that was unrelated.
      baba7478
    • Mike Hibler's avatar
      Adjust the set of unix gids used for a download server. · fc89eb38
      Mike Hibler authored
      When downloading an image, start the frisbeed process with the minimum set of
      gids necessary to access the image. This includes the unix gid of the
      project that the image is in and, optionally, the unix gid of the project
      subgroup if the image is part of one.
      
      Previously, we just use the gid set of the uid of the swapper of the
      experiment. Not only was this excessive, but it might also not include the
      gids needed in the case of a "global" image that is not in the world-readable
      /usr/testbed/images directory.
      fc89eb38
  3. 03 Oct, 2011 1 commit
  4. 30 Sep, 2011 1 commit
  5. 21 Sep, 2011 1 commit
  6. 20 Sep, 2011 1 commit
  7. 15 Sep, 2011 1 commit
  8. 14 Sep, 2011 1 commit
  9. 30 Aug, 2011 1 commit
  10. 17 Aug, 2011 1 commit
  11. 12 Aug, 2011 4 commits
    • Leigh B Stoller's avatar
      Lets make it easier to manage pre reservations (Mike, this was Rob's · 261c2a1a
      Leigh B Stoller authored
      idea).
      
      New script and table to manage node pre reservations. Lets just look
      at the script.
      
      To create a reservation:
      
          myboss> wap prereserve -t pc850 testbed 2
          Node reservation request for 2 nodes has been created.
      
      To see the reservation status for testbed
      
          myboss> wap prereserve -i testbed
          Project         Cnt (Cur)  Creator    When               Pri Types
          -------------------------------------------------------------
          testbed         1 (1)      stoller    2011-08-12 12:39:07 0   pc850
      
          which says 1 node is pending and 1 node has already been
          pre-reserved. 
      
      To clear the above reservation request (and optionally, clean
      reserved_pid from the nodes table).
      
          myboss> wap prereserve -c -r testbed
      
          The -r is optional, otherwise just the reservation request is
          cleared, and nodes continue to be pre-reserved to the project.
      
      To see a list of all reservation requests:
      
          myboss> wap prereserve -l
      
      
      So, when a node is released in nfree, we look at the reservation
      status for the node and any pending reservation requests.
      
      1. If the node has a reserved_pid and that request is still pending
         (still in the table), nothing is changed.
      
      2. If the node has a reserved_pid, but the request has been cleared
         from the pending table, then clear reserved_pid.
      
      3. If reserved_pid is null, and there are pending requests, then pick
         the highest priority, most recent dated, request, and set
         reserved_pid to that project.
      
      Options:
      
      * -n <pri> - is how you set a priority. Lowest is zero, choose a
        higher number if you want this reservation request to be considered
        before others. In a tie, look at the date of creation, and use the
        oldest.
      
      * -t <typelist> - a comma separated list of types you want to
        consider. Types are considered in order, but not in the fancy way
        you might imagine.
      261c2a1a
    • Leigh B Stoller's avatar
      Minor fixes. · 02f94859
      Leigh B Stoller authored
      02f94859
    • Leigh B Stoller's avatar
      Lets make it easier to manage pre reservations (Mike, this was Rob's · 5c998ffc
      Leigh B Stoller authored
      idea).
      
      New script and table to manage node pre reservations. Lets just look
      at the script.
      
      To create a reservation:
      
          myboss> wap prereserve -t pc850 testbed 2
          Node reservation request for 2 nodes has been created.
      
      To see the reservation status for testbed
      
          myboss> wap prereserve -i testbed
          Project         Cnt (Cur)  Creator    When               Pri Types
          -------------------------------------------------------------
          testbed         1 (1)      stoller    2011-08-12 12:39:07 0   pc850
      
          which says 1 node is pending and 1 node has already been
          pre-reserved. 
      
      To clear the above reservation request (and optionally, clean
      reserved_pid from the nodes table).
      
          myboss> wap prereserve -c -r testbed
      
          The -r is optional, otherwise just the reservation request is
          cleared, and nodes continue to be pre-reserved to the project.
      
      To see a list of all reservation requests:
      
          myboss> wap prereserve -l
      
      
      So, when a node is released in nfree, we look at the reservation
      status for the node and any pending reservation requests.
      
      1. If the node has a reserved_pid and that request is still pending
         (still in the table), nothing is changed.
      
      2. If the node has a reserved_pid, but the request has been cleared
         from the pending table, then clear reserved_pid.
      
      3. If reserved_pid is null, and there are pending requests, then pick
         the highest priority, most recent dated, request, and set
         reserved_pid to that project.
      
      Options:
      
      * -n <pri> - is how you set a priority. Lowest is zero, choose a
        higher number if you want this reservation request to be considered
        before others. In a tie, look at the date of creation, and use the
        oldest.
      
      * -t <typelist> - a comma separated list of types you want to
        consider. Types are considered in order, but not in the fancy way
        you might imagine.
      5c998ffc
    • Jonathon Duerig's avatar
  12. 10 Aug, 2011 1 commit
  13. 30 Jun, 2011 1 commit
    • Mike Hibler's avatar
      Avoid a race condition with frisuploadd. · 362bfbfe
      Mike Hibler authored
      frisuploadd will move <foo>.tmp to <foo> when it completes. Since create_image
      monitors the size of the former, it might get an undefined value if it checks
      the size right after it have been renamed. Compensate.
      362bfbfe
  14. 20 Jun, 2011 1 commit
  15. 27 May, 2011 1 commit
  16. 25 May, 2011 2 commits
  17. 19 May, 2011 1 commit
  18. 18 May, 2011 1 commit
    • Mike Hibler's avatar
      Support image PUT (aka, "upload") and assorted minor changes. · 77dbad39
      Mike Hibler authored
      1. Support for PUT.
      
      The big change is support for uploading via the master server, based heavily
      on the prototype that Grant did. Currently only host-based (IP-based)
      authentication is done as is the case with download. Grant's SSL-based
      authentication code is "integrated" but has not even been compiled in.
      
      The PUT protocol allows for assorted gewgaws, like specifying a maximum size,
      setting a timeout value, returning size and signature info, etc.
      
      There is a new, awkwardly-named client utility "frisupload" which, like the
      download client, takes an "image ID" as an argument and requests to upload
      (PUT) that image via the master server. As with download, the image ID can
      be either of the form "<pid>/<emulab-image-name>", to upload/update an actual
      Emulab image or it can start with a "/" in which case it is considered to
      be a pathname on the server.
      
      On the server side, the master server takes PUT requests, verifies permission
      to upload the image, fires up a separate instance of an upload daemon (with
      the even catchier moniker "frisuploadd"), and returns the unicast addr/port
      info to the client which then begins the upload. The master server also acts
      as a traffic cop to make sure that downloads and uploads (or uploads and
      uploads) don't overlap.
      
      This has been integrated into the Emulab "create image" process in a
      backward-compatible way (i.e., so old admin MFSes will continue to work).
      Boy, was that fun. One not-so-desirable effect of this integration is that
      images now traverse our network twice, once to upload from node to boss and
      once for boss to write out the image file across NFS to ops. This is not
      really something that should be "fixed" in frisbee, it is only "undesirable"
      because we have a crappy NFS server.
      
      What has NOT been done includes: support of hierarchical PUT operations
      (we don't need it for either the elabinelab or subboss case), support for
      uploading standard images stored on boss (we really want something better
      than host-based authentication here), and the aforementioned support of
      SSL-based authentication.
      
      2. Other tidbits that got mixed in with PUT support:
      
      Added two new site variables:
          images/frisbee/maxrate_std
          images/frisbee/maxrate_usr
      which replace the hardwired (in mfrisbeed and frisbeelauncher before that)
      bandwidth limits for image download. mfrisbeed reads these (and the
      images/create/* variables) when it starts up or receives a HUP signal.
      These could be read from the DB on every GET/PUT, but they really don't change
      much and I needed something to test the reread-the-config-on-a-HUP code!
      
      Fixed avoidance of "problematic multicast addresses" so it would actually
      work as intended.
      
      Lots of internal "refactoring" to make up for things I did wrong the first
      time and to give the general impression that "Wow, Mike did a LOT!"
      77dbad39
  19. 12 May, 2011 1 commit
  20. 27 Apr, 2011 1 commit
  21. 05 Apr, 2011 1 commit
  22. 04 Apr, 2011 1 commit
  23. 01 Apr, 2011 1 commit
  24. 18 Mar, 2011 1 commit
  25. 18 Jan, 2011 2 commits
  26. 13 Jan, 2011 1 commit
  27. 11 Jan, 2011 1 commit
    • Mike Hibler's avatar
      More work toward getting this working on subboss. · 8d80301e
      Mike Hibler authored
      More work on the hierarchical configuration for subboss. When doing host-based
      authentication, allow client to pass an explicit host (IP) to the mserver.
      If the mserver is configured to allow it, that IP is used for authenticating
      the request instead of the caller's IP. Add a default ("null") configuration
      so the mserver can operate out-of-the-box with no config file. The goal of
      these two changes is for an mserver instance with the default config and a
      proxy option to serve the needs of a subboss node (i.e., so no explicit
      configuration will be needed).
      8d80301e
  28. 07 Jan, 2011 1 commit
  29. 15 Dec, 2010 1 commit
  30. 12 Nov, 2010 1 commit
  31. 25 Oct, 2010 1 commit
    • Leigh B Stoller's avatar
      New module, called Emulab Features. The basic usage (see tbswap) is: · 1d430992
      Leigh B Stoller authored
      use EmulabFeatures;
      
      if (EmulabFeatures->FeatureEnabled("NewMapper", $user, $group, $experiment)) {
         # Do something
      }
      else {
         # Do something else.
      }
      
      where $user, $group, and $experiment is the current Emulab user, group, and
      experiment the script is operating as. Any of them can be undef. Note that
      features can easily be globally enabled or disabled (bypassing user/group
      check). See below.
      
      There are two scripts to deal with features. The easy one is the script to
      grant (or revoke) feature usage to a particular user or group or experiment:
      
      boss> wap grantfeature -u stoller NewMapper
      boss> wap grantfeature -p geni NewMapper
      boss> wap grantfeature -e geni,myexp NewMapper
      
      Add -r to revoke the feature.
      
      The other script is for managing features. To create a new feature:
      
      boss> wap emulabfeature create NewFeature 'A pithy description'
      
      which adds the feature to the emulab_features DB table. Use "delete"
      to remove a feature from the DB.
      
      You can globally enable and disable features for all users/groups (the
      user/group checks are bypassed). Global disable overrides global
      enable. There are actually two different flags. Lots of rope, I mean
      flexibility.
      
      boss> wap emulabfeature enable NewFeature 1
      boss> wap emulabfeature enable NewFeature 0
      
      boss> wap emulabfeature disable NewFeature 1
      boss> wap emulabfeature disable NewFeature 0
      
      To display a list of all features and associated settings:
      
      boss> wap emulabfeature list
      
      To show the details (including the users and groups) of a specific
      feature:
      
      boss> wap emulabfeature show NewFeature
      
      Oh, if a test is made in the code for a feature, and that feature is
      not in the emulab_features table (as might be the case on other
      Emulab's), the feature is "disabled".
      1d430992
  32. 20 Oct, 2010 3 commits
    • Leigh B Stoller's avatar
    • Leigh B Stoller's avatar
      Set new sitevar general/testbed_shutdown ... intended to replace · 551d0933
      Leigh B Stoller authored
      some of the places we use nologins.
      551d0933
    • Mike Hibler's avatar
      Support for no shared filesystem (unsupport for shared filesystem?) and · c1c1bce2
      Mike Hibler authored
      (eventual) support for NFS servers without race conditions!
      
      This means no NFS between nodes and ops/fs. There are still NFS mounts of
      ops on boss however.
      
      Added new defs-* variable NOSHAREDFS, which when set non-zero will disable
      the export of NFS filesystems to nodes.  Involved lots of little changes:
      
       * /users, /proj, and /share filesystems are not exported to nodes.
      
       * Returned mount info now includes an FSTYPE key which will be set to "LOCAL"
         if NOSHAREDFS is in effect (by default it is set to "NFS-RACY"; more on
         this later).  In the case where it is set to LOCAL, the other mount lines
         no longer contain REMOTE=foo settings.  Because of this change,
         THE TMCD VERSION NUMBER HAS BEEN BUMPED TO 32.
      
       * The client rc.mounts script will now create local versions of /users/*,
         /proj/<pid>, and /share when FSTYPE=LOCAL.  It first runs mkextrafs to
         create a large partition for these, since someday we will likely want
         to pre-populate these with a non-trivial amount of data.  Right now,
         the only thing that is put in the user's homedir is the standard dotfiles
         for the OS and the Emulab authorized_keys file (so you can login).
      
       * Linktest had to be modified to fetch the various results files (via
         loghole) rather than just assuming they were in /proj.  And also changed
         to invoke tevc with the local copy of the event key so it won't try to
         read it over NFS.
      
       * create_image was modified to ssh to the node and run the imagezip
         command, capturing the output of ssh.  This is controlled via the "-s"
         option which defaults to on for a NOSHAREDFS system, but can also be
         used on a normal system.
      
       * elabinelab's can be configured with/without a shared FS via the
         CONFIG_SHAREDFS attribute (note polarity change) which defaults to 1.
      
      Another new defs-* variable, NFSRACY, will some day allow you to specify
      (by setting to 0) that your NFS server does NOT have the nefarious mountd
      race condition when changing /etc/exports.  Currently, this defaults to 1
      since all versions of FreeBSD supported as an "fs" node have this "feature."
      Rumor has it that FreeBSD 8 does not have this problem nor, presumably,
      would a Linux NFS server.
      
      The only use of this variable right now is to set the FSTYPE returned by the
      tmcd "mounts" call, which in turn is used by one client script, rc.topomap
      (via a libsetup function) to determine whether it should try copying
      the topo file multiple times.
      
      Random: add python2.6 to list of python's checked for in configure.
      Random: resync defs-example-privatecnet with defs-example.
      Random: did a little code-pissin here and there.
      c1c1bce2