1. 14 May, 2013 1 commit
    • Leigh B Stoller's avatar
      Add prototype EC2 image import plumbing. · 980aa180
      Leigh B Stoller authored
      To create a new descriptor that will be an import from EC2 (and thus
      run under XEN), add ?ec2=1 to newimage_ez.php3. Eventually will link
      it in someplace. The form will create a XEN based VM, but instead of
      node to snapshot from, provide user@host for the EC2 instance.
      
      On the image snapshot page, instead of node use user@host for the EC2
      instance.
      
      The backend script (create_image) will call over to ops and invoke
      Srikanth's code. I have called that script ec2import-image.pl. See
      create_image for how arguments are passed to the script.
      980aa180
  2. 03 May, 2013 1 commit
  3. 14 Jan, 2013 2 commits
    • Leigh B Stoller's avatar
      Fix to previous revision. · 253f18e2
      Leigh B Stoller authored
      253f18e2
    • Leigh B Stoller's avatar
      Add creator_urn to images table so we can track actual creator · 23ad8819
      Leigh B Stoller authored
      when created via the ProtoGeni interface; we run everything as
      GeniUser and so lose that info. We want to know this for history
      information and so we can manage permissions on image deletion
      effectively.
      
      Also add updater, updater_idx, and updater_urn to track last change
      to the image.
      
      CAVEAT: For system images, we do not actually overwrite the image, but
      store it out in /proj/$pid/images and then sometime later maybe we
      copy it back to boss. This wrecks the tracking info quite effectively
      since the image may never be copied back. Needs more thought.
      23ad8819
  4. 25 Sep, 2012 1 commit
  5. 24 Sep, 2012 1 commit
    • Eric Eide's avatar
      Replace license symbols with {{{ }}}-enclosed license blocks. · 6df609a9
      Eric Eide authored
      This commit is intended to makes the license status of Emulab and
      ProtoGENI source files more clear.  It replaces license symbols like
      "EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
      blocks that contain actual license statements.
      
      This change was driven by the fact that today, most people acquire and
      track Emulab and ProtoGENI sources via git.
      
      Before the Emulab source code was kept in git, the Flux Research Group
      at the University of Utah would roll distributions by making tar
      files.  As part of that process, the Flux Group would replace the
      license symbols in the source files with actual license statements.
      
      When the Flux Group moved to git, people outside of the group started
      to see the source files with the "unexpanded" symbols.  This meant
      that people acquired source files without actual license statements in
      them.  All the relevant files had Utah *copyright* statements in them,
      but without the expanded *license* statements, the licensing status of
      the source files was unclear.
      
      This commit is intended to clear up that confusion.
      
      Most Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the Affero GNU General Public License, version 3
      (AGPLv3).
      
      Most Utah-copyrighted files related to ProtoGENI are distributed under
      the terms of the GENI Public License, which is a BSD-like open-source
      license.
      
      Some Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the GNU Lesser General Public License, version 2.1
      (LGPL).
      6df609a9
  6. 04 Sep, 2012 1 commit
  7. 29 Aug, 2012 2 commits
  8. 27 Aug, 2012 1 commit
    • Leigh B Stoller's avatar
      Add support for creating snapshot images of OpenVZ containers via the · fdd08bb4
      Leigh B Stoller authored
      web interface.
      
      Most of this commit is about making it easier to create the
      descriptors (of any kind) The newimageid_ez page now takes an optional
      node_id on which to base the new descriptor; most of the fields are
      initialized.
      
      You can also provide optional imageid and osinfo arguments, which
      allows for slightly less info to be initialized.
      
      Then on the ShowNode page, the link to create an image will take you
      to the snapshot image page if the user has write access to the image,
      otherwise to the newimageid_iz page to create and snapshot a new
      image descriptor.
      fdd08bb4
  9. 07 Jun, 2012 1 commit
    • Leigh B Stoller's avatar
      New script, clone_image to simplify create/snapshot from a node. · b01c991d
      Leigh B Stoller authored
      clone_image is a wrapper around newimageid_ez and create_image, that
      simplifies the most common operation; creating a new imageid derived
      from the image/os that is currently running in the node, and then
      taking a snapshot of the node. So for example, if node pcXXX is
      running image FREEBSD, and you want to create a custom image from that
      node, all you need to do:
      
      	boss> clone_image myfreebsd pcXXX
      
      which will create the new descriptor, deriving everything from the
      FREEBSD image on the node, and then take a snapshot from pcXXX. If
      the descriptor already exists, just take the snapshot.
      
      So what if you do:
      
      	boss> clone_image FREEBSD pcXXX
      
      well, the image is always looked up in the project the node is
      currently attached to, so in fact a new descriptor is created in that
      project, and you do not actually overwrite an image from some other
      project. 
      
      I've added some locking to images to prevent concurrent snapshots.
      This seemed like a good idea since this script is going to be used
      from the ProtoGeni interface. More on this in another commit.
      b01c991d
  10. 15 May, 2012 1 commit
  11. 04 Nov, 2011 1 commit
  12. 11 Oct, 2011 1 commit
  13. 08 Oct, 2011 2 commits
    • Mike Hibler's avatar
      Revert "Adjust the set of unix gids used for a download server." · baba7478
      Mike Hibler authored
      This reverts commit fc89eb38.
      
      Checked in a bunch of crap that was unrelated.
      baba7478
    • Mike Hibler's avatar
      Adjust the set of unix gids used for a download server. · fc89eb38
      Mike Hibler authored
      When downloading an image, start the frisbeed process with the minimum set of
      gids necessary to access the image. This includes the unix gid of the
      project that the image is in and, optionally, the unix gid of the project
      subgroup if the image is part of one.
      
      Previously, we just use the gid set of the uid of the swapper of the
      experiment. Not only was this excessive, but it might also not include the
      gids needed in the case of a "global" image that is not in the world-readable
      /usr/testbed/images directory.
      fc89eb38
  14. 30 Jun, 2011 1 commit
    • Mike Hibler's avatar
      Avoid a race condition with frisuploadd. · 362bfbfe
      Mike Hibler authored
      frisuploadd will move <foo>.tmp to <foo> when it completes. Since create_image
      monitors the size of the former, it might get an undefined value if it checks
      the size right after it have been renamed. Compensate.
      362bfbfe
  15. 20 Jun, 2011 1 commit
  16. 27 May, 2011 1 commit
  17. 25 May, 2011 1 commit
  18. 19 May, 2011 1 commit
  19. 18 May, 2011 1 commit
    • Mike Hibler's avatar
      Support image PUT (aka, "upload") and assorted minor changes. · 77dbad39
      Mike Hibler authored
      1. Support for PUT.
      
      The big change is support for uploading via the master server, based heavily
      on the prototype that Grant did. Currently only host-based (IP-based)
      authentication is done as is the case with download. Grant's SSL-based
      authentication code is "integrated" but has not even been compiled in.
      
      The PUT protocol allows for assorted gewgaws, like specifying a maximum size,
      setting a timeout value, returning size and signature info, etc.
      
      There is a new, awkwardly-named client utility "frisupload" which, like the
      download client, takes an "image ID" as an argument and requests to upload
      (PUT) that image via the master server. As with download, the image ID can
      be either of the form "<pid>/<emulab-image-name>", to upload/update an actual
      Emulab image or it can start with a "/" in which case it is considered to
      be a pathname on the server.
      
      On the server side, the master server takes PUT requests, verifies permission
      to upload the image, fires up a separate instance of an upload daemon (with
      the even catchier moniker "frisuploadd"), and returns the unicast addr/port
      info to the client which then begins the upload. The master server also acts
      as a traffic cop to make sure that downloads and uploads (or uploads and
      uploads) don't overlap.
      
      This has been integrated into the Emulab "create image" process in a
      backward-compatible way (i.e., so old admin MFSes will continue to work).
      Boy, was that fun. One not-so-desirable effect of this integration is that
      images now traverse our network twice, once to upload from node to boss and
      once for boss to write out the image file across NFS to ops. This is not
      really something that should be "fixed" in frisbee, it is only "undesirable"
      because we have a crappy NFS server.
      
      What has NOT been done includes: support of hierarchical PUT operations
      (we don't need it for either the elabinelab or subboss case), support for
      uploading standard images stored on boss (we really want something better
      than host-based authentication here), and the aforementioned support of
      SSL-based authentication.
      
      2. Other tidbits that got mixed in with PUT support:
      
      Added two new site variables:
          images/frisbee/maxrate_std
          images/frisbee/maxrate_usr
      which replace the hardwired (in mfrisbeed and frisbeelauncher before that)
      bandwidth limits for image download. mfrisbeed reads these (and the
      images/create/* variables) when it starts up or receives a HUP signal.
      These could be read from the DB on every GET/PUT, but they really don't change
      much and I needed something to test the reread-the-config-on-a-HUP code!
      
      Fixed avoidance of "problematic multicast addresses" so it would actually
      work as intended.
      
      Lots of internal "refactoring" to make up for things I did wrong the first
      time and to give the general impression that "Wow, Mike did a LOT!"
      77dbad39
  20. 18 Jan, 2011 1 commit
  21. 20 Oct, 2010 1 commit
    • Mike Hibler's avatar
      Support for no shared filesystem (unsupport for shared filesystem?) and · c1c1bce2
      Mike Hibler authored
      (eventual) support for NFS servers without race conditions!
      
      This means no NFS between nodes and ops/fs. There are still NFS mounts of
      ops on boss however.
      
      Added new defs-* variable NOSHAREDFS, which when set non-zero will disable
      the export of NFS filesystems to nodes.  Involved lots of little changes:
      
       * /users, /proj, and /share filesystems are not exported to nodes.
      
       * Returned mount info now includes an FSTYPE key which will be set to "LOCAL"
         if NOSHAREDFS is in effect (by default it is set to "NFS-RACY"; more on
         this later).  In the case where it is set to LOCAL, the other mount lines
         no longer contain REMOTE=foo settings.  Because of this change,
         THE TMCD VERSION NUMBER HAS BEEN BUMPED TO 32.
      
       * The client rc.mounts script will now create local versions of /users/*,
         /proj/<pid>, and /share when FSTYPE=LOCAL.  It first runs mkextrafs to
         create a large partition for these, since someday we will likely want
         to pre-populate these with a non-trivial amount of data.  Right now,
         the only thing that is put in the user's homedir is the standard dotfiles
         for the OS and the Emulab authorized_keys file (so you can login).
      
       * Linktest had to be modified to fetch the various results files (via
         loghole) rather than just assuming they were in /proj.  And also changed
         to invoke tevc with the local copy of the event key so it won't try to
         read it over NFS.
      
       * create_image was modified to ssh to the node and run the imagezip
         command, capturing the output of ssh.  This is controlled via the "-s"
         option which defaults to on for a NOSHAREDFS system, but can also be
         used on a normal system.
      
       * elabinelab's can be configured with/without a shared FS via the
         CONFIG_SHAREDFS attribute (note polarity change) which defaults to 1.
      
      Another new defs-* variable, NFSRACY, will some day allow you to specify
      (by setting to 0) that your NFS server does NOT have the nefarious mountd
      race condition when changing /etc/exports.  Currently, this defaults to 1
      since all versions of FreeBSD supported as an "fs" node have this "feature."
      Rumor has it that FreeBSD 8 does not have this problem nor, presumably,
      would a Linux NFS server.
      
      The only use of this variable right now is to set the FSTYPE returned by the
      tmcd "mounts" call, which in turn is used by one client script, rc.topomap
      (via a libsetup function) to determine whether it should try copying
      the topo file multiple times.
      
      Random: add python2.6 to list of python's checked for in configure.
      Random: resync defs-example-privatecnet with defs-example.
      Random: did a little code-pissin here and there.
      c1c1bce2
  22. 19 Oct, 2010 2 commits
  23. 13 May, 2010 2 commits
  24. 12 May, 2010 1 commit
  25. 13 Jun, 2008 1 commit
  26. 26 Nov, 2007 1 commit
  27. 24 Aug, 2007 1 commit
  28. 13 Aug, 2007 1 commit
  29. 06 Aug, 2007 1 commit
  30. 02 Aug, 2007 1 commit
  31. 05 Mar, 2007 1 commit
  32. 20 Nov, 2006 1 commit
  33. 20 Oct, 2006 1 commit
    • Mike Hibler's avatar
      Wow, this should make me look important! · afa5e919
      Mike Hibler authored
      Two-day boondoggle to support "/scratch", an optional large, shared filesystem
      for users.  To do this, I needed to find all the instances where /proj is used
      and behave accordingly.  The boondoggle part was the decision to gather up all
      the hardwired instances of shared directory names ("/proj", "/users", etc.)
      so that they are set in a common place (via unexposed configure variables).
      This is a boondoggle because:
      
      1. I didn't change the client-side scripts.  They need a different mechanism
         (e.g., tmcd) to get the info, configure is the wrong way.
      
      2. Even if I had done #1 it is likely--no, certain--that something would
         fail if you tried to rename "/proj" to be "/mike".  These names are just
         too ingrained.
      
      3. We may not even use "/scratch" as it turns out.
      
      Note, I also didn't fix any of the .html documentation.  Anyway, it is done.
      To maintain my illusion in the future you should:
      
      1. Have perl scripts include "use libtestbed" and use the defined PROJROOT(),
         et.al. functions where possible.  If not possible, make sure they run
         through configure and use @PROJROOT_DIR@, etc.
      
      2. Use the configure method for python, C, php and other languages.
      
      3. There are perl (TBValidUserDir) and php (VALIDUSERPATH) functions which
         you should call to determine if an NS, template parameter, tarball or
         other file are in "an acceptable location."  Use these functions where
         possible.  They know about the optional "scratch" filesystem.  Note that
         the perl function is over-engineered to handles cases that don't occur
         in nature.
      afa5e919
  34. 18 Oct, 2006 1 commit
  35. 18 Jul, 2006 1 commit
    • Leigh B. Stoller's avatar
      Changes necessary for moving most of the stuff in the node_types · 624a0364
      Leigh B. Stoller authored
      table, into a new table called node_type_attributes, which is intended
      to be a more extensible way of describing nodes.
      
      The only things left in the node_types table will be type,class and the
      various isXXX boolean flags, since we use those in numerous joins all over
      the system (ie: when discriminating amongst nodes).
      
      For the most part, all of that other stuff is rarely used, or used in
      contexts where the information is needed, but not for type descrimination.
      Still, it made for a lot of queries to change!
      
      Along the way I added a NodeType library module that represents the type
      info as a perl object. I also beefed up the existing Node module, and
      started using it in more places. I also added an Interfaces module, but I
      have not done much with that yet.
      
      I have not yet removed all the slots from the node_types table; I plan to
      run the new code for a few days and then remove the slots.
      
      Example using the new NodeType object:
      
      	use NodeType;
      
      	my $typeinfo = NodeType->Lookup($type);
      
              if ($typeinfo->control_interface(\$control_iface) ||
                  !$control_iface) {
        	    warn "No control interface for $type is defined in the DB!\n";
              }
      
      or using the Node:
      
      	use Node;
      
              my $nodeobject = Node->Lookup($node_id);
              my $imageable  = $nodeobject->NodeTypeInfo()->imageable();
      or
              my $rebootable = $nodeobject->isrebootable();
      or
              $nodeobject->NodeTypeAttribute("control_interface", \$control_iface);
      
      Lots of way to accomplish the same thing, but the main point is that the
      Node is able to override the NodeType (if it wants to), which I think is
      necessary for flexibly describing one/two of a kind things like switches, etc.
      624a0364