1. 04 Jun, 2018 1 commit
    • David Johnson's avatar
      Docker server-side core, esp new libimageops support for Docker images. · 66366489
      David Johnson authored
      The docker VM server-side goo is mostly identical to Xen, with slightly
      different handling for parent images.  We also support loading external
      Docker images (i.e. those without a real imageid in our DB; in that
      case, user has to set a specific stub image, and some extra per-vnode
      metadata (a URI that points to a Docker registry/image repo/tag);
      the Docker clientside handles the rest.
      
      Emulab Docker images map to a Emulab imageid:version pretty seamlessly.
      For instance, the Emulab `emulab-ops/docker-foo-bar:1` image would map
      to `<local-registry-URI>/emulab-ops/emulab-ops/docker-foo-bar:1`; the
      mapping is `<local-registry-URI>/pid/gid/imagename:version`.  Docker
      repository names are lowercase-only, so we handle that for the user; but
      I would prefer that users use lowercase Emulab imagenames for all Docker
      images; that will help us.  That is not enforced in the code; it will
      appear in the documentation, and we'll see.
      
      Full Docker imaging relies on several other libraries
      (https://gitlab.flux.utah.edu/emulab/pydockerauth,
      https://gitlab.flux.utah.edu/emulab/docker-registry-py).  Each
      Emulab-based cluster must currently run its own private registry to
      support image loading/capture (note however that if capture is
      unnecessary, users can use the external images path instead).  The
      pydockerauth library is a JWT token server that runs out of boss's
      Apache and implements authn/authz for the per-Emulab Docker registry
      (probably running on ops, but could be anywhere) that stores images and
      arbitrates upload/download access.  For instance, nodes in an experiment
      securely pull images using their pid/eid eventkey; and the pydockerauth
      emulab authz module knows what images the node is allowed to pull
      (i.e. sched_reloads, the current image the node is running, etc).  Real
      users can also pull images via user/pass, or bogus user/pass + Emulab
      SSL cert.  GENI credential-based authn/z was way too much work, sadly.
      There are other auth/z paths (i.e. for admins, temp tokens for secure
      operations) as well.
      
      As far as Docker image distribution in the federation, we use the same
      model as for regular ndz images.  Remote images are pulled in to the
      local cluster's Docker registry on-demand from their source cluster via
      admin token auth (note that all clusters in the federation have
      read-only access to the entire registries of any other cluster in the
      federation, so they can pull images).  Emulab imageid handling is the
      same as the existing ndz case.  For instance, image versions are lazily
      imported, on-demand; local version numbers may not match the remote
      image source cluster's version numbers.  This will potentially be a
      bigger problem in the Docker universe; Docker users expect to be able to
      reference any image version at any time anywhere.  But that is of course
      handleable with some ex post facto synchronization flag day, at least
      for the Docker images.
      
      The big new thing supporting native Docker image usage is the guts of a
      refactor of the utils/image* scripts into a new library, libimageops;
      this is necessary to support Docker images, which are stored in their
      own registry using their own custom protocols, so not amenable to our
      file-based storage.  Note: the utils/image* scripts currently call out
      to libimageops *only if* the image format is docker; all other images
      continue on the old paths in utils/image*, which all still remain
      intact, or minorly-changed to support libimageops.
      
      libimageops->New is the factory-style mechanism to get a libimageops
      that works for your image format or node type.  Once you have a
      libimageops instance, you can invoke normal image logical operations
      (CreateImage, ImageValidate, ImageRelease, et al).  I didn't do every
      single operation (for instance, I haven't yet dealt with image_import
      beyond essentially generalizing DownLoadImage by image format).
      Finally, each libimageops is stateless; another design would have been
      some statefulness for more complicated operations.   You will see that
      CreateImage, for instance, is written in a helper-subclass style that
      blurs some statefulness; however, it was the best match for the existing
      body of code.  We can revisit that later if the current argument-passing
      convention isn't loved.
      
      There are a couple outstanding issues.  Part of the security model here
      is that some utils/image* scripts are setuid, so direct libimageops
      library calls are not possible from a non-setuid context for some
      operations.  This is non-trivial to resolve, and might not be worthwhile
      to resolve any time soon.  Also, some of the scripts write meaningful,
      traditional content to stdout/stderr, and this creates a tension for
      direct library calls that is not entirely resolved yet.  Not hard, just
      only partly resolved.
      
      Note that tbsetup/libimageops_ndz.pm.in is still incomplete; it needs
      imagevalidate support.  Thus, I have not even featurized this yet; I
      will get to that as I have cycles.
      66366489
  2. 31 May, 2017 1 commit
  3. 16 Mar, 2017 1 commit
  4. 15 Mar, 2017 1 commit
    • Leigh Stoller's avatar
      Okay, tricky change; ignore the path that comes into newimageid_ez; the · 4ed3d431
      Leigh Stoller authored
      code to construct a path is duplicated is 5 different places and they
      have gone out of sync. We have not allowed users to set the path for
      years, but we were still constructing a path in php code and passing it
      along. Throw them all away and construct the path the way we want it.
      
      Apply same rules to admins; admins don't know what they are doing
      either. I know, I'm an admin.
      
      But if the admin sets the path to start with /usr/testbed/images, throw
      that away, and construct proper path in /usr/testbed/images.
      4ed3d431
  5. 06 Oct, 2016 1 commit
  6. 21 Mar, 2016 1 commit
  7. 21 Aug, 2015 1 commit
  8. 19 Jun, 2015 1 commit
  9. 22 May, 2015 1 commit
  10. 21 May, 2015 1 commit
  11. 18 May, 2015 1 commit
  12. 15 May, 2015 1 commit
    • Leigh Stoller's avatar
      Directory based image paths. · 3a21f39e
      Leigh Stoller authored
      Soon, we will have images with both full images and deltas, for the same
      image version. To make this possible, the image path will now be a
      directory instead of a file, and all of the versions (ndz,sig,sha1,delta)
      files will reside in the directory.
      
      A new config variable IMAGEDIRECTORIES turns this on, there is also a check
      for the ImageDiretories feature. This is applied only when a brand new
      image is created; a clone version of the image inherits the path it started
      with. Yes, you can have a mix of directory based and file based image
      descriptors.
      
      When it is time to convert all images over, there is a script called
      imagetodir that will go through all image descriptors, create the
      directory, move/rename all the files, and update the descriptors.
      Ultimately, we will not support file based image paths.
      
      I also added versioning to the image metadata descriptors so that going
      forward, old clients can handle a descriptor from a new server.
      3a21f39e
  13. 10 Mar, 2015 1 commit
  14. 09 Mar, 2015 1 commit
  15. 05 Mar, 2015 1 commit
  16. 03 Feb, 2015 1 commit
  17. 04 Nov, 2014 1 commit
    • Leigh Stoller's avatar
      Add runsonxen script to set the bits of DB state required. · 04c35b0b
      Leigh Stoller authored
      	usage: runsonxen [-p <parent>] <imageid>
      	usage: runsonxen -a [-p <parent>]
      	usage: runsonxen -c <imageid>
      	Options:
      	 -n      - Impotent mode
      	 -c      - Clear XEN parent settings completely
      	 -a      - Operate on all current XEN capable images
      	 -p      - Set default parent; currently XEN43-64-STD
      04c35b0b
  18. 01 Nov, 2014 1 commit
  19. 13 Oct, 2014 1 commit
  20. 08 Oct, 2014 1 commit
  21. 09 Jul, 2014 1 commit
  22. 12 Jun, 2014 1 commit
    • Kirk Webb's avatar
      Update the use of realpath across all perl scripts · 3f167217
      Kirk Webb authored
      Change to use the realpath function in the 'Cwd' module instead of
      calling realpath via the shell.  The shell command varies in its
      reaction to a missing final path component.  On some platforms (Linux,
      FBSD10+) realpath reports an error if the final component doesn't exist
      on the filesystem.  On others (FBSD < 10), it does not report an error.
      
      The perl function from 'Cwd' emulates the same behavior as FBSD prior to
      version 10, which is the behavior the scripts expect.
      
      From here on out, instead of using `realpath`, do the following:
      
      use Cwd qw(realpath);
      ..
      ..
      my $realpath = realpath($somepath);
      3f167217
  23. 06 Feb, 2014 1 commit
  24. 28 Jun, 2013 1 commit
  25. 24 Sep, 2012 1 commit
    • Eric Eide's avatar
      Replace license symbols with {{{ }}}-enclosed license blocks. · 6df609a9
      Eric Eide authored
      This commit is intended to makes the license status of Emulab and
      ProtoGENI source files more clear.  It replaces license symbols like
      "EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
      blocks that contain actual license statements.
      
      This change was driven by the fact that today, most people acquire and
      track Emulab and ProtoGENI sources via git.
      
      Before the Emulab source code was kept in git, the Flux Research Group
      at the University of Utah would roll distributions by making tar
      files.  As part of that process, the Flux Group would replace the
      license symbols in the source files with actual license statements.
      
      When the Flux Group moved to git, people outside of the group started
      to see the source files with the "unexpanded" symbols.  This meant
      that people acquired source files without actual license statements in
      them.  All the relevant files had Utah *copyright* statements in them,
      but without the expanded *license* statements, the licensing status of
      the source files was unclear.
      
      This commit is intended to clear up that confusion.
      
      Most Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the Affero GNU General Public License, version 3
      (AGPLv3).
      
      Most Utah-copyrighted files related to ProtoGENI are distributed under
      the terms of the GENI Public License, which is a BSD-like open-source
      license.
      
      Some Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the GNU Lesser General Public License, version 2.1
      (LGPL).
      6df609a9
  26. 04 Sep, 2012 2 commits
  27. 29 Aug, 2012 1 commit
  28. 27 Aug, 2012 1 commit
  29. 11 Jul, 2012 1 commit
  30. 28 Jun, 2012 1 commit
  31. 15 May, 2012 1 commit
  32. 12 Mar, 2012 1 commit
    • Mike Hibler's avatar
      Print a less obtuse error if they attempt to image an "empty" partition. · 31e17da2
      Mike Hibler authored
      This is in the node type check where we look at what is currently in the
      partition they are imaging, and base the set of valid node type on that.
      However, if they specify a partition that doesn't currently have an image,
      we would previously effectively tell them that the image will not run on
      the node type that it is currently running on!
      
      Now we just say "there is no current image in that partition."
      31e17da2
  33. 06 Mar, 2012 1 commit
  34. 13 Oct, 2010 1 commit
  35. 12 Oct, 2010 1 commit
  36. 10 Aug, 2010 1 commit
    • Leigh Stoller's avatar
      A bunch of little changes to make bootstrapping images work properly. · 37ffe0b9
      Leigh Stoller authored
      Basically, we cannot create an image descriptor without any types
      defined, and we cannot create any types without a default image
      descriptor. I broke the circle by adding a stub pc type into the
      database fill file so that all Emulabs start with a predefined type.
      Then you can load the descriptor file, create your actual types, and
      then go back and edit the image descriptors to set the types those
      images run on (you always had to do this).
      37ffe0b9
  37. 09 Apr, 2010 1 commit
  38. 24 Oct, 2008 1 commit
  39. 10 Sep, 2008 1 commit
    • Mike Hibler's avatar
      Slight beefing up of support for alternate MBRs: · 31009d09
      Mike Hibler authored
       * when creating an image from a node, make sure the new image
         gets the MBR version used by the existing image
       * when loading a single-partition image that requires a different
         MBR, invalidate all other existing partition ("invalidate" in the
         sense that we remove any partitions table entries, we don't do anything
         to the disk)
      31009d09