1. 05 Jun, 2018 1 commit
  2. 04 Jun, 2018 1 commit
    • David Johnson's avatar
      Docker server-side core, esp new libimageops support for Docker images. · 66366489
      David Johnson authored
      The docker VM server-side goo is mostly identical to Xen, with slightly
      different handling for parent images.  We also support loading external
      Docker images (i.e. those without a real imageid in our DB; in that
      case, user has to set a specific stub image, and some extra per-vnode
      metadata (a URI that points to a Docker registry/image repo/tag);
      the Docker clientside handles the rest.
      Emulab Docker images map to a Emulab imageid:version pretty seamlessly.
      For instance, the Emulab `emulab-ops/docker-foo-bar:1` image would map
      to `<local-registry-URI>/emulab-ops/emulab-ops/docker-foo-bar:1`; the
      mapping is `<local-registry-URI>/pid/gid/imagename:version`.  Docker
      repository names are lowercase-only, so we handle that for the user; but
      I would prefer that users use lowercase Emulab imagenames for all Docker
      images; that will help us.  That is not enforced in the code; it will
      appear in the documentation, and we'll see.
      Full Docker imaging relies on several other libraries
      https://gitlab.flux.utah.edu/emulab/docker-registry-py).  Each
      Emulab-based cluster must currently run its own private registry to
      support image loading/capture (note however that if capture is
      unnecessary, users can use the external images path instead).  The
      pydockerauth library is a JWT token server that runs out of boss's
      Apache and implements authn/authz for the per-Emulab Docker registry
      (probably running on ops, but could be anywhere) that stores images and
      arbitrates upload/download access.  For instance, nodes in an experiment
      securely pull images using their pid/eid eventkey; and the pydockerauth
      emulab authz module knows what images the node is allowed to pull
      (i.e. sched_reloads, the current image the node is running, etc).  Real
      users can also pull images via user/pass, or bogus user/pass + Emulab
      SSL cert.  GENI credential-based authn/z was way too much work, sadly.
      There are other auth/z paths (i.e. for admins, temp tokens for secure
      operations) as well.
      As far as Docker image distribution in the federation, we use the same
      model as for regular ndz images.  Remote images are pulled in to the
      local cluster's Docker registry on-demand from their source cluster via
      admin token auth (note that all clusters in the federation have
      read-only access to the entire registries of any other cluster in the
      federation, so they can pull images).  Emulab imageid handling is the
      same as the existing ndz case.  For instance, image versions are lazily
      imported, on-demand; local version numbers may not match the remote
      image source cluster's version numbers.  This will potentially be a
      bigger problem in the Docker universe; Docker users expect to be able to
      reference any image version at any time anywhere.  But that is of course
      handleable with some ex post facto synchronization flag day, at least
      for the Docker images.
      The big new thing supporting native Docker image usage is the guts of a
      refactor of the utils/image* scripts into a new library, libimageops;
      this is necessary to support Docker images, which are stored in their
      own registry using their own custom protocols, so not amenable to our
      file-based storage.  Note: the utils/image* scripts currently call out
      to libimageops *only if* the image format is docker; all other images
      continue on the old paths in utils/image*, which all still remain
      intact, or minorly-changed to support libimageops.
      libimageops->New is the factory-style mechanism to get a libimageops
      that works for your image format or node type.  Once you have a
      libimageops instance, you can invoke normal image logical operations
      (CreateImage, ImageValidate, ImageRelease, et al).  I didn't do every
      single operation (for instance, I haven't yet dealt with image_import
      beyond essentially generalizing DownLoadImage by image format).
      Finally, each libimageops is stateless; another design would have been
      some statefulness for more complicated operations.   You will see that
      CreateImage, for instance, is written in a helper-subclass style that
      blurs some statefulness; however, it was the best match for the existing
      body of code.  We can revisit that later if the current argument-passing
      convention isn't loved.
      There are a couple outstanding issues.  Part of the security model here
      is that some utils/image* scripts are setuid, so direct libimageops
      library calls are not possible from a non-setuid context for some
      operations.  This is non-trivial to resolve, and might not be worthwhile
      to resolve any time soon.  Also, some of the scripts write meaningful,
      traditional content to stdout/stderr, and this creates a tension for
      direct library calls that is not entirely resolved yet.  Not hard, just
      only partly resolved.
      Note that tbsetup/libimageops_ndz.pm.in is still incomplete; it needs
      imagevalidate support.  Thus, I have not even featurized this yet; I
      will get to that as I have cycles.
  3. 30 Mar, 2018 1 commit
    • Mike Hibler's avatar
      Support for frisbee direct image upload to fs node. · 99943a19
      Mike Hibler authored
      We have had issues with uploading images to boss where they are then written
      across NFS to ops. That seems to be a network hop too far on CloudLab Utah
      where we have a 10Gb control network. We get occasional transcient timeouts
      from somewhere in the TCP code. With the convoluted path through real and
      virtual NICs, some with offloading, some without, packets wind up getting
      out of order and someone gets far enough behind to cause problems.
      So we work around it.
      If IMAGEUPLOADTOFS is defined in the defs-* file, we will run a frisbee
      master server on the fs (ops) node and the image creation path directs the
      nodes to use that server. There is a new hack configuration for the master
      server "upload-only" which is extremely specific to ops: it validates the
      upload with the boss master server and, if allowed, fires up an upload
      server for the client to talk to. The image will thus be directly uploaded
      to the local (ZFS) /proj or /groups filesystems on ops. This seems to be
      enough to get around the problem.
      Note that we could allow this master server to serve downloads as well to
      avoid the analogous problem in that direction, but this to date has not
      been a problem.
      NOTE: the ops node must be in the nodes table in the DB or else boss will
      not validate proxied requests from it. The standard install procedure is
      supposed to add ops, but we have a couple of clusters where it is not in
      the table!
  4. 03 Feb, 2018 1 commit
  5. 09 Jan, 2018 1 commit
    • Mike Hibler's avatar
      Yet another layer of backward compat... · afa1569d
      Mike Hibler authored
      If we support provenance but not deltas, then we do not use the
      newer create-versioned-image when creating images from Xen vnodes.
      However, we had a bug in that path where we would then not pass the
      imageid argument to the old script, resulting in us spewing the image
      out to stdout which got put in the logfile.
  6. 25 Apr, 2017 2 commits
    • Leigh B Stoller's avatar
      Changes to checkquota and create_image: · 028d4164
      Leigh B Stoller authored
      1. Add option (-m) to checkquota that says to look for a minimum amount
         of space. This is an additional check, on the file system in
         question.  So when -p option is give, we want that amount of space in
         the project directory. Add new -g option for checking quota in the
         group directory.
      2. Change create_image to supply -p or -g option as needed, and check
         for at 3GB of space (-m 3GB). I will make this a sitevar if it works
      3. manage_instance returns the space error to the web ui.
    • Leigh B Stoller's avatar
  7. 24 Apr, 2017 3 commits
  8. 27 Mar, 2017 1 commit
  9. 14 Mar, 2017 1 commit
    • Leigh B Stoller's avatar
      Minor cleanup and fixes: · ad48bbdb
      Leigh B Stoller authored
      1. Small bits of cleanup to remove hardwired paths and test for images
         that live in /usr/testbed and need special handling.
      2. Make sure that the image directory exists in /proj when taking a
         snapshot of a system image that lives in /usr/testbed since that is
         were we always write new images.
      3. When deleting an image, remove the DB state for the highest numbered
         version if it never came ready/released.
  10. 21 Feb, 2017 1 commit
  11. 13 Feb, 2017 1 commit
  12. 25 Jan, 2017 1 commit
  13. 06 Oct, 2016 1 commit
  14. 26 Sep, 2016 1 commit
  15. 25 Mar, 2016 1 commit
  16. 23 Mar, 2016 1 commit
  17. 27 Jan, 2016 1 commit
  18. 10 Sep, 2015 1 commit
  19. 21 Aug, 2015 1 commit
  20. 10 Aug, 2015 1 commit
  21. 01 Jul, 2015 1 commit
  22. 29 Jun, 2015 1 commit
  23. 24 Jun, 2015 1 commit
  24. 10 Jun, 2015 1 commit
  25. 21 May, 2015 1 commit
    • Mike Hibler's avatar
      Realpath strikes again. · e822de15
      Mike Hibler authored
      Perl realpath returns undef or '' (take your pick) if the path does not
      exist, rendering it pretty much useless for our check. So we run realpath
      on the directory part of the path (which should exist) and then do some
      other paranoid checks on the filename part (no funky chars, cannot be a
  26. 18 May, 2015 1 commit
  27. 15 May, 2015 2 commits
    • Leigh B Stoller's avatar
      Minor bug fix. · f1d1fc0b
      Leigh B Stoller authored
    • Leigh B Stoller's avatar
      Directory based image paths. · 3a21f39e
      Leigh B Stoller authored
      Soon, we will have images with both full images and deltas, for the same
      image version. To make this possible, the image path will now be a
      directory instead of a file, and all of the versions (ndz,sig,sha1,delta)
      files will reside in the directory.
      A new config variable IMAGEDIRECTORIES turns this on, there is also a check
      for the ImageDiretories feature. This is applied only when a brand new
      image is created; a clone version of the image inherits the path it started
      with. Yes, you can have a mix of directory based and file based image
      When it is time to convert all images over, there is a script called
      imagetodir that will go through all image descriptors, create the
      directory, move/rename all the files, and update the descriptors.
      Ultimately, we will not support file based image paths.
      I also added versioning to the image metadata descriptors so that going
      forward, old clients can handle a descriptor from a new server.
  28. 30 Apr, 2015 1 commit
    • Mike Hibler's avatar
      Another skirmish in the on-going do/don't generate relocations battle. · bb01665c
      Mike Hibler authored
      For imagezip. Since I am the founding--and only--member of both camps,
      I get to decide what is right! I am back to generating imagezip relocations
      (for FreeBSD images only) since those images are just a big pile of bit-slag
      if you want to load them anywhere other than their standard location.
  29. 14 Apr, 2015 1 commit
  30. 19 Mar, 2015 1 commit
  31. 16 Mar, 2015 2 commits
  32. 10 Mar, 2015 1 commit
  33. 06 Mar, 2015 1 commit
  34. 05 Mar, 2015 1 commit
  35. 14 Dec, 2014 1 commit