1. 04 Jun, 2018 1 commit
    • David Johnson's avatar
      Docker server-side core, esp new libimageops support for Docker images. · 66366489
      David Johnson authored
      The docker VM server-side goo is mostly identical to Xen, with slightly
      different handling for parent images.  We also support loading external
      Docker images (i.e. those without a real imageid in our DB; in that
      case, user has to set a specific stub image, and some extra per-vnode
      metadata (a URI that points to a Docker registry/image repo/tag);
      the Docker clientside handles the rest.
      
      Emulab Docker images map to a Emulab imageid:version pretty seamlessly.
      For instance, the Emulab `emulab-ops/docker-foo-bar:1` image would map
      to `<local-registry-URI>/emulab-ops/emulab-ops/docker-foo-bar:1`; the
      mapping is `<local-registry-URI>/pid/gid/imagename:version`.  Docker
      repository names are lowercase-only, so we handle that for the user; but
      I would prefer that users use lowercase Emulab imagenames for all Docker
      images; that will help us.  That is not enforced in the code; it will
      appear in the documentation, and we'll see.
      
      Full Docker imaging relies on several other libraries
      (https://gitlab.flux.utah.edu/emulab/pydockerauth,
      https://gitlab.flux.utah.edu/emulab/docker-registry-py).  Each
      Emulab-based cluster must currently run its own private registry to
      support image loading/capture (note however that if capture is
      unnecessary, users can use the external images path instead).  The
      pydockerauth library is a JWT token server that runs out of boss's
      Apache and implements authn/authz for the per-Emulab Docker registry
      (probably running on ops, but could be anywhere) that stores images and
      arbitrates upload/download access.  For instance, nodes in an experiment
      securely pull images using their pid/eid eventkey; and the pydockerauth
      emulab authz module knows what images the node is allowed to pull
      (i.e. sched_reloads, the current image the node is running, etc).  Real
      users can also pull images via user/pass, or bogus user/pass + Emulab
      SSL cert.  GENI credential-based authn/z was way too much work, sadly.
      There are other auth/z paths (i.e. for admins, temp tokens for secure
      operations) as well.
      
      As far as Docker image distribution in the federation, we use the same
      model as for regular ndz images.  Remote images are pulled in to the
      local cluster's Docker registry on-demand from their source cluster via
      admin token auth (note that all clusters in the federation have
      read-only access to the entire registries of any other cluster in the
      federation, so they can pull images).  Emulab imageid handling is the
      same as the existing ndz case.  For instance, image versions are lazily
      imported, on-demand; local version numbers may not match the remote
      image source cluster's version numbers.  This will potentially be a
      bigger problem in the Docker universe; Docker users expect to be able to
      reference any image version at any time anywhere.  But that is of course
      handleable with some ex post facto synchronization flag day, at least
      for the Docker images.
      
      The big new thing supporting native Docker image usage is the guts of a
      refactor of the utils/image* scripts into a new library, libimageops;
      this is necessary to support Docker images, which are stored in their
      own registry using their own custom protocols, so not amenable to our
      file-based storage.  Note: the utils/image* scripts currently call out
      to libimageops *only if* the image format is docker; all other images
      continue on the old paths in utils/image*, which all still remain
      intact, or minorly-changed to support libimageops.
      
      libimageops->New is the factory-style mechanism to get a libimageops
      that works for your image format or node type.  Once you have a
      libimageops instance, you can invoke normal image logical operations
      (CreateImage, ImageValidate, ImageRelease, et al).  I didn't do every
      single operation (for instance, I haven't yet dealt with image_import
      beyond essentially generalizing DownLoadImage by image format).
      Finally, each libimageops is stateless; another design would have been
      some statefulness for more complicated operations.   You will see that
      CreateImage, for instance, is written in a helper-subclass style that
      blurs some statefulness; however, it was the best match for the existing
      body of code.  We can revisit that later if the current argument-passing
      convention isn't loved.
      
      There are a couple outstanding issues.  Part of the security model here
      is that some utils/image* scripts are setuid, so direct libimageops
      library calls are not possible from a non-setuid context for some
      operations.  This is non-trivial to resolve, and might not be worthwhile
      to resolve any time soon.  Also, some of the scripts write meaningful,
      traditional content to stdout/stderr, and this creates a tension for
      direct library calls that is not entirely resolved yet.  Not hard, just
      only partly resolved.
      
      Note that tbsetup/libimageops_ndz.pm.in is still incomplete; it needs
      imagevalidate support.  Thus, I have not even featurized this yet; I
      will get to that as I have cycles.
      66366489
  2. 11 Oct, 2017 1 commit
  3. 06 Jul, 2017 1 commit
  4. 20 Jun, 2017 1 commit
    • Leigh Stoller's avatar
      Work on firewall support: · 72fa735e
      Leigh Stoller authored
      1. Get this working on the NS conversion path.
      
      2. Add support for additional firewall rules along the path,
         the CM never had support for firewall rules.
      
      3. Set the security_level to zapdisk when firewalling is on.
      72fa735e
  5. 15 Jun, 2017 1 commit
  6. 09 Jun, 2017 2 commits
  7. 30 May, 2017 2 commits
  8. 23 May, 2017 1 commit
  9. 16 May, 2017 2 commits
  10. 28 Apr, 2017 1 commit
    • Leigh Stoller's avatar
      Various converter changes; · 668fd8de
      Leigh Stoller authored
      1. Add adb_target and rwclone support.
      
      2. Name the link/node variables by the user client_id with dashes
         converted to underscore, and prefix with node_/link_ if the user name
         does not already start with node/link.
      
      3. Add "permissive" mode to ignore profile_parameters, data_set, and the
         routable_control_ip attribute so that we can test more profiles.
      
      4. Add options to run just the geni-lib based profiles through
         regression testing.
      668fd8de
  11. 26 Apr, 2017 1 commit
    • Leigh Stoller's avatar
      Various changes: · 4dfe954b
      Leigh Stoller authored
      1. Support for Tour steps. See associated change in geni-lib.
      
      2. Add regression testing of the Tour. I had left that out, now we are
         doing that.
      
      3. Add option to rspec2genilib to not add the stub Tour description,
         since that makes regression testing difficult.
      4dfe954b
  12. 24 Apr, 2017 2 commits
  13. 17 Apr, 2017 1 commit
    • Leigh Stoller's avatar
      rspecs are so passe ... · 7c38809d
      Leigh Stoller authored
      Redo the rspectogenilib converter with the goal of supporting both
      translation *and* regression testing. A new library is responsible for
      taking the output of libXML and creating a data structure representing
      the rspec. While this is being done, we look for any constructs or
      attributes we cannot handle with geni-lib and report that as an error;
      we are not going to convert an rspec unless we can do it correctly. and
      completely.
      
      Regression testing is done with another part of this library, that knows
      how to compare each element of two rspecs. Basically start at the root
      and compare all the way down, failing if the two "parses" of the XML are
      not equal at any level.
      
      rspectogenilib now has an option that does regression testing by running
      the new geni-lib and comparing the resulting rspec against the original.
      
      On the UI side, there is a new button on existing rspec based
      profiles (currently only for admin and studs) called "Convert to
      geni-lib" that runs the converter to convert the profile to geni-lib.
      The user does not have to accept the new script of course.
      
      However, a converted profile is marked in the database, and the user can
      still use Jacks on it, we just run rspectogenilib geni-lib again on the
      new rspec. If the user edits the geni-lib, we switch back to normal
      geni-lib (clear the flag) when the new version is saved, and Jacks is
      once again read-only. This is explained in the UI, and is one of the
      things people need to give comment on.
      
      There is also a mode on the Create Profile page for converting new rspec
      based profiles to geni-lib, but that is fully turned off for now, we can
      get to that later.
      7c38809d
  14. 13 Sep, 2016 1 commit
  15. 12 Sep, 2016 1 commit
  16. 09 Aug, 2016 1 commit
  17. 21 Jul, 2016 1 commit
  18. 05 Jul, 2016 1 commit
  19. 10 Jun, 2016 1 commit
  20. 25 May, 2016 1 commit
  21. 20 May, 2016 1 commit
  22. 18 May, 2016 1 commit
  23. 11 May, 2016 3 commits