1. 04 Jun, 2019 1 commit
  2. 15 Apr, 2019 1 commit
  3. 07 Nov, 2018 1 commit
  4. 05 Nov, 2018 1 commit
    • Leigh Stoller's avatar
      Changes to how we handle/report mapping failures that also fail the · 11074445
      Leigh Stoller authored
      empty testbed test.
      
      Prior to this commit, we were not invoking the empty testbed case
      consitently. Now we do, but that exposed another problem; reporting that
      to the error to the Portal in a meaningful way. Basically, we can report
      a different error code for an impossible to map error, but then we lose
      the info we store now about what the actual failure was (which we show
      to the user with additional helpful info). Since we cannot (easily)
      change the Geni API for CreateSliver(), I have elected to continue the
      practice of returning the specific error codes (which also go into the
      database for long term historical info), and add more helpful text that
      for the Portal user that explains clearly that the mapping is impossible
      on the target cluster. This extra text also go into the database in the
      attached message field, so we ccan come back later and post process if
      we decide to do something different.
      11074445
  5. 30 Jul, 2018 1 commit
  6. 04 Jun, 2018 1 commit
    • David Johnson's avatar
      Docker server-side core, esp new libimageops support for Docker images. · 66366489
      David Johnson authored
      The docker VM server-side goo is mostly identical to Xen, with slightly
      different handling for parent images.  We also support loading external
      Docker images (i.e. those without a real imageid in our DB; in that
      case, user has to set a specific stub image, and some extra per-vnode
      metadata (a URI that points to a Docker registry/image repo/tag);
      the Docker clientside handles the rest.
      
      Emulab Docker images map to a Emulab imageid:version pretty seamlessly.
      For instance, the Emulab `emulab-ops/docker-foo-bar:1` image would map
      to `<local-registry-URI>/emulab-ops/emulab-ops/docker-foo-bar:1`; the
      mapping is `<local-registry-URI>/pid/gid/imagename:version`.  Docker
      repository names are lowercase-only, so we handle that for the user; but
      I would prefer that users use lowercase Emulab imagenames for all Docker
      images; that will help us.  That is not enforced in the code; it will
      appear in the documentation, and we'll see.
      
      Full Docker imaging relies on several other libraries
      (https://gitlab.flux.utah.edu/emulab/pydockerauth,
      https://gitlab.flux.utah.edu/emulab/docker-registry-py).  Each
      Emulab-based cluster must currently run its own private registry to
      support image loading/capture (note however that if capture is
      unnecessary, users can use the external images path instead).  The
      pydockerauth library is a JWT token server that runs out of boss's
      Apache and implements authn/authz for the per-Emulab Docker registry
      (probably running on ops, but could be anywhere) that stores images and
      arbitrates upload/download access.  For instance, nodes in an experiment
      securely pull images using their pid/eid eventkey; and the pydockerauth
      emulab authz module knows what images the node is allowed to pull
      (i.e. sched_reloads, the current image the node is running, etc).  Real
      users can also pull images via user/pass, or bogus user/pass + Emulab
      SSL cert.  GENI credential-based authn/z was way too much work, sadly.
      There are other auth/z paths (i.e. for admins, temp tokens for secure
      operations) as well.
      
      As far as Docker image distribution in the federation, we use the same
      model as for regular ndz images.  Remote images are pulled in to the
      local cluster's Docker registry on-demand from their source cluster via
      admin token auth (note that all clusters in the federation have
      read-only access to the entire registries of any other cluster in the
      federation, so they can pull images).  Emulab imageid handling is the
      same as the existing ndz case.  For instance, image versions are lazily
      imported, on-demand; local version numbers may not match the remote
      image source cluster's version numbers.  This will potentially be a
      bigger problem in the Docker universe; Docker users expect to be able to
      reference any image version at any time anywhere.  But that is of course
      handleable with some ex post facto synchronization flag day, at least
      for the Docker images.
      
      The big new thing supporting native Docker image usage is the guts of a
      refactor of the utils/image* scripts into a new library, libimageops;
      this is necessary to support Docker images, which are stored in their
      own registry using their own custom protocols, so not amenable to our
      file-based storage.  Note: the utils/image* scripts currently call out
      to libimageops *only if* the image format is docker; all other images
      continue on the old paths in utils/image*, which all still remain
      intact, or minorly-changed to support libimageops.
      
      libimageops->New is the factory-style mechanism to get a libimageops
      that works for your image format or node type.  Once you have a
      libimageops instance, you can invoke normal image logical operations
      (CreateImage, ImageValidate, ImageRelease, et al).  I didn't do every
      single operation (for instance, I haven't yet dealt with image_import
      beyond essentially generalizing DownLoadImage by image format).
      Finally, each libimageops is stateless; another design would have been
      some statefulness for more complicated operations.   You will see that
      CreateImage, for instance, is written in a helper-subclass style that
      blurs some statefulness; however, it was the best match for the existing
      body of code.  We can revisit that later if the current argument-passing
      convention isn't loved.
      
      There are a couple outstanding issues.  Part of the security model here
      is that some utils/image* scripts are setuid, so direct libimageops
      library calls are not possible from a non-setuid context for some
      operations.  This is non-trivial to resolve, and might not be worthwhile
      to resolve any time soon.  Also, some of the scripts write meaningful,
      traditional content to stdout/stderr, and this creates a tension for
      direct library calls that is not entirely resolved yet.  Not hard, just
      only partly resolved.
      
      Note that tbsetup/libimageops_ndz.pm.in is still incomplete; it needs
      imagevalidate support.  Thus, I have not even featurized this yet; I
      will get to that as I have cycles.
      66366489
  7. 13 Apr, 2018 1 commit
    • Leigh Stoller's avatar
      A fix to the alloc state machine; on the Geni Path we do two mapper · 4acd0bfe
      Leigh Stoller authored
      runs, the first in "alloconly" mode to nalloc resources, the second to
      "commit" the actual physical state changes. Well when we do that we end
      up changing newly allocated nodes to node that are dirty. This turns out
      to be bad for the switch ossetup code, and since the above is wrong, I
      decided to fix it instead of change David's switch ossetup code.
      4acd0bfe
  8. 03 May, 2017 1 commit
  9. 20 Jan, 2017 1 commit
  10. 08 Dec, 2016 1 commit
  11. 11 Nov, 2016 1 commit
  12. 20 Sep, 2016 1 commit
    • Mike Hibler's avatar
      Initial support for ephemeral RW clones of persistent blockstores. · f98ab0e5
      Mike Hibler authored
      Using "set-rwclone" ala:
      
          set $bsobj [$ns blockstore]
          $bsobj set-lease "emulab-ops/bar"
          $bsobj set-node $node
          $bsobj set-rwclone 1
          ...
      
      in your NS file will create a clone of the indicated persistent blockstore.
      
      Somewhat limited in utility since you can only have one clone of a
      particular blockstore per experiment.
      f98ab0e5
  13. 10 Nov, 2015 1 commit
  14. 05 Nov, 2015 1 commit
  15. 05 Oct, 2015 1 commit
  16. 25 Sep, 2015 1 commit
  17. 06 May, 2015 1 commit
  18. 27 Jan, 2015 1 commit
    • Leigh Stoller's avatar
      Two co-mingled sets of changes: · 85cb063b
      Leigh Stoller authored
      1) Implement the latest dataset read/write access settings from frontend to
         backend. Also updates for simultaneous read-only usage.
      
      2) New configure options: PROTOGENI_LOCALUSER and PROTOGENI_GENIWEBLOGIN.
      
         The first changes the way that projects and users are treated at the
         CM. When set, we create real accounts (marked as nonlocal) for users and
         also create real projects (also marked as nonlocal). Users are added to
         those projects according to their credentials. The underlying experiment
         is thus owned by the user and in the project, although all the work is
         still done by the geniuser pseudo user. The advantage of this approach
         is that we can use standard emulab access checks to control access to
         objects like datasets. Maybe images too at some point.
      
         NOTE: Users are not removed from projects once they are added; we are
         going to need to deal with this, perhaps by adding an expiration stamp
         to the groups_membership tables, and using the credential expiration to
         mark it.
      
         The second new configure option turns on the web login via the geni
         trusted signer. So, if I create a sliver on a backend cluster when both
         options are set, I can use the trusted signer to log into my newly
         created account on the cluster, and see it (via the emulab classic web
         interface).
      
         All this is in flux, might end up being a bogus approach in the end.
      85cb063b
  19. 11 Dec, 2013 2 commits
    • Mike Hibler's avatar
    • Mike Hibler's avatar
      Mapping changes to support persistent blockstores. · a721e561
      Mike Hibler authored
      Quite simple actually. In ptopgen we add all persistent stores, which
      are identified by having a non-zero lease_idx. To ensure they are only
      matched up with the correct virt_blockstore, we add a feature "bs-lease-NN"
      where NN is the unique lease index with weight 1.0.
      
      In vtopgen, when a request is made to map a persistent store, we first
      verify that the experiment pid matches that of the blockstore's lease
      (this is a fer-now simple check which is actually redundant as we make
      this check in the parser too) and ensure that the blockstore is not already
      mapped (indicated by the blockstore_state entry having size==0) and then
      we emit an entry with desire "bs-lease-NN" and weight 1.0.
      
      Most of the file changes were actually to get the "preassign" flag to
      vtopgen forwarded into the various libvtop modules so that I can check it
      and not perform the permission/size check if it is set.
      a721e561
  20. 26 Aug, 2013 2 commits
  21. 04 Jun, 2013 1 commit
  22. 30 Apr, 2013 1 commit
  23. 02 Apr, 2013 1 commit
  24. 01 Mar, 2013 1 commit
  25. 10 Jan, 2013 1 commit
  26. 24 Sep, 2012 1 commit
    • Eric Eide's avatar
      Replace license symbols with {{{ }}}-enclosed license blocks. · 6df609a9
      Eric Eide authored
      This commit is intended to makes the license status of Emulab and
      ProtoGENI source files more clear.  It replaces license symbols like
      "EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
      blocks that contain actual license statements.
      
      This change was driven by the fact that today, most people acquire and
      track Emulab and ProtoGENI sources via git.
      
      Before the Emulab source code was kept in git, the Flux Research Group
      at the University of Utah would roll distributions by making tar
      files.  As part of that process, the Flux Group would replace the
      license symbols in the source files with actual license statements.
      
      When the Flux Group moved to git, people outside of the group started
      to see the source files with the "unexpanded" symbols.  This meant
      that people acquired source files without actual license statements in
      them.  All the relevant files had Utah *copyright* statements in them,
      but without the expanded *license* statements, the licensing status of
      the source files was unclear.
      
      This commit is intended to clear up that confusion.
      
      Most Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the Affero GNU General Public License, version 3
      (AGPLv3).
      
      Most Utah-copyrighted files related to ProtoGENI are distributed under
      the terms of the GENI Public License, which is a BSD-like open-source
      license.
      
      Some Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the GNU Lesser General Public License, version 2.1
      (LGPL).
      6df609a9
  27. 02 Jul, 2012 1 commit
  28. 16 May, 2012 1 commit
    • Leigh Stoller's avatar
      Another protogeni checkbox; scriptify and simplify adding "special" · cf517af6
      Leigh Stoller authored
      devices with network interfaces. Emulab's spp and bbg nodes are
      examples, but I did all that by hand inserting sql. An spp node is a
      shared node with some interfaces. Users can allocate one or more of
      those interfaces and establish vlans to the interfaces. The node is a
      "fakenode" in "shared" mode, and everything else falls out. The mapper
      assigns virtual nodes until all of the interfaces are allocated,
      snmpit does its work on the interfaces, and the user then does the
      rest.
      
      Anyway, to added a special device:
      
        boss> wap addspecialdevice -s -t goober goober1
      
      The -t argument is the name of the node type, created if it does not
      exists. The last argument is the name of the fakenode to create in the
      DB. The -s option says the special device is shared. Without -s, the
      device is allocated exclusively.
      
      Then to add interfaces to the device:
      
        boss> wap addspecialiface -b 1Gb -s cisco4,100,100 goober1 eth0
      
      The -b option is the speed (either 100Mb or 1Gb). The -s option is the
      switch side of the interface (switchname,card,port). The last two
      arguments are the nodename and iface name for the interfaces table.
      
      After the interface and wires table entry are added to the DB, snmpit
      is called to put the switch port into tagged mode (if the node is
      shared). To skip the snmpit step, add the -t option.
      cf517af6
  29. 21 Apr, 2012 1 commit
  30. 23 Jan, 2012 1 commit
  31. 02 Dec, 2011 1 commit
    • Leigh Stoller's avatar
      Minor changes so that we can allow the subos map to be used for · fb5fb526
      Leigh Stoller authored
      determining the jailosid for "static" images like OPENVZ-STD. For
      example:
      
      	tb-set-hardware $n1 d710-vm
      	tb-set-node-os $n1 OPENVZ-STD FEDORA15-64-OVZ-STD
      
      The main point is that OPENVZ-STD is a placeholder with no associated
      image. It exists to tell the mapper and osload what to do. This is
      accomplished with nextosid. The new approach makes things consistent
      with subos stuff.
      fb5fb526
  32. 30 Nov, 2011 1 commit
  33. 07 Nov, 2011 1 commit
  34. 30 Oct, 2011 1 commit
  35. 13 Oct, 2011 1 commit
  36. 10 Oct, 2011 1 commit
  37. 08 Oct, 2011 2 commits
    • Mike Hibler's avatar
      Revert "Adjust the set of unix gids used for a download server." · baba7478
      Mike Hibler authored
      This reverts commit fc89eb38.
      
      Checked in a bunch of crap that was unrelated.
      baba7478
    • Mike Hibler's avatar
      Adjust the set of unix gids used for a download server. · fc89eb38
      Mike Hibler authored
      When downloading an image, start the frisbeed process with the minimum set of
      gids necessary to access the image. This includes the unix gid of the
      project that the image is in and, optionally, the unix gid of the project
      subgroup if the image is part of one.
      
      Previously, we just use the gid set of the uid of the swapper of the
      experiment. Not only was this excessive, but it might also not include the
      gids needed in the case of a "global" image that is not in the world-readable
      /usr/testbed/images directory.
      fc89eb38