- 04 Jun, 2018 2 commits
-
-
David Johnson authored
The docker VM server-side goo is mostly identical to Xen, with slightly different handling for parent images. We also support loading external Docker images (i.e. those without a real imageid in our DB; in that case, user has to set a specific stub image, and some extra per-vnode metadata (a URI that points to a Docker registry/image repo/tag); the Docker clientside handles the rest. Emulab Docker images map to a Emulab imageid:version pretty seamlessly. For instance, the Emulab `emulab-ops/docker-foo-bar:1` image would map to `<local-registry-URI>/emulab-ops/emulab-ops/docker-foo-bar:1`; the mapping is `<local-registry-URI>/pid/gid/imagename:version`. Docker repository names are lowercase-only, so we handle that for the user; but I would prefer that users use lowercase Emulab imagenames for all Docker images; that will help us. That is not enforced in the code; it will appear in the documentation, and we'll see. Full Docker imaging relies on several other libraries (https://gitlab.flux.utah.edu/emulab/pydockerauth, https://gitlab.flux.utah.edu/emulab/docker-registry-py). Each Emulab-based cluster must currently run its own private registry to support image loading/capture (note however that if capture is unnecessary, users can use the external images path instead). The pydockerauth library is a JWT token server that runs out of boss's Apache and implements authn/authz for the per-Emulab Docker registry (probably running on ops, but could be anywhere) that stores images and arbitrates upload/download access. For instance, nodes in an experiment securely pull images using their pid/eid eventkey; and the pydockerauth emulab authz module knows what images the node is allowed to pull (i.e. sched_reloads, the current image the node is running, etc). Real users can also pull images via user/pass, or bogus user/pass + Emulab SSL cert. GENI credential-based authn/z was way too much work, sadly. There are other auth/z paths (i.e. for admins, temp tokens for secure operations) as well. As far as Docker image distribution in the federation, we use the same model as for regular ndz images. Remote images are pulled in to the local cluster's Docker registry on-demand from their source cluster via admin token auth (note that all clusters in the federation have read-only access to the entire registries of any other cluster in the federation, so they can pull images). Emulab imageid handling is the same as the existing ndz case. For instance, image versions are lazily imported, on-demand; local version numbers may not match the remote image source cluster's version numbers. This will potentially be a bigger problem in the Docker universe; Docker users expect to be able to reference any image version at any time anywhere. But that is of course handleable with some ex post facto synchronization flag day, at least for the Docker images. The big new thing supporting native Docker image usage is the guts of a refactor of the utils/image* scripts into a new library, libimageops; this is necessary to support Docker images, which are stored in their own registry using their own custom protocols, so not amenable to our file-based storage. Note: the utils/image* scripts currently call out to libimageops *only if* the image format is docker; all other images continue on the old paths in utils/image*, which all still remain intact, or minorly-changed to support libimageops. libimageops->New is the factory-style mechanism to get a libimageops that works for your image format or node type. Once you have a libimageops instance, you can invoke normal image logical operations (CreateImage, ImageValidate, ImageRelease, et al). I didn't do every single operation (for instance, I haven't yet dealt with image_import beyond essentially generalizing DownLoadImage by image format). Finally, each libimageops is stateless; another design would have been some statefulness for more complicated operations. You will see that CreateImage, for instance, is written in a helper-subclass style that blurs some statefulness; however, it was the best match for the existing body of code. We can revisit that later if the current argument-passing convention isn't loved. There are a couple outstanding issues. Part of the security model here is that some utils/image* scripts are setuid, so direct libimageops library calls are not possible from a non-setuid context for some operations. This is non-trivial to resolve, and might not be worthwhile to resolve any time soon. Also, some of the scripts write meaningful, traditional content to stdout/stderr, and this creates a tension for direct library calls that is not entirely resolved yet. Not hard, just only partly resolved. Note that tbsetup/libimageops_ndz.pm.in is still incomplete; it needs imagevalidate support. Thus, I have not even featurized this yet; I will get to that as I have cycles.
-
Leigh Stoller authored
-
- 01 Jun, 2018 1 commit
-
-
Leigh Stoller authored
-
- 31 May, 2018 2 commits
-
-
Leigh Stoller authored
users on the local cluster).
-
Leigh Stoller authored
expire, now that we store that info in the history table.
-
- 30 May, 2018 9 commits
-
-
Leigh Stoller authored
-
Leigh Stoller authored
-
Leigh Stoller authored
-
Leigh Stoller authored
1. Return current set of reservations (if any) for a user when getting the max extension (piggy backing on the call to reduce overhead). 2. Add RPC to get the reservation history for a user (all past reservations that were approved). Aside; the reservation_history table was not being updated properly, only expired reservations were saved, not deleted (but used) reservations, so we lost a lot of history. We could regen some of it from the history tables I added at the Portal for Dmitry, but not sure it is worth the trouble. 3. And then the main content of this commit is that for both of the lists above, also return the experiment usage history for the project an dthe user who created the reservation. This takes the form of a time line of allocation changes so that we can graph node usage against the reservation bounds, to show graphically how well utilized the reservation is.
-
Leigh Stoller authored
within a project with reservations active during an experiment.
-
Leigh Stoller authored
-
Leigh Stoller authored
-
Leigh Stoller authored
-
Leigh Stoller authored
need uid_idx and pid_idx. Also store the cancel date. Fix bug; we were not storing a history entry for deleted reservations, so we lost all those entries even though they were reservations that were used.
-
- 23 May, 2018 1 commit
-
-
Leigh Stoller authored
-
- 15 May, 2018 1 commit
-
-
Leigh Stoller authored
-
- 10 May, 2018 1 commit
-
-
Leigh Stoller authored
-
- 27 Apr, 2018 1 commit
-
-
David Johnson authored
-
- 26 Apr, 2018 1 commit
-
-
Leigh Stoller authored
-
- 25 Apr, 2018 1 commit
-
-
David Johnson authored
There was a period where we didn't generate unencrypted SSL certs for nonlocal users; create them now (and generally create any missing unencrypted/encrypted cert for active users). Also, now renew certs for nonlocal users.
-
- 17 Apr, 2018 1 commit
-
-
Leigh Stoller authored
-
- 13 Apr, 2018 1 commit
-
-
Leigh Stoller authored
physical state stuff, since we do call those directly on the Geni path.
-
- 03 Apr, 2018 1 commit
-
-
Leigh Stoller authored
there is an adminmfs, instead of defaulting to null. The ONIE based switches operate more like normal nodes wrt reload/admin MFSs
-
- 02 Apr, 2018 2 commits
-
-
Leigh Stoller authored
-
Gary Wong authored
(If the reservation being approved isn't in the list returned by LookupAll(), presumably because it was previously pending, then append it during feasibility checking.)
-
- 29 Mar, 2018 2 commits
-
-
Mike Hibler authored
-
Leigh Stoller authored
1) Rework so that instead of relying on swapin__last + autoswap timeout, set expt_expires for classic experiments at the beginning of swapin time. This is cause swapin_last is not set till the end of swapin, and so during swapin the res system is in an inconsistent state since there is no way to determine when the experiment ends. 2) On the Geni path, simplify expiration handling; do not allow a slice modification and expiration change at the same time; the bookkeeping and failure rollback is a pain, especially wrt reservation system, and this rarely ever actually happens, so get rid of a lot of complication.
-
- 28 Mar, 2018 1 commit
-
-
Mike Hibler authored
-
- 19 Mar, 2018 1 commit
-
-
Leigh Stoller authored
-
- 08 Mar, 2018 1 commit
-
-
Leigh Stoller authored
-
- 16 Feb, 2018 1 commit
-
-
Leigh Stoller authored
-
- 09 Feb, 2018 2 commits
-
-
Leigh Stoller authored
going to leave that state, and it causes the Portal to paint then in yellow since the are not "ready". A better approach would be to move nodes out of updating_users after some amount of time has passed. Maybe on the next trip through this part of the forest.
-
Leigh Stoller authored
images and do not send isalive. This stuff is pretty old and crufty! Remove a bunch of obsolete code while I was here.
-
- 05 Feb, 2018 1 commit
-
-
Leigh Stoller authored
We never use this stuff, we should kill it.
-
- 02 Feb, 2018 1 commit
-
-
Leigh Stoller authored
for experiments with nodes reserved that have never been swapped in. Happens in the emulab-ops project, we nalloc nodes to experiments that like hwdown.
-
- 30 Jan, 2018 3 commits
-
-
Gary Wong authored
-
Mike Hibler authored
Previously, we just added to the old end_time which could result in a time that was still in the past!
-
Gary Wong authored
-
- 22 Jan, 2018 1 commit
-
-
Leigh Stoller authored
being used by the cached objects.
-
- 16 Jan, 2018 1 commit
-
-
Leigh Stoller authored
-