GitLab

1. Split the resource stuff (where we ask for an advertisement and
   process it) into a separate script, since that takes a long time to
   cycle through cause of the size of the ads from the big clusters.

2. On the monitor, distinguish offline (nologins) from actually being
   down.

3. Add a table to store changes in status so we can see over time how
   much time the aggregates are usable.

This is actually implemented in a backend perl script, so you can
do the seach from the command line, but that would be silly, right?

* New tables to store policies for users and projects/groups. At the
  moment, there is only one policy (with associated reason); disabled.
  This allows us to mark projects/groups/users with enable/disable
  flags. Note that policies are applied consecutively, so you can
  disable extensions for a project, but enable them for a user in that
  project.

* Apply extensions when experiments are created, send mail to the audit
  log when policies cause extensions to be disabled.

* New driver script (manage_extensions) to change the policy tables.

Usage: checkprofile [-a | pid,name]
Options:
  -a  : Run rspec2genilib converter on all rspec profiles
  -r  : Run converted geni-lib
  -c  : Compare rspecs after running geni-lib
  -t  : Run RTE check on converted geni-lib script
  -g  : Print the geni-lib
  -x  : Only include rspecs with matching token

Redo the rspectogenilib converter with the goal of supporting both
translation *and* regression testing. A new library is responsible for
taking the output of libXML and creating a data structure representing
the rspec. While this is being done, we look for any constructs or
attributes we cannot handle with geni-lib and report that as an error;
we are not going to convert an rspec unless we can do it correctly. and
completely.

Regression testing is done with another part of this library, that knows
how to compare each element of two rspecs. Basically start at the root
and compare all the way down, failing if the two "parses" of the XML are
not equal at any level.

rspectogenilib now has an option that does regression testing by running
the new geni-lib and comparing the resulting rspec against the original.

On the UI side, there is a new button on existing rspec based
profiles (currently only for admin and studs) called "Convert to
geni-lib" that runs the converter to convert the profile to geni-lib.
The user does not have to accept the new script of course.

However, a converted profile is marked in the database, and the user can
still use Jacks on it, we just run rspectogenilib geni-lib again on the
new rspec. If the user edits the geni-lib, we switch back to normal
geni-lib (clear the flag) when the new version is saved, and Jacks is
once again read-only. This is explained in the UI, and is one of the
things people need to give comment on.

There is also a mode on the Create Profile page for converting new rspec
based profiles to geni-lib, but that is fully turned off for now, we can
get to that later.

We are running another apache server on boss, on port 51369, which
invokes a backend perl script that maps the URL path argument to the
profile, and then calls out to manage_profile to pull from the
repository and update the profile to reflect the new HEAD branch.
Using mod_rewrite in the apache config to restrict URLs to exactly
the one URL that is accepted, modulo the value of the secret token.

I had to refactor a bunch of code in manage_profile to make it easier to
add a new entrypoint for modification from a git repo. This needed to be
done for a long time, I had never cleaned up the original profile
creation code.

On the edit profile web page, there is a new row in the Repository panel
providing the Push URL, and an explanatory help modal.

There is a new slow polling timer that looks for a change to the repo
hash and causes the web page to update in place from the repo, as when a
push hook is invoked and changes the repo.

remove are run on ops via the proxy script. Other smaller operations
like getting the source, logs, commit info are currently run on boss via
NFS to /usr/testbed/opsdir/repositories, which are one per profile.
Lots of work still to do on optimizing repositories.

monitoring system.

New portal_monitor daemon does a GetVersion/ListResources call at each
of the clusters every five minutes, and updates the new table in the
DB called apt_aggregate_status. We calculate free/inuse counts for
physical nodes and a free count for VMs. Failure to contact the
aggregate for more then 10 minutes sets the aggregate as down, since
from our perspective if we cannot get to it, the cluster is down.

Unlike the jfed monitoring system, we are not going to try to
instantiate a new experiment or ssh into it. Wait and see if that is
necessary in our context.

On the instantiate page, generate a json structure for each cluster,
similar the one described in issue #172 by Keith. This way we can easily
switch the existing code over to this new system, but fail back to the
old mechanism if this turn out to be a bust.

Some other related changes to how we hand cluster into the several web
pages.

for all running experiments so that the web interface is updated. Normally
the event stream tells us about changes, but having some problems with
that, so put this in place until I get that figured out.

proof of concept, the NS converter path now uses ns2rspec piped into the
newly added rspec2genlib, which becomes the source code for NS based
experiments that are being converted to Portal experiments.

at all participating clusters. Not finished yet, but good enough to let
people at it.

interface. We do this with the AMV3 PerformOperationalAction call. The
actual updates are invoked from the apt_daemon; after a key change we just
set a flag in the apt_instances table.

Also some unrelated debugging code in APT_Instance.

the alternate pubsubd, and updates the portal experiment as indicates
(slice and sliver status, and imaging status/size).

Portal, what clusters are usable by each portal, ditto datasets. Remove
all hardwired cluster urns and urls.

genilib-iocage uses the FreeBSD "iocage" jail management package to
setup a jail, run the script, and teardown the jail. Unfortunately,
this version is really, really slow (11 seconds for a one-shot jail).

So instead we will use genilib-jail which uses the template jail instance
I built using iocage, but creates the one-off jails by using raw zfs and
jail commands. It runs in about 1.3 seconds. genilib-iocage is left in
case the author speeds it up someday.

N.B. these are NOT plug in replacements for rungenilib.proxy.in.
In particular, the new scripts run as root and don't do any validation
of the caller or arguments. So genilib-jail will be called from rungenilib
for now (though I have not done that part yet!)

from all clusters, and regen the json files. Still need to add live
updating.

NOTE: This is not ready yet, Rob needs to figure out why the labels
are broken.

daemon.

UI to not use modals.

Project leases are now per-group, so we build a sub authority certificate
for a remote dataset so that on the remote side, it is created inside the
group named by the project on the local side.

Many bug fixes.

used to manage local blockstores (datasets). A cheesy iframe hack is used
to embed the APT pages (slick new interface) inside the antique Emulab
interface.

Currently the only hooks from the admin drop down menu, since there is some
additional work before we let users see it.

allows the user to create a new disk image in place, instead of having
to create a new profile (Clone).

* Add a .htaccess file that does the rewrites, instead of in the httpd
  confile file. Added Rob's stuff for rewriting urls to hide the .php
  although not sure this is working correctly yet.

* Add simple MyExperiments page so that logged in users can find their
  way back to running profiles.

* Move the DB table holding the running experiment records from the
  geni-sa DB into the main Emulab DB. Lots of little changes for that.

* Change logout to plain link instead of ajax call. That was a silly
  thing to do.

* Bug fixes to ssh keys and shell login from the status page.

and libraries. Rough, needs plenty more work.

This commit is intended to makes the license status of Emulab and
ProtoGENI source files more clear.  It replaces license symbols like
"EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
blocks that contain actual license statements.

This change was driven by the fact that today, most people acquire and
track Emulab and ProtoGENI sources via git.

Before the Emulab source code was kept in git, the Flux Research Group
at the University of Utah would roll distributions by making tar
files.  As part of that process, the Flux Group would replace the
license symbols in the source files with actual license statements.

When the Flux Group moved to git, people outside of the group started
to see the source files with the "unexpanded" symbols.  This meant
that people acquired source files without actual license statements in
them.  All the relevant files had Utah *copyright* statements in them,
but without the expanded *license* statements, the licensing status of
the source files was unclear.

This commit is intended to clear up that confusion.

Most Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the Affero GNU General Public License, version 3
(AGPLv3).

Most Utah-copyrighted files related to ProtoGENI are distributed under
the terms of the GENI Public License, which is a BSD-like open-source
license.

Some Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the GNU Lesser General Public License, version 2.1
(LGPL).

checking.

     www/newimageid_ez.php3 - The reworked PHP page.  Calls Image::NewImageId with ez=1.
     www/newimageid.php3 - Call Image::NewImageId with ez=0.
     www/imageid_defs.php - Re-use the Image::NewImageId class method, adding an 'ez' arg.
     backend/{newimageid_ez,GNUmakefile}.in configure configure.in - New backend script.
         After checks, calls OSinfo->Create and Image->Create with the same XML args array.
     db/Image.pm.in - Re-use the Image->Create method, adding an imageid over-ride arg.
     db/OSinfo.pm.in - Filter out extraneous db slot args from XML in the Create method.
     db/libdb.pm.in - Add TB_{OS,IMAGE}ID_* constants from dbdefs.php3 .
     sql/database-fill.sql - Add OS entries to the table_regex 'images' pattern set.

     www/editgroup.php3 - The reworked PHP page.
     www/editgroup_form.php3 - Removed, form merged into editgroup.php3 .
     www/showgroup.php3 - Link to editgroup.php3, rather than editgroup_form.php3 .
     www/group_defs.php - Add an Image::EditGroup class method
                            bridging to the script via XML.
     backend/{editgroup,GNUmakefile}.in configure configure.in - New backend script.
     db/Group.pm.in - Add an EditGroup worker class method for script arg checking.
                      Also the missing NonMemberList and CheckTrustConsistency methods,
                      and a GETTRUST flag to MemberList, as in the PHP version.
     db/User.pm.in - Add the missing but tiny {Set,Get}TempData methods.
     sql/database-fill.sql - Add gid_idx to the table_regex 'groups' checking patterns.

     www/newimageid.php3 - The reworked PHP page.
     www/imageid_defs.php - Add an Image::NewImageId class method
                            bridging to the script via XML.
     backend/{newimageid,GNUmakefile}.in configure configure.in - New backend script.
     db/Image.pm.in - Add a Create class method for script arg checking.
     sql/database-fill.sql - Add to the table_regex 'images' checking patterns.