GitLab

process that hands off).

* Add portal url to the existing emulab extension that tells the CM the
  CreateSliver() is coming from the Portal. Always send this info, not
  just for the Emulab Portal.

* Stash that info in the geni slice data structure so we can add links
  back to the portal status page for current slices.

* Add routines to generate a portal URL for the history entries, since
  we will not have those links for historical slices. Add links back to
  the portal on the showslice and slice history pages.

compat with existing code since the stringify method just returns the
URN. Over time, this will be made more explicit. See previous commit for
how it interacts with XMLRPC and JSON encoding.

control network port modification), and add front end support to the Portal
status page (admin mode only of course)

and always import into the GeniSlices project. Previously, images were
being imported into the project of the slice experiment, by the geniuser.

When PROTOGENI_LOCALUSER is turned off, this change does not affect
anything, since it is still geniuser doing the import, and all imported
images are consider global and thus cross-project usable. So where we stick
the image is not really important, but putting all geni imported images in
one place is more convenient (sure makes it easier to find them). But more
important, this change is backwards compatible with existing imports. 

Later, if the source image is updated, and a new user (in another project)
uses that image, the update (pulling the updated image scross) is done by
geniuser (who is the leader of all geni holding projects), who has write
access to the image whatever project it is in. 

What about when PROTOGENI_LOCALUSER is turned on? There are actually two
sub cases here.

1. The user is using an aggregate in a different domain then their SA. Say,
   when a Cloudlab Portal user is creating an experiment at the Clemson
   cluster (which has PROTOGENI_LOCALUSER=1). In this case, clemson does
   not know anything about the user anyway, and so its pretty much like the
   case described above since everything is done by the geniuser in holding
   projects owned by the geniuser.

2. The user is using the same aggregate as their SA. Say, when a Cloudlab
   Portal user is creating an experiment at the Emulab cluster. In this
   case Emulab knows the user and project, and everything is done as that
   user in the actual project (there is no geni holding project).

   If we import the image into that project as the actual user, we are okay
   at first; as above, all images are global and cross-project, so anyone
   can use it. But what if the source image changes and then a different
   user in a different project tries to use it? The backend is going to try
   to import the new version, but that fails cause the current user does
   not have write access to the image.

   Hence the real reason for this change; if always import into GeniSlices
   as geniuser, we do not get into this permission problem.

private key.

The goal is to distribute an experiment wide certificate and private
key. At the moment this is just a self signed x509 certificate and the
accompanying rsa key. In PEM format. The same cert/key will be distributed
across multiple aggregates.

An openssh key pair can be trivially derived from the private key. Or the
public part can be derived from the certificate. A quick google will show
show.

Initially, you will need to run tmcc directly to get them, using the
geni_certificate and geni_key commands.

picking pieces out, so I added this:

  my $foo = GeniHRN->new("urn:publicid:IDN+emulab.net:testbed+slice+myexp3");

and provides:

  $foo->domain();
  $foo->type();
  $foo->id();
  $foo->urn();

and has a stringify method that returns the urn, so that existing code all
works.

The problem though is the XML::RPC and Frontier libraries provide no hooks
to catch this, and rather then using a stringification, they both convert
all blessed references into structs, and so anyplace that puts a urn into
something to go out on the wire, has to be changed to force to the string.

Damn, how disappointing! So all the code is here but basically disabled
until I find time to go through all the code.

experiments. Lockdown is implemented at the portal and at the backend
cluster (ignore by cm and sa daemons). At present, the only policy is
that admin extensions longer then 14 days, set the lockdown.

CM V2 (and thus the AM) now accept a type=speaksfor credential along
with regular credentials. When supplied, the speaksfor caller must be
equal to the owner of the speaksfor credential and the target must be
equal to the owner of the regular credential(s). All operations take
place in the context of the spokenfor user.

Added speaksfor slots to geni_slices,geni_aggregates and geni_tickets.
Also to the history table. But these are just the most recent data.
Each transaction is logged as normal, and the metadata now includes
the speaksfor data and the log always includes all of the credentials.

For testing, there is a new script in the scripts directory to
generate a speaksfor credential. Not installed since it is really
a hack. But to create one:

  perl genspeaksfor urn:publicid:IDN+emulab.net+user+leebee \
	urn:publicid:IDN+emulab.net+user+stoller

which generates a speaksfor credential that says stoller is speaking
for leebee.

Given a slice credential issued to leebee, the test scripts can be
invoked as follows (by stoller):

  createsliver.py -S speaksfor.cred -s slice.cred -c leebee.cred

A copy of leebee's self credential is needed simply cause of the test
script's desire to talk to the SA (which does not support speaksfor).
Not otherwise needed.

Oh, not tested on the AM interface yet.

It is possible for the CM daemon to grab the lock for a moment,
so lets not fail. Rare case.

and SetPublicID().

This commit is intended to makes the license status of Emulab and
ProtoGENI source files more clear.  It replaces license symbols like
"EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
blocks that contain actual license statements.

This change was driven by the fact that today, most people acquire and
track Emulab and ProtoGENI sources via git.

Before the Emulab source code was kept in git, the Flux Research Group
at the University of Utah would roll distributions by making tar
files.  As part of that process, the Flux Group would replace the
license symbols in the source files with actual license statements.

When the Flux Group moved to git, people outside of the group started
to see the source files with the "unexpanded" symbols.  This meant
that people acquired source files without actual license statements in
them.  All the relevant files had Utah *copyright* statements in them,
but without the expanded *license* statements, the licensing status of
the source files was unclear.

This commit is intended to clear up that confusion.

Most Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the Affero GNU General Public License, version 3
(AGPLv3).

Most Utah-copyrighted files related to ProtoGENI are distributed under
the terms of the GENI Public License, which is a BSD-like open-source
license.

Some Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the GNU Lesser General Public License, version 2.1
(LGPL).

This is necessary to comply with GENI AM API v3.

1. Change default slice expiration to a new site variable called
   protogeni/default_slice_lifetime, defaults to six hours.

2. Add a site variable (protogeni/warn_short_slices) to tell the
   sa_daemon if it should send email to war about short lived slices
   expiring, defaults to off.

over and over. Add a flag to geni_slices, to indicate the slice is
down.

current and maintained DBI::mysql module. A couple of things make this
a little more work then you might think.

Mysql exports a slightly different API then DBI, both at the DB *and*
the statement level. The former required some restructuring of
emdbi.pm, partly cause we want external sites to continue using Mysql
for a while longer. So, emdbi suppports both interfaces, via the
configure variable TBUSEDBI.

I also took the opportunity to also scrap the existing fork()
detection code and redo it in an easier to understand manner.
Actually, I had no idea what the previous code was trying to do, so it
was easier to just get rid of it, rather then try to make it work for
the DBI API.

There are also API differences in the "statement" class, but
fortunately this can be hidden by wrapping the statement class with a
wrapper that adds the routines we need to avoid making silly changes
to 1000s of queries. They are all trivial little things since mostly
its a matter of naming (numrows --> rows).

I also changed the library we use on ops (db/libtbdb.pm.in) to use
DBI, but in this case I just switched it over. Seemed like overkill to
worry about supporting both APIs on ops. If it works it works, and so
far it does. 

Lastly, the following modules still use Mysql directly. They all need
to be changed, but none of these are on the critical path to swapin
and swapout, so they can change later.

db/dumperrorlog.proxy.in
db/showgraph.in
db/sitevarscheck.in
bgmon/find-asymmetric
db/pelab_opspush.proxy.in
slothd/sdisrunning.in
utils/export_tables.in
utils/setbuildinfo.in
pelab/bgmon/libpelabdb.pm
pelab/dbmonitor/libtbdb.pm

sides try to stitch at the same time. One side has to back off and let
the other proceed. The problem is with the slice locking, which had to
be changed to allow one side to drop the lock so the other side could
proceed. I ended up doing this with an additional stitching lock, used
only when stitching.

is used in the vlan tag reservation code; when another CM tries to
reserve a tag for a slice we have not seen yet, we need to create
enough local state to make it work. Later when the local CM sees a
real getticket, we turn it from a placeholder slice into a real slice.

to the GeniRegistry file; move the slice part back into GeniSlice
where it belongs.

starting any one of our scripts can take a second or two. That time is
spent including and compiling 10000s of thousands of lines of perl
code, both from our libraries and from the perl libraries.

Mostly this is just a maintenance thing; we just never thought about
it much and we have a lot more code these days.

So I have done two things.

1) I have used SelfLoader() on some of our biggest perl modules.
   SelfLoader delays compilation until code is used. This is not as
   good as AutoLoader() though, and so I did it with just a few 
   modules (the biggest ones).

2) Mostly I reorganized things:

  a) Split libdb into an EmulabConstants module and all the rest of
     the code, which is slowly getting phased out.

  b) Move little things around to avoid including libdb or Experiment
     (the biggest files).

  c) Change "use foo" in many places to a "require foo" in the
     function that actually uses that module. This was really a big
     win cause we have dozens of cases where we would include a
     module, but use it in only one place and typically not all.

Most things are now starting up in 1/3 the time. I am hoping this will
help to reduce the load spiking we see on boss, and also help with the
upcoming Geni tutorial (which kill boss last time).

(this time for the urn). Also change the protoype to allow an error to
be returned.