GitLab

thankless job but someone has to do it. I'm expecting to finish by the
time Bush 43 leaves office.

during rollout.

of the record. Basically, record them in the XML file, then remove
them before the import, and restore them after. Did I mention that
this was totally bogus?

On modify, create the repo, but add checks to make sure that we do not
bother with operations on branches that do not exist cause they
predate the repo creation. This allows you to move forward after a
modifying an existing template. Also add checks for instantiating a
old template that has no repo at all cause it has not been modified since
adding the repo support.

A few other fixes to make sure that instances are cleaned up after errors.

  DB alongside the resource records. Previously, we stored only the
  nsfiles for current experiments, and purged them when the experiment
  was terminated. The new approach saves them forever using the resource
  record ID. Note that we do not store copies of NS files, but reference
  them indirectly instead so that we can MD5 them and avoid the dups.

  I put a "compressed" bit into the table cause at some point we will
  start compressing the data before storing them into the DB. Or maybe
  we bag this and start using GFS!

  Also note that this addresses the problem of losing the NS file
  history when using swapmod, since the NS file is overwritten.

* Add a pmapping table stores the nodes (and their types) used by an
  experiment. This data is also saved forever, alongside the resource
  records, so that we can more accurately replay an experiment. As Rob
  points out, the node names can also be used in conjunction with the
  ptop files that are saved, to get a 100% accurate remap of resources.

Python's TemporaryFile.mkstemp() may now put dashes in the generated filenames, causing batchexp to fail when invoked through XMLRPC.

run, to store the original attributes, owner, group, and mtime/ctime/atime.

highest revision on the branch (or trunk) of the magic .template file,
since doing otherwise would seem to indicate an inconsistency in
what the user is doing. Why? Templates are modified via a tag
operation of the .template file, but splitting off can only be done
via a branch operation. Subsequent mods after a branch have to happen
along a straight line.

Also allow for tag moving to initiate a template modify since the
above implied error can be corrected by doing a cvs update, and then a
retag with the -F option to move the tag.

for each delegated slice we create, we set the 'delegations' attribute to
'utah_nmcontrol', the name of our nm-controller slice that has permissions
to talk to the NM.

Also, a few other fixes.

the run. Includes the bindings for the run and the metadata name/value
pairs. This is the same data that is dumped by template export, so
at this point template_export has been completely superceded by the
CVS repo.

  * Prior to this commit, libplab depended on db state to create slivers
    and slices.  Now it can be done using the regular command line tools
    without the metadata in the db.  This makes development and debugging
    much easier and allows us to use the command-line tools even if state has
    been cleared out of the db (i.e., for sliver garbage collection).
  * Add support for sliver start/stop/restart via the v4 NM.
  * Some support for sliver garbage collection.
  * Various other improvements and cleanup.

where the current stuff goes (datastore, nsfile, etc.). The other
subdir is called "records" and will be populated with stuff that we
currently put into the archive (the contents of the experiment
directory).

There is a record for each run, currently named XXX,YY where XXX is
the instance ID and YY is the run number. I plan to add an XML file to
map these to the symbolic names, but these are not required to be
unique, which is why the directories are not named with them.

experiment underlying a template. Rather then using the experiment directory
we use the template directory in /proj/$pid/templates ...

  since we import a bunch of dumped db state from the outer to the inner db.
  Added/removed columns in the schema misalign row data, causing much chaos.

tbsetup/elabinelab.in - Dump and send the outer db schema with the db state.
  Runs mysqldump just like schemacheck does.

tmcd/common/config/rc.mkelab - Add the schema check using schemadiff.

utils/schemadiff - Add an option to ignore table creates and drops.

a locally modifed version of CVS that does a couple of extra things. The
hacks are quite simple and currently on users. Briefly, there is a new
config file option to specify a program to run when the tag operation is
complete (similar to a module hook for rtag), and in tag.c I run that hook.

Check out a sandbox as before:

  cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo rtag mytag XXXXX

To create a new template (along the trunk), use the tag operation instead:

	cd sandbox
	cvs tag mtag

and what for the email that says the template is created. You can
continue to work along the trunk in this manner, although note that
the .template file is changing (by the backend commit), and you might
find it less distriacting to do a cvs update each time.

To work along a branch:

	cd sandbox
	cvs tag -r existing_tag -b newtag1
        cvs update -r newtag
        make changes and commits
	cvs tag newtag2

When working along branched you will want to do a cvs update to get
the .template file in sync.

"rtag" directive to initiate template modify operations. So, to get started
you do a checkout:

  cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo checkout XXXXX

where XXXXX is the part of the guid (10000/1) before the slash. Might try
and roll all templates into a single project wide repo at some point, to
avoid the extraneous path stuff, but didn't want to worry that just yet.

Okay, so have a checkout. You can work along the trunk, doing commits. To
create a new template (a modify of the existing template), you tag the tree
using rtag:

  cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo rtag mytag XXXXX

A template modify is started at the end, and you should probably wait for
email before continuing. Eventually I will need to add locking of some
kind, but I have to do the modify in the background, or else I get deadlock
cause cvs keeps the repo locked, and the modify also needs to access it.

Each time you tag along the trunk, you get a modified template, which in
the history diagram looks like:

  10000/1 --> 10000/2 --> 10000/3 ...

If you want to branch, say at 10000/2 you can create a branch tag using rtag:

  cvs -d [cut] rtag -r T10000/2 -b mytag2 XXXXX

You can also use your own tags for -r option, but I also create a TXXXXX/YY
tag at each template modify, which is easy to remember.

Then update your sandbox to the new branch, commit changes along that
branch, and then later use rtag again to initiate a template modify
operation:

  cvs update -r mytag2
  cvs commit ...
  cvs -d [cut] rtag -r mytag2 mytag3 XXXXX

And now the history diagram looks like:

  10000/1 --> 10000/2 --> 10000/3 ...
                |
                |
                -> 10000/4 ...

You should be able to mix interaction via the web with interaction via the
cvs interface. I've tested it, although not extensively.

processing the node_history table, but that is *way* too slow to do on
the fly (say, from the web interface) cause of the number of records,
so the summary info is stored in the new node_utilization table. I
generate the summary info as new entries are added to node_history (in
SetNodeHistory) but if that becomes too messy, we can just as easily
shift to processing the table on a nightly basis.

Note that I added a new "inception" date field to the nodes table,
which will get set on new nodes, but for existing nodes I have to
derive it from the first entry in the node_history table, or else the
numbers will not make sense.

This change attempts to make the stats gathering code more reliable by
not relying on the testbed_stats records to reconstruct usage
statistics.  The main source of errors and total confusion in the
current stats code is that testbed_stats includes all the errors and
transitions, from which I have to reconstruct what happened in order
to determine usage by a project or user.

The new stats code still generates the testbed_stats code, but actual
usage is recorded as it happens, in the experiment_resources table, as
swapins, swapouts, and swapmods occur. Its also much faster to compute
the data for the tables in the web interface, not having to scan a
zillion testbed_stats records in php.

There is a time consuming update to the records that takes place with
a lot of tables locked.

That is, they are not in the nodes table but they are in the outlets table.
Useful during initial bring up.

changed slightly, change parsing a bit).  The iperf bw
measurement src seems to have gone missing, so we're just not getting
these anymore.  They could come from flexlab if we ever care.

* The checkout directory now includes just the datastore directory and
  tbdata/nsfile.ns. This removes a bunch of clutter.

* You can now run template_commit in any subdir of a checkout; the
  python wrapper will crawl upwards looking for the .template file.

  Note that I removed the option that allowed you to do a commit from
  the template tree in /proj/pid/templates since that directory will
  ultimately go away (or at least hide from view), and cause it
  conflicted with the new option and I didn't want to make things any
  messier for no reason.

if the parsing code couldn't find the line it was looking for it, it
would loop forever. Now, it will detect this and return an error.

described is the one exported to ops via the XMLRPC interface. This is
just playing aroundl no doubt this stuff is going to change.

* template_checkout guid/vers

  Checkout a copy of the template to the current working directory.

* template_commit

  Modify the previous template checkout, using the nsfile contained in
  the tbdata directory (subdir of the current directory). In other words,
  the current template is modified, creating a new template in the
  current working directory (the current directory refers to the new
  template).

  The datastore subdir is imported into the new template, but that is
  the only directory that is imported at present. Might change that.

So this sounds much cooler then it really is. Why?

* This only works from ops.

* The "current directory" must be one of the standard approved directories
  (/proj, /users, /groups).

* Cause, boss reads and writes that directory via NFS, as told to it
  by the xmlrpc client.

At some point in the future it would be nice to support something
fancier, using a custom transport, but lets see how this goes.

checkboxes.

I have disabled the speculative run code, and restored the original
code.  It was a pretty minor change to bring it back, and we can
around circles as many times as we like.

if a swapmod is done, the new program agents see the right strings.

signature is that it is unique, and now the imageid is a UUID so these
mechanisms should be reconciled.  Some day...

entire experiment directory, which now that we zip files, was putting
thinks in the wrong place.