GitLab

widearea reloading support.

* New slot in the images table to store an access key which remote
  sites must provide in order to download an image (via https).

* tmcd returns a different kind of ADDRESS field from doloadinfo.
  Instead of the multicast stuff, return a URL that points to boss'
  web server. The URL is of the form:

   https://www.myemulab.net/spewimage.php?imageid=10013&access_key=abcdef

  which as you can see is fully specified; the client does not need
  to know anything else.

* New webpage and backend scripts appropriately called "spewimage"
  which also includes support for the http HEAD request (from wget) to
  avoid downloading images that are already on the node. I just
  learned about this HEAD request stuff today ... but otherwise these
  operate as expected, spewing the image if the access key is provided.

* Changes to rc.frisbee to deal with remote loading. In addition to
  URL support, I also added support for simple paths, the intent being
  that we will probably distribute images offline (say, at night) so
  that when a node reboots it doesn't actually have to wait 60 minutes
  for an image to download. I have not added any server side support
  for this yet though. Maybe later this week.

* Other bits and pieces and fixes to make this work.

This will superceed the widearea_privkeys table that was used for
RON/Netbed but was never done well.

The privkey is just something we generate and store on the dongle (as
a simplification, we just create a dongle per node instead of
generating them on the fly) since there are not very many of these
widearea (CMU) nodes. We store the IP here for convenience (there is
an interfaces table entry of course).

The IP is updated when the node tells us it has changed, using the
privkey to uniquely identify it (for checkin).

  DB alongside the resource records. Previously, we stored only the
  nsfiles for current experiments, and purged them when the experiment
  was terminated. The new approach saves them forever using the resource
  record ID. Note that we do not store copies of NS files, but reference
  them indirectly instead so that we can MD5 them and avoid the dups.

  I put a "compressed" bit into the table cause at some point we will
  start compressing the data before storing them into the DB. Or maybe
  we bag this and start using GFS!

  Also note that this addresses the problem of losing the NS file
  history when using swapmod, since the NS file is overwritten.

* Add a pmapping table stores the nodes (and their types) used by an
  experiment. This data is also saved forever, alongside the resource
  records, so that we can more accurately replay an experiment. As Rob
  points out, the node names can also be used in conjunction with the
  ptop files that are saved, to get a 100% accurate remap of resources.

processing the node_history table, but that is *way* too slow to do on
the fly (say, from the web interface) cause of the number of records,
so the summary info is stored in the new node_utilization table. I
generate the summary info as new entries are added to node_history (in
SetNodeHistory) but if that becomes too messy, we can just as easily
shift to processing the table on a nightly basis.

Note that I added a new "inception" date field to the nodes table,
which will get set on new nodes, but for existing nodes I have to
derive it from the first entry in the node_history table, or else the
numbers will not make sense.

This change attempts to make the stats gathering code more reliable by
not relying on the testbed_stats records to reconstruct usage
statistics.  The main source of errors and total confusion in the
current stats code is that testbed_stats includes all the errors and
transitions, from which I have to reconstruct what happened in order
to determine usage by a project or user.

The new stats code still generates the testbed_stats code, but actual
usage is recorded as it happens, in the experiment_resources table, as
swapins, swapouts, and swapmods occur. Its also much faster to compute
the data for the tables in the web interface, not having to scan a
zillion testbed_stats records in php.

There is a time consuming update to the records that takes place with
a lot of tables locked.

cache latest statistics about planetlab nodes (a reduction of the
data in plab_nodehist).

deal with the issue of what a hidden run means when doing a replay, cause
more generally, replay needs a lot of work.

Other stuff leaked in too ...

I did separate out a lot of tbsetup/libArchive into db/Archive, and
whats left in libArchive.pm will eventually move over into the
Template library.

Note that I have dropped archiving of plain experiments; this is not
really owrth it outside the workbench context, and it just wastes
space and makes a lot if stuff painful in the web interface.

this as a temporary measure.

Also, renumber the last few database-migrate entries, there was a missing
number and things had gotten out of sync with database-create.

working on this a few weeks ago when I got sidetracked by the pubsub
project.

3.x to 4.x continues).

connected.

table. Note that workbench permission implies archive permission too.
Added the toggle to the project page.

Added a "stud" toggle to the user page while I was there.

(also knows an integers).

indexed by exptidx. I also got the last of the pid and pid,gid tables.

most of the rest of the tables in the system (still a few exceptions).
Bound to be some bugs ...

uid_idx to the uid for compatability with unchanged code.

declared NOT NULL (not necessary in 5.0).

term archiving of firstclass objects like users, projects, and of
course templates.

* Projects, Users, and Groups are now uniquely identified inside the
  DB by a index value that will not be reused. If necessary, this
  could easily be a globally unique identifier, but without federation
  there is no reason to do that yet.

* Currently, pid, gid, and uid still need to be locally unique until
  all of the changes are in place (which is going to take a fairly
  long time since the entire system operates in terms of those, except
  for the few places that I had to change to get the ball rolling).

* We currently archive deleted users to the deleted_users table (their
  user_stats are kept forever since they are indexed by the new index
  column). Eventually do the same with projects (not sure about
  groups) but since we rarely if ever delete a project, there is no
  rush on this one.

* At the same time, I have started a large reorg of the code, to move
  all of the user, group, project code into modules, both in php and
  perl, turning them into first class "objects" (as far as that goes
  in php and perl). Eventually, the number of query statements
  scattered around the code will be manageable, or so I hope.

* Another related part of this reorg is to make it easier to move the
  new user/project/group code in the perl backend so that it can be
  made available via the xmlrpc interface (without duplication of the
  code).

  Make an attempt to discover what the error was before an swap-* was
  canceled, if any.  Both the main error (canceled) and the other error
  are stored in the error table.  To support this a new column in the
  error table is added "rank".  The primary error has a rank 0 while the
  other error has a rank 1.

  Make an attempt to determine when an error is a "me too" error or the
  real cause of the problem.  "Me too" errors are errors which are
  generally reported when the callie script determined that the caller
  script fails.  The caller script should have reported the error, but
  in some cases the error didn't make it into the database.  Thus if a
  "me too" is reported as the cause of a "swap-*" more info is needed to
  determine the true cause.  When a "me too" error is reported it is
  followed by a "..." on it's own line.  It is also recorded in the
  errors table under the new column "need_more_info".

  Add inferred column to the errors table.  This is the same value as
  the inferred variable in tblog_find_error.

  Add revision column to errors table to make it easy to tell which
  algorithm was used to determine the error.

done in revision 4.74 above.

created.

commit logs in tbsetup for more info.