GitLab

Other stuff leaked in too ...

I did separate out a lot of tbsetup/libArchive into db/Archive, and
whats left in libArchive.pm will eventually move over into the
Template library.

Note that I have dropped archiving of plain experiments; this is not
really owrth it outside the workbench context, and it just wastes
space and makes a lot if stuff painful in the web interface.

this as a temporary measure.

Also, renumber the last few database-migrate entries, there was a missing
number and things had gotten out of sync with database-create.

working on this a few weeks ago when I got sidetracked by the pubsub
project.

3.x to 4.x continues).

connected.

table. Note that workbench permission implies archive permission too.
Added the toggle to the project page.

Added a "stud" toggle to the user page while I was there.

(also knows an integers).

indexed by exptidx. I also got the last of the pid and pid,gid tables.

most of the rest of the tables in the system (still a few exceptions).
Bound to be some bugs ...

uid_idx to the uid for compatability with unchanged code.

declared NOT NULL (not necessary in 5.0).

term archiving of firstclass objects like users, projects, and of
course templates.

* Projects, Users, and Groups are now uniquely identified inside the
  DB by a index value that will not be reused. If necessary, this
  could easily be a globally unique identifier, but without federation
  there is no reason to do that yet.

* Currently, pid, gid, and uid still need to be locally unique until
  all of the changes are in place (which is going to take a fairly
  long time since the entire system operates in terms of those, except
  for the few places that I had to change to get the ball rolling).

* We currently archive deleted users to the deleted_users table (their
  user_stats are kept forever since they are indexed by the new index
  column). Eventually do the same with projects (not sure about
  groups) but since we rarely if ever delete a project, there is no
  rush on this one.

* At the same time, I have started a large reorg of the code, to move
  all of the user, group, project code into modules, both in php and
  perl, turning them into first class "objects" (as far as that goes
  in php and perl). Eventually, the number of query statements
  scattered around the code will be manageable, or so I hope.

* Another related part of this reorg is to make it easier to move the
  new user/project/group code in the perl backend so that it can be
  made available via the xmlrpc interface (without duplication of the
  code).

  Make an attempt to discover what the error was before an swap-* was
  canceled, if any.  Both the main error (canceled) and the other error
  are stored in the error table.  To support this a new column in the
  error table is added "rank".  The primary error has a rank 0 while the
  other error has a rank 1.

  Make an attempt to determine when an error is a "me too" error or the
  real cause of the problem.  "Me too" errors are errors which are
  generally reported when the callie script determined that the caller
  script fails.  The caller script should have reported the error, but
  in some cases the error didn't make it into the database.  Thus if a
  "me too" is reported as the cause of a "swap-*" more info is needed to
  determine the true cause.  When a "me too" error is reported it is
  followed by a "..." on it's own line.  It is also recorded in the
  errors table under the new column "need_more_info".

  Add inferred column to the errors table.  This is the same value as
  the inferred variable in tblog_find_error.

  Add revision column to errors table to make it easy to tell which
  algorithm was used to determine the error.

done in revision 4.74 above.

created.

commit logs in tbsetup for more info.

per-experiment user and password for the per-experiment DB.

we have a record of what nodes failed to respond during the
synchronous parts of the start and stop experiment run. Data is stored
for now, but not sure how it will be used later. The main goal is to
avoid the problems we see with plab nodes where they do not respond,
and start/stop run hang and fail.

was resquested by the user.

single end of run archive tag.

Add bwlimit column to widearea_nodeinfo table.

* Add XMLRPC interface for template swapin,stoprun,startrun,swapout and
  add the appropriate wrappers to the script_wrapper on ops.

* Allow parameter descriptions in NS files. This is probably not in its
  final form since its a bit confusing as to what has priority; something
  in the NS file or a metadata item. Anyway, you can do this in your NS
  file:

	$ns define-template-parameter GUID "0/0" "The GUID to be analyzed"

  The rules are currently that the NS file description has priority and
  is copied to child templates, unless the user has modified a description
  via the web interface, in which case the NS file description is ignored.
  I know, sounds awful, but for the most part people are going to use the
  NS file anyway.

* Add "clear" option when starting a new experiment run; the per
  experiment DB at the logholes are cleared. Note that this is *not* the
  default behaviour; you have to either check the checkbox on the web form
  or use the -c option to the script wrapper, or clear=yes if talking
  directly to the XMLRPC server.

* Fix up how email is generated for template_swapin and template_create,
  so that Kevin can debug tblog/tbreport stuff, but also so that we maintain
  mail logs as before. I have made some improvements to libaudit so as to
  centralize the mail goo, and avoid duplicating all that stuff.

* Minor fixes to the program agent so that the new environment strings are
  sent before the program agent exits and reloads them!

* Other minor little things.

Change types of latitude and longitude columns in widearea_nodeinfo to
float (instead of float(6,5)) to fix problem with values getting adjusted
improperly during insert.  Apparently the definition of the float(n,m)
type is non-standard, and has changed since sql 3.x.  Regular float
columns will do fine here.

More tbreport changes from Mike Kasick <mkasick@andrew.cmu.edu>:

- Added tblog support to nscheck.

- Added ns_parse_failed error to nscheck.

- Added invocation column to report_errors to differentiate between assign
  runs in infeasible resource assignments.

Prep for Mike Kasick report code.  Updated database schema and
installed hooks for his code.

Cleaned up how errors were handled in tblog(...).

Allow SENDMAIL to be called before the path is untained in '-T' scripts.

Other small changes.

draft is that the user will at the end of an experiment run, log into one
of his nodes and perform some analysis which is intended to be repeated at
the end of the next run, and in future instantiations of the template.

A new table called experiment_template_events holds the dynamic events for
the template. Right now I am supporting just program events, but it will be
easy to support arbitrary events later. As an absurd example:

	node6> /usr/local/bin/template_analyze ~/data_analyze arg arg ...

The user is currently responsible for making sure the output goes into a
file in the archive. I plan to make the template_analyze wrapper handle
that automatically later, but for now what you really want is to invoke a
script that encapsulates that, redirecting output to $ARCHIVE (this
variable is installed in the environment template_analyze.

The wrapper script will save the current time, and then run the program.
If the program terminates with a zero exit status, it will ssh over to ops
and invoke an xmlrpc routine to tell boss to add a program event to both
the eventlist for the current instance, and to the template_eventlist for
future instances. The time of the event is the relative start time that was
saved above (remember, each experiment run replays the event stream from
time zero).

For the future, we want to allow this to be done on ops as well, but
that will take more infrastructure, to run "program agents" on ops.

It would be nice to install the ssl xmlrpc client side on our images so
that we do not have to ssh to ops to invoke the client.