GitLab

New version of unified tbswap in/out.
startexp/endexp/swapexp have been changed to use new script.

tbswapin and tbswapout have been replaced with a script which
spits out a warning message, then calls tbswap appropriately.

The README has also been modified.

Attempts to replay an experiment by rebooting all the nodes, clearing
the various startup bits (ready, startstatus, bootstatus, portstats),
and then restarting the event system. I am dubious that this is a
workable solution because of the asynchronous nature of the testbed
(nodes happily cruise from TBRESET to ISUP and beyond without
stopping), and so its hard to truly replicate the initial lack of
state that a freshly swapped in experiment has. Still, people
requested it and I cheerfully provided it cause thats what I do;
service with a smile and not a wit of complaint. Is anyone reading
this?

experiment. Here is mail to tbops:

* Moved the working directory for experiment setup/swap/end to a new
  directory located on boss instead of over NFS to /proj/$pid/$eid. This
  new location is /usr/testbed/expwork/$pid/$eid.

* Changed the name of the directories we create in /usr/testbed/expinfo to
  $pid-$eid.$index where $index is a new autoincrement field in the DB
  table. I really hated the names that were created before.

* Changed where logs are written from /tmp to the new location in
  /usr/testbed/expwork/$pid/$eid.

Okay, why.

* We no longer operate on NFS mounted directories that might hang. Its
  easier to catch the situation where a copy of the log file over at the
  end of experiment creation fails cause of an NFS problem.

* We no longer have user writable files that are inputs to other parts of
  the system (like top and ptop files).  Not that a user would be bad, but
  it closes a hole.

* We no longer copy user writable files from /proj to boss where we might
  fill up an important filesystem cause the user put a .ndz file in the the
  working directory. Not that a user would be bad, but it closes a hole.

* Its easier to save all the log files this way, for each swap in and
  out.

* Removing a directory over NFS is a royal irritant when someone is CD'ed
  into that directory or looking at a file on the other side (the astute
  observer will peg this as the reason I went down this idiotic path in the
  first place!).

* About 6 other reasons that I can no longer remember. Seriously, I really
  had more reasons I can no longer remember! :-)

directory so that they can be viewed later after the operation is
complete. I've also cleaned up the mechanism for determining when
a log file is active (for the web spew) by using another slot in the
experiments table, and added some libdb routines to manage that slot.
At present just the last (or latest) log can be viewed after the fact,
but we can change that later if think its really necessary. At the
same time, make it possible for admin types to view the log files for
other peoples expierments; spew is setuid, but flips back after
opening the file (does usual checks too). I've also incorporated the
log changes into the batch daemon, so you can view the last batch log
too, although I have not tested that yet!

transition error when you click too fast after creating it. Instead of
looking at experiment state, use the logile slot of the experiments
table, and make sure its cleared/set properly in start/swap experiment
scripts.

Also added a spew option to the swap page so you can watch experiments
swap in/out.

cleanup of tbreport.

line in all email from the system. Remove all of the TESTBED: tags
and modify the email function in the web server and perl library to
prepend @DOMAIN@: to the message.

are created 664, so that group members other than the experiment
creator can swap them.

all those top and ptop files are collecting in /usr/testbed/www!

but simply entered into the DB record for the experiment until we know
what to do with them. Add to batchexp script arguments, since all that
stuff is done outside the web interface. Add a swapexp perl script to
swap an an experiment in/out form the command line. Add web links on
the Experiment Information page to do this from the web interface. A
bunch of locking changes. Previously expt_terminating in the
experiment record prevented multiple calls to terminate an experiment,
but now we have a more general locking problem with
start,swapin,swapout, and terminate, so change expt_terminating to
expt_locked (still a datetime field) and add locking to all of
startexp, swapexp, and endexp. Note that batch experiments cannot be
swapped yet because of locking issues still to be resolved. Minor
cleanup in tbreport to make email message look better.

experiment code. No longer uses another table. Rather, the experiment
record contains a couple of extra fields for the batch system. Also
combined some of the backend code (no longer a killbatch script).
Also added scriptable experiments; the batchexp program in the bin
directory can start an experiment from the command line, and in fact
is used from the web page for both batch experiments and immediate
experiments (-i option). All of the DB code that was in the web
interfaces was moved to batchexp.

terminated.

experiment.

email, for the case when someone other than creator does the
termination.

will terminate properly through web interface.

proper headers. Split out some of the mail into testbed-logs,
testbed-ops, and testbed-approval. Added a library for including from
our perl scripts. Contains a couple of mail helper functions, but will
hopefully contain more as time goes by.

Fixed a bug in the web interface that was causing breakage for people
with multiple accounts. Mac and Jay have noticed this, when logging
out and trying to join or create a project under a new or different
name.

Change to startexp/endexp, which are almost scriptable now (can be
called directly in addition from the web page). Add front ends to
these for the web page (webstartexp and webendexp). These changes are
mostly support for batch mode.

termination, setup now exits immediately and sends email to the user
when the experiment is fully configured.

by the web server forks a child to do the actual work of calling tbend
and other stuff. The parent returns right away and the script ends.
When the experiment termination (child) ends, an email message is sent
to the user that issued the termination request. To prevent multiple
clicks, I added a DB field called expt_terminating that is a DATETIME
field.  If the field is set, the script fails and the user is told to
be more patient. I used a DATETIME field mostly for debugging purposes
so we can track and future problems.

the testbed list.

using a popen now.

webserver, as the uid of the experimentor. Not fully debugged yet
since the testbed has been kinda broken all day, but this needs to get
in if anyone else works on it.

We now pass through <pid> and <eid>, and do not munge the filenames.