GitLab

the entire archive, lock just the "view" (current experiment subdir).
This will help some situations ... hopefully without screwing things
up otherwise.

when the experiment is deleted. This has not been happening for
templates since the archive is shared.

the repo for the stamping. This code is all temporary or at the very
least will change.

(recursively) to the UID of the real user (not effective) user. The
user must have write permission on the enclosing directory.

Currently, this script is is called from the Archive code, when
copying in files, to avoid permission errors when the current user is
not the same as the previous user.

This script can also be used from several other places that have
exhibited similar directory permission problems.

run display pages.

* Add new code to archive_control and libarchive to create tags (in
  optional subdirs) of the tags directory. These tags are simply tags
  on the current trunk at the time.

* Change to template instantiate and start/top run code to create a
  "runs" directory. Within the runs directory are subdirs named for
  each runid, and within this directory are start and stop run tags.
  Eventually add some other tags when doing swapmods, but do not have
  that figured out yet.

* Change the viewcvs code to add an "embedded" view of an archive.
  This is just an alternate view that has no header/footer goo, but
  only the actual svn contents. The main reason for this change is
  that the header and footers take up a huge amount of space and
  contribute nothing. Smaller means its easier to add to the instance
  and run show pages.

* Add the above mentioned embedded archive view to the instance and
  run show pages. On the instance page you see the directory of all
  runids, and on the show run page you see the subdir for that run.

archived.  Rather then a special "archive" directory in the experiment
directory, we know archive the entire experiment directory.

This change should be backwards compatable, but let me know if not.

Note that the nsdata directory is gone; the nsfile comes from the
tbdata, but I know place a copy in nsfile.ns so that the name is well
known.

Two-day boondoggle to support "/scratch", an optional large, shared filesystem
for users.  To do this, I needed to find all the instances where /proj is used
and behave accordingly.  The boondoggle part was the decision to gather up all
the hardwired instances of shared directory names ("/proj", "/users", etc.)
so that they are set in a common place (via unexposed configure variables).
This is a boondoggle because:

1. I didn't change the client-side scripts.  They need a different mechanism
   (e.g., tmcd) to get the info, configure is the wrong way.

2. Even if I had done #1 it is likely--no, certain--that something would
   fail if you tried to rename "/proj" to be "/mike".  These names are just
   too ingrained.

3. We may not even use "/scratch" as it turns out.

Note, I also didn't fix any of the .html documentation.  Anyway, it is done.
To maintain my illusion in the future you should:

1. Have perl scripts include "use libtestbed" and use the defined PROJROOT(),
   et.al. functions where possible.  If not possible, make sure they run
   through configure and use @PROJROOT_DIR@, etc.

2. Use the configure method for python, C, php and other languages.

3. There are perl (TBValidUserDir) and php (VALIDUSERPATH) functions which
   you should call to determine if an NS, template parameter, tarball or
   other file are in "an acceptable location."  Use these functions where
   possible.  They know about the optional "scratch" filesystem.  Note that
   the perl function is over-engineered to handles cases that don't occur
   in nature.

Each template has a datastore, which is really just a subdirectory that can
be populated with files, and committed to the subversion archive.  Note,
the datastore os specific to the template itself. The Template Archive link
on the Show Template page takes you to the subdirectory, which by
convention I am calling "datastore".

The directory actually lives in /proj/pid/exp/eid/TGUID-VERS ... but that
path is printed out for you on the archive page.

Anyway, put stuff in the datastore directory, and then commit the template
archive so there is a tag associated with it.

When an instance is created, a checkout of the datastore is placed in the
experiment directory (/proj/pid/eid/exp/template_datastore). The current
tag (from above) is stored with the instance so that we can later recreate
the enviroment for the instance, say for rerun.

Tarfiles and rpms in the datastore can be referenced as xxx://foo.rpm (in
your NS file).  tarfiles_setup transforms those when the instance is
swapped in, sorta like it does other URLs, only it does not actually fetch
them, just need to rewrite the paths so they reference datastore.

The program agent gets another environment variable so you can refer to the
datastore without hardwiring paths ($DATASTORE). Eventually I want to move
the checkout someplace else, but it was easy to drop it into the experiment
directory for now.

* Prefix the svn importer script with a 'ulimit -c -' to avoid core
  dump file from some internal SVN thing, that appears to be
  harmeless.

* Use rsync instead of tar to archive the archives; it appears to be
  smarter about setgid directories.

use loghole to capture the log files and store them in the experiment
archive before the commit point between runs (and at swapout).

Some other small unrelated changes as well.

but that stuff is still not totally hooked in yet.

a huge amount of work, but really needs someone to test drive it.

Add some code for adding missing files (as determined by the NFS
tracing code) to the archive. This is code is not complete or tested,
but getting it committed anyway.

of experiments that share an archive.

does his own commit.

* Add support for linking to the NS file that will be used, from the
  begin experiment page, when duplicating or branching an experiment.
  Ultimately we want to separate things so that user can first edit
  the NS file and then proceed to branching.

* In discussion we agreed to use the convention that a directory called
  "archive" in experiment directory, will always be saved and restored.
  This has been implemented.

* Add more of the support for branching an experiment (the archive).
  Batchexp takes a couple of new arguments:

	-c pid,eid[:tag]  or
	-c exptidx[:tag]

  The above specifies what and where to duplicate or branch. Simply
  giving pid,eid does not use the archive, but just copies right out
  of the existing experiment directory.

  Adding the -b option says to branch instead of duplicate.

by not including them in the actual pathnames to be copied with
tar/rsync.

permissions. Fixes one directory permission problem.

added in a rush and I need to take a step back and think about it.

to the archive does not exist (specifically, from the NFS tracing).

of the archive.

* The rest of the backend support for simplistic experiment duplication,
  either from an existing (current) experiment, or from a specific
  archive revision of a current or terminated experiment.

* Rework the swapmod code so that the archive is committed at the exact
  end of the swapout phase. This required adding (and moving) some code
  from swapexp to tbswap sine that is where the actual swapout/swapin
  happens during a swapmod

* Add a special directory called "archive" to the experiment
  directory, which is a place where users can store stuff they want
  saved away. This will eventually be a user defined set of
  directories, but this was good for getting the basic mechanism in
  place. Note that the when the contents of this directory are copied
  out for placement into the archive, it is an exact copy made with
  rsync.

* No longer "clean" the contents of the temporary store between
  commits of the archive. This was creating a lot of headaches, and
  was also causing the revision history to get messed up. The downside
  of this is that we have to be more careful to explicitly delete
  files that the user no longer uses. I have not solved all these
  issues yet, so in the meantime files will get left in the archive
  even if the user no longer references them.