GitLab

seconds it takes the kernel to decide the other machine is not
responding (as will happens as machines are rebooting), down to 5
seconds, using a SIGARLM.

used. Add .pid file creation in /var/run to kill daemon from the new
rc.syncserver script in tmcd client code.  Add -v flag to increase
server verbosity. Add -l argument to specify where to write the log
file.

Not perfect, as it takes about 30 seconds to timeout. Need to add
a timer to it.

enthusiasm of those who want to replace expensive commercial solutions
(and so I don't have to tell people over and over...)

Currently, this only means defining a signature file and creating a
utility to make them and check them against a disk.  The signature file
is not used by frisbee/imageunzip yet.

is correct.

path.

Ok, not really any fundamental changes, mostly a source reorganization,
but some new features as well:

1. Split off FS specific code for all FS types into subdirectories.
   imagezip.c was just getting unwieldy.  You can now (in theory, I
   haven't enabled this in any easy way) build a version of imagezip
   with only the FSes you care about.

2. As part of the above, localize the filesystem specific headers in
   the subdirs instead of relying on the installed system headers.
   This allows easy building of, for example, BSD FS support under
   Linux.

3. Hacked up the GNUmakefiles some so that imagezip with all FS support
   will build (in the Emulab context) under Linux and FreeBSD 5.1 as
   well as FreeBSD 4.x.  Still a hack, for Linux you need to first do:
	setenv LINUX 1
	setenv HAVE_GCC3 1
	setenv HAVE_LOCALE 1
   and for FreeBSD 5.x:
	setenv HAVE_GCC3 1
	setenv HAVE_LOCALE 1
   these trigger ifdefs in some of the Makefiles to DTRT.  Imagezip
   and frisbee still need a standalone configure...

4. Added support for UFS2 (aka FreeBSD 5.x default filesystems).

5. Added "-S dos-type" option to imagezip.  This is the general form
   of the -b, -l, and -n options to force imagezip to treat the device
   as though it contains a filesystem of the indicated type.

from looking at the embedded (and deprecated) blockindex field in
the headers.  Now imagedump won't complain if you run it on a
piece of a split image.

change awkward "uncompress" to "decompress"

   data block" field was non-zero.  This happened with the Intel-provided
   Dell boxen and their initial Linux disk.

2. Added some more sanity checks in the Linux code.

3. Fixed a bug I just introduced where slice mode would fail on a disk
   with an extended partition.

as defined in the defs-* file (e.g. "TBLOGFACIL=local2").  The default is
"local5" which is what we are setup to use so you shouldn't need to mess
with your defs- file!

perl scripts just get this value configured in when configure is run.
C programs get the value in two ways.  For programs that are intimate with
the testbed infrastructure, and include "config.h", they just get it from
that file.  For programs that we sometimes use outside the Emulab build
environment (e.g., frisbee, capture) and that don't include config.h,
the value is set via a "-DLOG_TESTBED=..." in the GNUmakefile build line.
If the value isn't set, it defaults to what it used to be (usually LOG_USER).

Still to do: healthd, hmcd (whose build doesn't seem to be completely
integrated) and plabdaemon.in (since its icky python :-)

* install-rpm, install-tarfile, spewrpmtar.php3, spewrpmtar.in: Pumped
  up even more! The db file we store in /var/db now records both the
  timestamp (of the file, or if remote the install time) and the MD5
  of the file that was installed. Locally, we can get this info when
  accessing the file via NFS (copymode on or off). Remote, we use wget
  to get the file, and so pass the timestamp along in the URL request,
  and let spewrpmtar.in determine if the file has changed. If the
  timestamp it gets is >= to the timestamp of the file, an error code
  of 304 (Not Modifed) is returned. Otherwise the file is returned.

  If the timestamps are different (remote, server sends back an actual
  file), the MD5 of the file is compared against the value stored. If
  they are equal, update the timestamp in the db file to avoid
  repeated MD5s (or server downloads) in the future. If the MD5 is
  different, then reinstall the tarball or rpm, and update the db file
  with the new timestamp and MD5. Presto, we have auto update capability!

  Caveat: I pass along the old MD5 in the URL, but it is currently
  ignored. I do not know if doing the MD5 on the server is a good
  idea, but obviously it is easy to add later. At the moment it
  happens on the node, which means wasted bandwidth when the timestamp
  has changed, but the file has not (probably not something that will
  happen in typical usage).

  Caveat: The timestamp used on remote nodes is the time the tarfile
  is installed (GM time of course). We could arrange to return the
  timestamp of the local file back to the node, but that would mean
  complicating the protocol (or using an http header) and I was not in
  the mood for that. In typical usage, I do not think that people will
  be changing tarfiles and rpms so rapidly that this will make a
  difference, but if it does, we can change it.

* node_update.in, client side watchdog, and various web pages:
  Deflated node_update, removing all of the older ssh code. We now
  assume that all nodes will auto update on a periodic basis, via the
  watchdog that runs on all client nodes, including plab nodes.

  Changed the permission check to look for new UPDATE permission (used
  to be UPDATEACCOUNT). As before, it requires local_root or better.
  The reason for this is that node_update now implies more than just
  updating the accounts/mounts. The web pages have been changed to
  explain that in addition to mounts/accounts, rpms and tarfiles will
  also be updated. At the moment, this is still tied to a single
  variable (update_accounts) in the nodes table, but as Kirk requested
  at the meeting, it will probably be nice to split these out in the
  future.

  Added the ability to node_update a single node in an experiment (in
  addition to all nodes option on the showexp page). This has been
  added to the shownode webpage menu options.

  Changed locking code to use the newer wrapper states, and to move
  the experiment to RUNNING_LOCKED until the update completes. This is
  to prevent mayhem in the rest of the system (which could be dealt
  with, but is not worth the trouble; people have to wait until their
  initiated update is complete, before they can swap out the
  experiment).

  Added "short" mode to shownode routine, equiv to the recently added
  short mode for showexp. I use this on the confirmation page for
  updating a single node, giving the user a couple of pertinent (feel
  good) facts before they comfirm.

program, it only took a couple of hours.  Heavily tested: it didn't core dump
examining my 20GB FAT32 partition, ship it!

Actually, I did imagezip/imageunzip a FAT12 DOS floppy.  Since imagezip
files are a minimum of 1MB (the chunk size), it is probably not practical
for saving 1.4MB floppies :-)

Also, updated the man page.

Changes to "info" option

"Piece of cake!" he says...
"15 minutes" he says...

Well, a day later, we can now deal with extended partition tables.
It was almost easy.  The offsets in an extended partition table are
relative to the location of that table...mostly.  The exception is if
an extended partition table contains an entry for another extended table.
Then the offset provided is relative to the *first* extended partition
table found, not the current one.

This was also complicated by the NTFS library code which needs to be
fed a device special file corresponding to the filesystem partition.
So if I am cruising alone creating a whole disk image using /dev/ad0
and I hit a partition containing an NTFS, I have to figure out the
BSD name for that particular partition: "hmm..I'm looking at partition 3?
That would be /dev/ad0s3".  Its easy for the first 4 partitions, but for
extended partitions, I had to figure out how BSD mapped those partitions
into device names.  Once I became aware of the DIOCGSLICEINFO ioctl, this
was easy as it gave back all the info needed.

rather than creating a new partition.

We have talked about these in the past, and I just happened to be thinking
about it this morning.

tmcd (which is bad, since tying up the tmcd threads blocks all nodes
in the testbed). The old functionality is left in tmcd for now.

On the server side, a new web page (www/spewrpmtar.php3) receives a
request for a file, along with the nodeid (pcXXX) making the request,
and the secret key that is generated for each new experiment and
transfered to the node via tmcd. If the key matches, the operation is
handed off to tbsetup/spewrpmtar.in which verifies that the file is in
the list of rpm/tar files for that node, and then spits it out to
stdout. The web page uses fpassthru() to send the file out to the
client. The client is using wget, and is required to use https (the
web page checks).

At present, the external script is run as the creator of the
experiment, and gid of the experiment. Perhaps this is not a good
idea. In any event, the file must be in the list of rpm/tarfiles,
either owned by the experiment creator or with a group of the
experiment, and the file must reside in either /proj or /groups.
I use the realpath() function to make sure there are no symlink tricks
pointing to outside those filesystems. I use the standard NFS read goo to
prevent transient mount problems that we all know and love.

add -c option to force copying file over locally before untaring,
the copy code knows how to retry read operations that fail

to treat the given slice as an NTFS.  Analogous to -b for bsd and -l
for linux.

Clean up a little lint in the NTFS code too

* Parser: Added new tb command to set the name of the sync server:

	tb-set-sync-server <node>

  This initializes the sync_server slot of the experiment entry to the
  *vname* of the node that should run the sync server for that
  experiment. In other words, the sync server is per-experiment, runs
  on a node in the experiment, and the user gets to chose which node
  it runs on.

* tmcd and client side setup. Added new syncserver command which
  returns the name of the syncserver and whether the requesting node
  is the lucky one to run the daemon:

    SYNCSERVER SERVER='nodeG.syncserver.testbed.emulab.net' ISSERVER=1

  The name of the syncserver is written to /var/emulab/boot/syncserver
  on the nodes so that clients can easily figure out where the server
  is.

  Aside: The ready bits are now ignored (no DB accesses are made) for
  virtual nodes; they are forced to use the new sync server.

* New os/syncd directory containing the daemon and the client. The
  daemon is pretty simple. It waits for TCP (and UDP, although that
  path is not complete yet) connections, and reads in a little
  structure that gives the name of the "barrier" to wait for, and an
  optional count of clients in the group (this would be used by the
  "master" who initializes barriers for clients). The socket is saved
  (no reply is made, so the client is blocked) until the count reaches
  zero. Then all clients are released by writting back to the
  sockets, and the sockets are closed. Obviously, the number of
  clients is limited by the numbed of FDs (open sockets), hence the
  need for a UDP variant, but that will take more work.

  The client has a simple command line interface:

    usage: emulab-sync [options]
    -n <name>         Optional barrier name; must be less than 64 bytes long
    -d                Turn on debugging
    -s server         Specify a sync server to connect to
    -p portnum        Specify a port number to connect to
    -i count          Initialize named barrier to count waiters
    -u                Use UDP instead of TCP

    The client figures out the server by looking for the file created
    above by libsetup (/var/emulab/boot/syncserver). If you do not
    specify a barrier "name", it uses an internal default. Yes, the
    server can handle multiple barriers (differently named of course)
    at once (non-overlapping clients obviously).

    Clients can wait before a barrier in "initialized." The count on
    the barrier just goes negative until someone initializes the
    barrier using the -i option, which increments the count by the
    count. Therefore, the master does not have to arrange to get there
    "first." As an example, consider a master and one client:

	nodeA> /usr/local/etc/emulab/emulab-sync -n mybarrier
	nodeB> /usr/local/etc/emulab/emulab-sync -n mybarrier -i 1

    Node A waits until Node B initializes the barrier (gives it a
    count).  The count is the number of *waiters*, not including the
    master. The master is also blocked until all of the waiters have
    checked in.

    I have not made an provision for timeouts or crashed clients. Lets
    see how it goes.

size of the file.

block size. I was only allowing 4K block size FS's, and assumed that
the group descriptor map was in the second block (not true when the
blocksize if 1K), and that the number of blocks in a descriptor as 8
times the block size. Also moved some sanity checks around, and
cleaned up a couple of seeks. Now works on the baleep image, which is
where we first noticed the problem.

1. From the "Thank you DOS for all your stupid limits" department
   we have an enhancement to the -I option to ignore a slice.
   The slice number can be followed by a BSD partition letter
   (e.g., "-I1g") to ignore a BSD partition within a slice.
   Ditto for the -R options which forces raw compression.

2. From the "Gee, I shoulda taken that grinding noise more seriously"
   department we have the -X option to force imagezip to try e(X)tremely
   hard (yeah, yeah, its the only letter I could come up with) to read
   the data off the disk, even faking (zero) data if a read fails after
   10 tries.