GitLab

Untested, but compiles!

Unified the 'dot' handling (status printing) of frisbee and imagezip.
They now both report the number of chunks remaining along with the dots.
Also put out a periodic splat for every GB of uncompressed data we write.
This is useful when you are zero-filling, since otherwise it appears that
frisbee has hung when it is really just zeroing the last unused 100GB of
your disk.

any partition or slice.

Hope to do this someday soon...

create "delta" images based on the hash signatures that can be computed
by imagehash for an image.

The checkpoint driver can return a list of blocks that have been modified
since its startup.  If you use "imagezip -C /dev/shd0 -s 1 /dev/ad0 foo.ndz"
imagezip will make an ioctl call on the shd device to get the modified
blocks it needs to save (rather than parsing the filesystem).  It then
creates an image using those blocks.

The end result should be an "incremental" image that can be put down on
the disk by frisbee after the original image has been loaded.

This is all a theory right now (including the ioctl :-) but should get
tested RSN.

- add some missing casts
- avoid void * arithmetic
- change "q" printf formats to "ll"

Still not there yet by any means.  Note that the version of the NTFS library
we use won't compile with "-pedantic -ansi" so those are not part of the
default CFLAGS.

   This way we can use frisbee as a fast network copy.

2. In imageunzip, don't truncate the output file until we have performed
   basic validity checks.  Otherwise, if you happened to reverse the order
   of the imagefile and the output disk, it would truncate the imagefile
   and THEN fail.

3. Strip binaries when they are installed.

The only meaningful change was to insert a sched_yield() in the frisbee
decompressor path.  Apparently, the decompressor can run long enough to
cause the incoming socket buffer to overflow.  I was under the assumption
that the decompressor would not run much longer than a single time slice
(0.001 seconds, about 8 packets) before its priority would force it to
be context switched.  But it was running much longer than that!  Forcing
a periodic yield seems to have taken care of this.

One other cause of retransmitted blocks that I saw was where the server
was taking a long time to read data from a file (up to 0.25 seconds).
This would stall the clients and force them to rerequest blocks (which
they do after about 0.10 seconds).  We can improve on this by splitting
the file reading off to a seperate thread.

Most other changes are related to the event logging code.

Don't define it for frisbee.

the embedded BIOS disk number in sector addresses.  So you still cannot
create an image on disk 0 and load it on disk 1.

a signature file, rather it compares the actual data in the image file vs.
what is on a disk.  Mostly a debugging option, trying to find the source
of a disk corruption problem when loading images with FBSD 4.10.

Teach it about relocations, and (optionally) have it not compute hashes
for blocks which involve relocations, since those areas will likely always
differ in the image and disk.  The default behavior is to have it look at
everything as before, use -N to have it skip relocation areas.

possible to:

	gmake client
	sudo gmake client-install

on a FBSD4, FBSD5, RHL7.3, and RHL9.0 client node.

There are still some dependencies that are not explicit and which would
prevent a build/install from working on a "clean" OS.  Two that I know of are:
you must install our version of the elvin libraries and you must install boost.

1. "make clean" will just remove stuff built in the process of a regular build
2. "make distclean" will also clean out configure generated files.

This is how it was always supposed to be, there was just some bitrot.

the top level.  This will build all the necessary binaries and then install
them.  This works on FBSD4 and RHL7.3.  It still doesn't work on FBSD5
(newer compiler that no longer supports a style of use of _FUNCTION_ in the
event lib) or RHL9 (event lib needs SSL lib which has a bad dependency
on Kerberos).  Notes:

- requires that elvin libraries be installed on nodes (they are) to build
  event agents, requires linuxthreads be installed on FBSD (it is now) to
  build imagezip (which is installed, but is not strictly necessary)

- installed event-agents and other binaries are stripped

- added a few missing files to the source tree for bsd (healthd.conf)
  and linux (healthd.conf, rc.local)

- the only thing that doesn't get rebuilt in /usr/local/etc/emulab is
  healthd, I couldn't quickly find how it gets built

- uses a scaled down version of libtb with no DB functions (since mysql
  isn't installed on nodes).  N.B. DO NOT DO A CLIENT INSTALL FROM YOUR
  REGULAR OBJ TREE OR ELSE YOU MAY WIND UP WITH A NEUTERED VERSION OF
  libtb.a!

The build-as-well-as-install semantics are counter to the regular install
targets, but this is what we gotta do for now.  Once the TB source builds
under Linux and newer BSDs, we could undo this and just require that people
do a regular "make" followed by "make client-install"  OTOH, there should
be no reason to require installation of mysql and other server-side packages
just to build clients (or make them sit through the compilation of assign),
so maybe we will keep the client build special.

with every DOS partition.  It was only used by NTFS and has been replaced
by equally hacky changes to the NTFS library.

The benefit of the latter is that it at least works all the time on
Linux and all FreeBSD versions.

Add a new interface to the library to enable us to use the whole-disk
device rather than having to (figure out and) use a specific slice device.

to build under FreeBSD 5 (really newer gcc's)

1. Make imagezip (and imageunzip) understand LILO bootblocks.
   Imagezip will create relocation information for all the absolute
   blocks that LILO embeds in the partition.  Imageunzip will correctly
   relocate those locations when laying down the new image.  Thus
   Linux slice images can be layed down anywhere on the disk (as can
   BSD images).

2. Because #1 is incompatible with the old imageunzip, I bumped the
   imagezip format version number (to 3) on images that contain relocations
   (i.e., slice images).  Make sure you have a new imageunzip to unzip
   new slice images.  Whole disk images are still created as version 2
   since they contain no relocs and thus are no different than before.

3. Added -N option to imagezip to create slice images without any
   relocations.  [ We could create our Emulab images this way, since we
   don't have any mechanism for, or reason to, let the user choose which
   partition to load an image in.  Hence our slice images always wind up
   at the same offset on the disk and relocations are unnecessary. ]
   Note that there is no significant expense in either time or space for
   relocations, this option was more of a debugging thing.

fix up standalone makefiles
have imagezip -v spit out the FSes it supports

Currently, this only means defining a signature file and creating a
utility to make them and check them against a disk.  The signature file
is not used by frisbee/imageunzip yet.

path.

Ok, not really any fundamental changes, mostly a source reorganization,
but some new features as well:

1. Split off FS specific code for all FS types into subdirectories.
   imagezip.c was just getting unwieldy.  You can now (in theory, I
   haven't enabled this in any easy way) build a version of imagezip
   with only the FSes you care about.

2. As part of the above, localize the filesystem specific headers in
   the subdirs instead of relying on the installed system headers.
   This allows easy building of, for example, BSD FS support under
   Linux.

3. Hacked up the GNUmakefiles some so that imagezip with all FS support
   will build (in the Emulab context) under Linux and FreeBSD 5.1 as
   well as FreeBSD 4.x.  Still a hack, for Linux you need to first do:
	setenv LINUX 1
	setenv HAVE_GCC3 1
	setenv HAVE_LOCALE 1
   and for FreeBSD 5.x:
	setenv HAVE_GCC3 1
	setenv HAVE_LOCALE 1
   these trigger ifdefs in some of the Makefiles to DTRT.  Imagezip
   and frisbee still need a standalone configure...

4. Added support for UFS2 (aka FreeBSD 5.x default filesystems).

5. Added "-S dos-type" option to imagezip.  This is the general form
   of the -b, -l, and -n options to force imagezip to treat the device
   as though it contains a filesystem of the indicated type.

from looking at the embedded (and deprecated) blockindex field in
the headers.  Now imagedump won't complain if you run it on a
piece of a split image.

change awkward "uncompress" to "decompress"

   data block" field was non-zero.  This happened with the Intel-provided
   Dell boxen and their initial Linux disk.

2. Added some more sanity checks in the Linux code.

3. Fixed a bug I just introduced where slice mode would fail on a disk
   with an extended partition.

program, it only took a couple of hours.  Heavily tested: it didn't core dump
examining my 20GB FAT32 partition, ship it!

Actually, I did imagezip/imageunzip a FAT12 DOS floppy.  Since imagezip
files are a minimum of 1MB (the chunk size), it is probably not practical
for saving 1.4MB floppies :-)

Also, updated the man page.