Commit a6a648e9 authored by Mike Hibler's avatar Mike Hibler
Browse files

all:

	add -Wall to CFLAGS and clean up lint
	update the TODO file
	explicitly size the header fields (e.g., int32_t not int)

imagezip:
	Version 2.

	Adds two ints to the header to help track free space.  Each chunk
	now has a first and last sector number which can describe any free
	block before or after the data contained in the chunk.  This is
	needed in order to properly zero all free space when laying down
	an image.  In practice: the first chunk describes any free space
	before the first allocated range and any free space after its
	contained ranges and before the first allocated range in the second
	chunk.  Every other chunk then describes just free space following
	itself (since the previous chunk has already described the space
	before this chunk).  The point being, we only describe each free
	range once.

	Added "relocation" information.  Relocation entries go in the chunk
	header along with region descriptors.  This allows us to identify
	chunks of data which need to be absolute disk blocks instead of
	offsets from the containing partition.  This is now used for BSD-slice
	partition tables which contain absolute disk blocks.  We can now
	create an image in one slice and reload it into another slice.

	Allow zlib compression level 0 (no compression).  This might be
	useful on machines that have slow CPUs: do just FS-compression and
	transfer the image elsewhere faster where it could be re-zipped
	with regular compression.

	Fix goof.  Previously we were not saving any DOS partition with
	an unrecognized type.  We should be naively compressing it instead.
	This is what we now do.  We continue to skip partitions of type 0
	("unused").

	mikeism: add handler for SIGINFO (^T) to report progress of
	a zip-age.

	Added everybody's favorite "dots" mode for reporting progress.

	Eliminate some excess copies left over from the conversion from
	write-every-little-piece to buffer-up-a-full-chunk-and-then-write.

	Eliminaged the special case handling of no skips (ranges) in
	compress_image by creating a single allocated range describing
	the whole disk/partition in this case.

	For NTFS, make the behavior of calling missing unicode routines
	be to return an error rather than exit.  These calls happen,
	but their failing doesn't seem to be fatal.

	Lots of typical mike-pissing on everything else.

imageunzip:

	Modify to handle both V1 and V2 images.

	In slice mode, make sure we don't write past the bounds of
	the slice.  ES&D if we try.

	Make output to unseekable devices work again (broken when
	pwrite was added)

	Add debug -F (Frisbee) option to randomize the presentation of
	chunks to the unzip/write threads.  Used to simulate frisbee.

	Add "-T DOS-type" option to tell imageunzip, when in slice mode,
	to set the type of the slice in the DOS partition table.
	This is useful if you are dropping say a BSD filesystem into
	an unused slice, you don't have to go back later and set this
	with fdisk.  Considered making this info part of the image
	itself (recorded by imagezip when creating a slice image),
	but decided against it.

	writezero takes an off_t for the size, we can be asked to write
	many gigabytes of zero at the end of a disk.

	Turn off dots mode by default.  Ya wanna see spots?  Ya gotta
	turn it on!

	Lots of typical mike-pissing on everything else.

imagedump:

	New tool for checking/dumping the structure of an image and
	reporting stats about it.
parent 2e77122f
......@@ -17,7 +17,7 @@ PTHREADCFLAGS = -D_THREAD_SAFE \
-I/usr/local/include/pthread/linuxthreads
PTHREADLIBS = -L/usr/local/lib -llthread -llgcc_r
CFLAGS = -O2 -g -static
CFLAGS = -Wall -O2 -g -static
LIBS = -lz $(PTHREADLIBS)
UNZIPCFLAGS = $(CFLAGS) $(PTHREADCFLAGS) -Wall
UNZIPLIBS = $(LIBS) $(PTHREADLIBS)
......@@ -33,7 +33,7 @@ LIBS += -Lntfs -lntfs
NTFSDIR = ntfs
endif
all: imagezip imageunzip frisbee.o
all: imagezip imageunzip imagedump frisbee.o
include $(TESTBED_SRCDIR)/GNUmakerules
......@@ -46,13 +46,16 @@ imageunzip: imageunzip.o version.o
imageunzip.o: imageunzip.c
$(CC) -c $(UNZIPCFLAGS) -o imageunzip.o $<
imagedump: imagedump.o version.o
$(CC) $(CFLAGS) imagedump.o version.o $(LIBS) -o imagedump
frisbee.o: imageunzip.c
$(CC) -c $(UNZIPCFLAGS) -DFRISBEE -o frisbee.o $<
ntfs:
@$(MAKE) -C ntfs all
version.c: imagezip.c imageunzip.c
version.c: imagezip.c imageunzip.c imagedump.c
echo >$@ "char build_info[] = \"Built `date +%d-%b-%Y` by `id -nu`@`hostname | sed 's/\..*//'`:`pwd`\";"
install: $(INSTALL_BINDIR)/imagezip $(INSTALL_BINDIR)/imageunzip
......
Things to do for netdisk and imagezip:
Things to do for image*:
1. Need an unimagezip.
1. Checksum chunks or the entire image file?
Maybe just as a debug option to check for bugs in the zipper itself.
In general use, we will be using TCP as the transport for image files
or we will be using frisbee and the UDP checksum in conjunction with
frisbee itself tracking blocks should be sufficient. Anyway, doing
an entire image checksum would be complicated by frisbee's out of order
receipt of chunks.
2. Header should include a checksum of the data.
Should be a header checksum as well.
2. Imagezip could be multithread so that we can be reading ahead on the
input device and overlapping IO with compression. Maybe a third thread
for doing output.
3. Netdisk should have an option to make a checksum pass over the disk
after the image has been loaded.
3. In imagezip, split out the FS-specific code into subdirectories.
4. Netdisk should have a command to print out the DOS partition table and
any embedded disklabels.
4. Write an imageconvert so that we can convert old version images into
new ones and maybe change the compression level used in an image.
5. Netdisk should should have a non-interactive mode, taking commands as
arguments on the command line.
6. When interactive, netdisk should possibly not reboot after loading the
image, allowing one to load another partition or check on the partition
table.
/*
* EMULAB-COPYRIGHT
* Copyright (c) 2000-2002 University of Utah and the Flux Group.
* All rights reserved.
*/
/*
* Usage: imagedump <input file>
*
* Prints out information about an image.
*/
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <assert.h>
#include <unistd.h>
#include <string.h>
#include <zlib.h>
#include <sys/stat.h>
#include "imagehdr.h"
static int detail = 0;
static int dumpmap = 0;
static int infd = -1;
static unsigned long long wasted;
static uint32_t sectinuse;
static uint32_t sectfree;
static void usage(void);
static void dumpfile(char *name, int fd);
static int dumpchunk(char *name, char *buf, int chunkno);
int
main(int argc, char **argv)
{
int ch, version = 0;
extern char build_info[];
while ((ch = getopt(argc, argv, "dmv")) != -1)
switch(ch) {
case 'd':
detail++;
break;
case 'm':
dumpmap++;
detail = 0;
break;
case 'v':
version++;
break;
case 'h':
case '?':
default:
usage();
}
argc -= optind;
argv += optind;
if (version || detail) {
fprintf(stderr, "%s\n", build_info);
if (version)
exit(0);
}
if (argc < 1)
usage();
while (argc > 0) {
int isstdin = !strcmp(argv[0], "-");
if (!isstdin) {
if ((infd = open(argv[0], O_RDONLY, 0666)) < 0) {
perror("opening input file");
exit(1);
}
} else
infd = fileno(stdin);
dumpfile(isstdin ? "<stdin>" : argv[0], infd);
if (!isstdin)
close(infd);
argc--;
argv++;
}
exit(0);
}
static void
usage(void)
{
fprintf(stderr, "usage: "
"imagedump options <image filename> ...\n"
" -v Print version info and exit\n"
" -d Turn on progressive levels of detail\n");
exit(1);
}
static char chunkbuf[SUBBLOCKSIZE];
static unsigned int magic;
static unsigned long chunkcount;
static uint32_t nextsector;
static void
dumpfile(char *name, int fd)
{
unsigned long long tbytes, dbytes, cbytes;
int count, chunkno, first = 1;
off_t filesize;
int isstdin;
char *bp;
isstdin = (fd == fileno(stdin));
wasted = sectinuse = sectfree = 0;
nextsector = 0;
if (!isstdin) {
struct stat st;
if (fstat(fd, &st) < 0) {
perror(name);
return;
}
if ((st.st_size % SUBBLOCKSIZE) != 0)
printf("%s: WARNING: "
"file size not a multiple of chunk size\n",
name);
filesize = st.st_size;
} else
filesize = 0;
for (chunkno = 0; ; chunkno++) {
bp = chunkbuf;
if (isstdin)
count = sizeof(chunkbuf);
else {
count = DEFAULTREGIONSIZE;
if (lseek(infd, (off_t)chunkno*sizeof(chunkbuf),
SEEK_SET) < 0) {
perror("seeking on zipped image");
return;
}
}
/*
* Parse the file one chunk at a time. We read the entire
* chunk and hand it off. Since we might be reading from
* stdin, we have to make sure we get the entire amount.
*/
while (count) {
int cc;
if ((cc = read(infd, bp, count)) <= 0) {
if (cc == 0)
goto done;
perror("reading zipped image");
return;
}
count -= cc;
bp += cc;
}
if (first) {
blockhdr_t *hdr = (blockhdr_t *)chunkbuf;
magic = hdr->magic;
if (magic < COMPRESSED_MAGIC_BASE ||
magic > COMPRESSED_MAGIC_CURRENT) {
printf("%s: bad version %x\n", name, magic);
return;
}
chunkcount = hdr->blocktotal;
if ((filesize / SUBBLOCKSIZE) != chunkcount) {
if (chunkcount == 0)
printf("%s: WARNING: zero chunk count,"
" ignoring block fields\n",
name);
else if (isstdin)
filesize = (off_t)chunkcount *
SUBBLOCKSIZE;
else
printf("%s: WARNING: "
"file size inconsistant with "
"chunk count (%lu != %lu)\n",
name,
(unsigned long)
(filesize/SUBBLOCKSIZE),
chunkcount);
}
printf("%s: %qu bytes, %lu chunks, version %d\n",
name, filesize,
(unsigned long)(filesize / SUBBLOCKSIZE),
hdr->magic - COMPRESSED_MAGIC_BASE + 1);
first = 0;
}
if (dumpchunk(name, chunkbuf, chunkno))
break;
}
done:
if (filesize == 0)
filesize = (off_t)(chunkno + 1) * SUBBLOCKSIZE;
cbytes = (unsigned long long)(filesize - wasted);
dbytes = (unsigned long long)sectinuse * SECSIZE;
tbytes = (unsigned long long)(sectinuse + sectfree) * SECSIZE;
if (detail > 0)
printf("\n");
printf(" %qu bytes of overhead/wasted space (%5.2f%% of image file)\n",
wasted, (double)wasted / filesize * 100);
printf(" %qu bytes of compressed data\n",
cbytes);
printf(" %5.2fx compression of allocated data (%qu bytes)\n",
(double)dbytes / cbytes, dbytes);
printf(" %5.2fx compression of total known disk size (%qu bytes)\n",
(double)tbytes / cbytes, tbytes);
}
static int
dumpchunk(char *name, char *buf, int chunkno)
{
blockhdr_t *hdr;
struct region *reg;
int i;
hdr = (blockhdr_t *)buf;
switch (hdr->magic) {
case COMPRESSED_V1:
reg = (struct region *)((struct blockhdr_V1 *)hdr + 1);
break;
case COMPRESSED_V2:
reg = (struct region *)((struct blockhdr_V2 *)hdr + 1);
break;
default:
printf("%s: bad magic (%x!=%x) in chunk %d\n",
name, hdr->magic, magic, chunkno);
return 1;
}
if (chunkcount && hdr->blockindex != chunkno) {
printf("%s: bad chunk index (%d) in chunk %d\n",
name, hdr->blockindex, chunkno);
return 1;
}
if (chunkcount && hdr->blocktotal != chunkcount) {
printf("%s: bad chunkcount (%d!=%lu) in chunk %d\n",
name, hdr->blocktotal, chunkcount, chunkno);
return 1;
}
if (hdr->size > (SUBBLOCKSIZE - hdr->regionsize)) {
printf("%s: bad chunksize (%d > %d) in chunk %d\n",
name, hdr->size, SUBBLOCKSIZE-hdr->regionsize, chunkno);
return 1;
}
#if 1
/* include header overhead */
wasted += SUBBLOCKSIZE - hdr->size;
#else
wasted += ((SUBBLOCKSIZE - hdr->regionsize) - hdr->size);
#endif
if (detail > 0) {
printf(" Chunk %d: %u compressed bytes, ",
chunkno, hdr->size);
if (hdr->magic > COMPRESSED_V1) {
printf("sector range [%u-%u], ",
hdr->firstsect, hdr->lastsect-1);
if (hdr->reloccount > 0)
printf("%d relocs, ", hdr->reloccount);
}
printf("%d regions\n", hdr->regioncount);
}
if (hdr->regionsize != DEFAULTREGIONSIZE)
printf(" WARNING: "
"unexpected region size (%d!=%d) in chunk %d\n",
hdr->regionsize, DEFAULTREGIONSIZE, chunkno);
for (i = 0; i < hdr->regioncount; i++) {
if (detail > 1)
printf(" Region %d: %d sectors [%u-%u]\n",
i, reg->size, reg->start,
reg->start + reg->size - 1);
if (reg->start < nextsector)
printf(" WARNING: chunk %d region %d "
"may overlap others\n", chunkno, i);
if (hdr->magic > COMPRESSED_V1) {
if (i == 0) {
if (hdr->firstsect > reg->start)
printf(" WARNING: chunk %d bad "
"firstsect value (%u>%u)\n",
chunkno, hdr->firstsect,
reg->start);
else
sectfree +=
(reg->start - hdr->firstsect);
} else
sectfree += (reg->start - nextsector);
if (i == hdr->regioncount-1) {
if (hdr->lastsect < reg->start + reg->size)
printf(" WARNING: chunk %d bad "
"lastsect value (%u<%u)\n",
chunkno, hdr->lastsect,
reg->start + reg->size);
else
sectfree += (hdr->lastsect -
(reg->start+reg->size));
}
} else
sectfree += (reg->start - nextsector);
sectinuse += reg->size;
if (dumpmap) {
switch (hdr->magic) {
case COMPRESSED_V1:
if (reg->start - nextsector != 0)
printf("F: [%08x-%08x]\n",
nextsector, reg->start-1);
printf("A: [%08x-%08x]\n",
reg->start, reg->start + reg->size - 1);
break;
case COMPRESSED_V2:
if (i == 0 && hdr->firstsect < reg->start)
printf("F: [%08x-%08x]\n",
hdr->firstsect, reg->start-1);
if (i != 0 && reg->start - nextsector != 0)
printf("F: [%08x-%08x]\n",
nextsector, reg->start-1);
printf("A: [%08x-%08x]\n",
reg->start, reg->start + reg->size - 1);
if (i == hdr->regioncount-1 &&
reg->start+reg->size < hdr->lastsect)
printf("F: [%08x-%08x]\n",
reg->start+reg->size,
hdr->lastsect-1);
break;
}
}
nextsector = reg->start + reg->size;
reg++;
}
if (hdr->magic == COMPRESSED_V1)
return 0;
for (i = 0; i < hdr->reloccount; i++) {
struct blockreloc *reloc = (struct blockreloc *)reg;
uint32_t offset = reloc->sector * SECSIZE + reloc->sectoff;
if (reloc->sector < hdr->firstsect ||
reloc->sector >= hdr->lastsect)
printf(" WARNING: "
"Reloc %d at %u not in chunk [%u-%u]\n", i,
reloc->sector, hdr->firstsect, hdr->lastsect-1);
if (detail > 1)
printf(" Reloc %d: %s [%u-%u] (sector %d)\n", i,
reloc->type == RELOC_BSDDISKLABEL ?
"BSDDISKLABEL" : "??",
offset, offset + reloc->size, reloc->sector);
}
return 0;
}
......@@ -3,8 +3,16 @@
/*
* Magic number when image is compressed
*
* This magic number has been commandeered for use as a version number.
* None of this wimpy start at version 1 stuff either, our first version
* is 1,768,515,945!
*/
#define COMPRESSED_MAGIC 0x69696969
#define COMPRESSED_MAGIC_BASE 0x69696969
#define COMPRESSED_V1 (COMPRESSED_MAGIC_BASE+0)
#define COMPRESSED_V2 (COMPRESSED_MAGIC_BASE+1)
#define COMPRESSED_MAGIC_CURRENT COMPRESSED_V2
/*
* Each compressed block of the file has this little header on it.
......@@ -12,14 +20,59 @@
* its internal size (it will probably be shorter than 1MB) since we
* have to know exactly how much to give the inflator.
*/
struct blockhdr {
int magic;
unsigned long size; /* Size of compressed part */
int blockindex;
int blocktotal;
int regionsize;
int regioncount;
struct blockhdr_V1 {
uint32_t magic;
uint32_t size; /* Size of compressed part */
int32_t blockindex;
int32_t blocktotal;
int32_t regionsize;
int32_t regioncount;
};
/*
* Version 2 of the block descriptor adds a first and last sector value.
* These are used to describe free space which is adjacent to the allocated
* sector data. This is needed in order to properly zero all free space.
* Previously free space between regions that wound up in different
* blocks could only be handled if the blocks were presented consecutively,
* this was not the case in frisbee.
*/
struct blockhdr_V2 {
uint32_t magic; /* magic/version */
uint32_t size; /* Size of compressed part */
int32_t blockindex; /* netdisk: which block we are */
int32_t blocktotal; /* netdisk: total number of blocks */
int32_t regionsize; /* sizeof header + regions */
int32_t regioncount; /* number of regions */
/* V2 follows */
uint32_t firstsect; /* first sector described by block */
uint32_t lastsect; /* last sector described by block */
int32_t reloccount; /* number of reloc entries */
};
/*
* Relocation descriptor.
* Certain data structures like BSD disklabels and LILO boot blocks require
* absolute block numbers. This descriptor tells the unzipper what the
* data structure is and where it is located in the block.
*
* Relocation descriptors follow the region descriptors in the header block.
*/
struct blockreloc {
uint32_t type; /* relocation type (below) */
uint32_t sector; /* sector it applies to */
uint32_t sectoff; /* offset within the sector */
uint32_t size; /* size of data affected */
};
#define RELOC_NONE 0
#define RELOC_BSDDISKLABEL 1 /* BSD disklabel */
/* XXX potential future alternatives to hard-wiring BSD disklabel knowledge */
#define RELOC_ADDPARTOFFSET 100 /* add partition offset to location */
#define RELOC_XOR16CKSUM 101 /* 16-bit XOR checksum */
#define RELOC_CKSUMRANGE 102 /* range of previous checksum */
typedef struct blockhdr_V2 blockhdr_t;
/*
* This little struct defines the pair. Each number is in sectors. An array
......@@ -29,8 +82,8 @@ struct blockhdr {
* (swap, free FS blocks).
*/
struct region {
unsigned long start;
unsigned long size;
uint32_t start;
uint32_t size;
};
/*
......@@ -50,3 +103,8 @@ struct region {
*/
#define SUBBLOCKSIZE (1024 * 1024)
#define SUBBLOCKMAX (SUBBLOCKSIZE - DEFAULTREGIONSIZE)
/*
* Assumed sector (block) size
*/
#define SECSIZE 512
This diff is collapsed.
This diff is collapsed.
......@@ -4,15 +4,12 @@
#include <unistd.h>
size_t wcrtomb(char *s, wchar_t wc, mbstate_t *ps) {
fprintf(stderr,"wcrtomb not implemented\n");
exit(1);
fprintf(stderr, "WARNING: wcrtomb not implemented\n");
return -1;
}
int mbsinit(const mbstate_t *ps) {
fprintf(stderr,"mbsinit not implemented\n");
fprintf(stderr,"ps==%p\n",ps);
exit(1);
fprintf(stderr, "WARNING: mbsinit not implemented\n");
return 0;
}
......@@ -21,14 +18,12 @@ int fdatasync(int fd) {
}
size_t mbsrtowcs(wchar_t *dest, const char **src, size_t len, mbstate_t *ps) {
fprintf(stderr,"mbsrtowcs not implemented\n");
exit(1);
return 0;
fprintf(stderr, "WARNING: mbsrtowcs not implemented\n");
return -1;
}
size_t mbrtowc(wchar_t *pwc, const char *s, size_t n, mbstate_t *ps) {
fprintf(stderr,"mbrtowc not implemented\n");
exit(1);
fprintf(stderr, "WARNING: mbrtowc not implemented\n");
return 0;
}
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment