GitLab

There are a couple of new packet types in the frisbee protocol which are
exchanged via TCP with the master server: GETREQUEST and GETREPLY.  The
client passes to the master server an opaque imageid and a couple of options
and gets back the addr/port to use to actually download the image.  The
implementation of the master server is fragile and is more of a test
framework, Grant is working on a more robust master server.  I am mostly
doing a backend that communicates with the Emulab DB to do its authentication
and making the client changes.

The client now uses the -S option to specify the IP address of the master
server and the -F option to specify an imageid.  If no error is returned,
the image is downloaded using the returned addr/port.  If -Q is used in place
of -F, then the client makes a "status only" call getting back info about
whether the named image is accessible to the client and whether a server is
currently running.

On the server side, the new master server (mserver.c) has an Emulab
configuration "backend" that supports host-based authentication.
The IP address of the caller is mapped to a node_id/pid/gid/eid combo
that is used to determine access.  On a request, the specified imageid is
treated either as a pathname (if it starts with '/') or an image identifier
of the form "<pid>/<imagename>".  If it is a pathname, we check to make
sure that pathname (after running through "realpath") is contained in one
of the directories accessible to that node in its current experiment context;
i.e., /share, /proj/<pid>, /groups/<pid>/<gid>, or /users/<swapper-uid>.
If it is an image identifier, the DB is queried to ensure that access is
allowed to that image; i.e., it must be "global" or in the appropriate
project/group.

The master server forks a frisbeed for each valid request, if one is not
already running.  The multicast address selection is still based on the
emulab_indicies.frisbee_index field, but the address/port/server info is no
longer stored in the frisbee_blobs table (frisbee_pid, load_address,
load_busy are not set).

Note that this is not yet integrated in the os_load path.  Further work is
required to replace frisbeelauncher.

The "-u" option depends on GNU tar and will do the incorrect thing
with BSD tar.  Thus, try to find "gtar" and use it if available, if
not use bsd tar and print a warning if the rarely used "-u" option is
used.

"tb-set-tarfiles" is like "tb-set-node-tarfiles" except that it
distributes the tarfile to all nodes rather than just one and that it
uses frisbee to distribute the file.

These changes involved 1) refactoring frisbee info from images table
into a new table, frisbee_blobs, 2) a new experiment_blobs table, and
3) a new tmcd command so the node knows how to get the files from the
server.

The changes where designed to be general purpose enough to eventually
support:
  1) Distributing arbitrary files (not just tarfiles) to nodes
  2) Perform arbitrary actions on those files
  3) Use arbitrary methods to get the files

As such the tmcd line is as follows:
  URL=* ACTION=*

where URL is currently:
  frisbee.mcast://<ADDR>/<FILE>
for example
  frisbee.mcast://234.16.184.192:18092/users/kevina/home-dir.tar.gz
and when we get around to using a master Frisbee server it could be
  frisbee://*
or it could be a file://, http://, etc.

and ACTION is currently:
  unpack:<LOCATION>
for example
  unpackt:/users
with future syntax to be determined.

I was using v2 fields even when I got a v1 reply.

Basically, make it possible to transfer a non imagezip image.  Previously
you had to wrap a regular file as an image in order to transfer it.  The
big hang up was that the frisbee protocol could only transfer files that
were a multiple of 1MB (the chunk size).

This commit changes the frisbee protocol slightly to allow transfer of
non-1MB-multiple files.  The protocol change was to add a new JOIN message
that returns the size of the file in bytes rather than in blocks.  This
allows the client to know that the file in question is not a multiple of 1MB
and allows it to request the correct partial number of blocks for the
final chunk and to extract the correct amount of data from the final 1K block
(that block is still padded to 1K by the server).  For the server side, the
request mostly allows it to do some sanity checking.  The fact that the
server is started with a file that is not a multiple of 1MB is what triggers
it to know about partial chunks.  The sanity checking is that the server will
not acknowledge clients that attempt to join with a version 1 JOIN message,
since nothing good would come of that pairing.

On the client side, frisbee must be invoked with the -N (nodecompress) option
in order to issue a v2 JOIN.  See the comment in the code for the rationale,
but it is largely a backward compat feature.

While I was changing the JOIN message, I added a couple of other future
features.  One is that by passing back a 64-bit value for the size of the
image in bytes, we can feed bigger images.  However there is still much to
be done to realize this.  The other was to add blocksize/chunksize fields
in the message so that the server/client can negotiate the transfer parameters,
e.g., 1024 blocks of 1024 bytes vs. 256 blocks of 8192 bytes, the latter being
for "jumbo" packets on a Gb ethernet.  But there is still more to be done to
get this working too.

This was blowing up the '-N' ("just save it") option of frisbee on big
files.

This is a brute-force hack.  We fail on ANY overlap even though some overlaps
(e.g., with unused partitions) could be handled.  But handling those is a lot
of work for little gain.

Also: removed one last reference to SHD.

Another idea whose time never came...

Instead, print the usage string using "%s".  This is a safety bug found by
-Wformat.

"ifdef FOO" is always true if FOO is not the empty value, so use 'ifeq
(FOO, 1)" instead when checking if support for each filesystem is
enabled.

The only reason it didn't before was because we didn't have a "portable"
version of disklabel.h.  We have had that for quite some time now...

This is a highly specialized (aka, useless) command line option I put in
just to get a sense of how common shared chunks between images are; to help
answer the question: could a frisbee server that serves multiple images take
advantage of this to significant effect?

at least with FBSD 8.x (or maybe it is "geom"?), they became relative to the
slice.  So we attempt to automatically differentiate them by looking for slice
starts that are less than the DOS partition offset.  Such a slice would
indicate relative addressing.  Should probably add an explicit option to
set/unset absolute addressing in case the heuristic is wrong.

compatible.  One important part: the new versions get rid of the infamous
"advertising clause" which has been gone from BSD files for a long time...

EXT4 specific changes done.

Added EXT4 specific directives

EXT4 support for imagezip

ext4 filesystem support for imagezip

is happening.  Also add a simple perl script to do a little interpretation
of the output stats.

Support for jumbo packets.  Setting WITH_JUMBO on the make command line
will change the image block size to 8192 bytes and reduces the number of
block per chunk to 256 (to maintain the 1MB chunk size for compat with old
images).  The default is still 1024.

Added the notion of a "dubious" chunk buffer in the client.  If an incoming
chunk buffer is marked as CHUNK_DUBIOUS, then its contents can be evicted and
the buffer reused for a more promising chunk.  This is a crude replacement
mechanism that is currently only used in one place: if we miss part of a
chunk and the server switches to sending a new chunk for which we have no
free buffer, we switch to collecting the new chunk.  The reasoning is that
it will take a while for the server to switch back to completing the former
chunk, during which time it may send one or more complete chunks that we
could more fruitfully use (decompress and write out).

Changed the meaning of the "done" field for a chunk.  It used to mean either
that we have completely processed the chunk or that we are currently collecting
it.  It took additional work (scanning all chunk buffers) to differentiate
these cases, so I make it explicit.

Allow the client and server to dynamically determine the maximum socket
buffer size.

Fix a couple more on-the-wire data structure size/alignment issues that
showed up on a 64-bit OS.

A few minor speedups to the bitmap handling code.  Think: "rearranging deck
chairs on the Titanic" here.  We need more serious algorithmic changes
to scale all this code going forward.

Add some more TRACE events and refine what is already there.

Added some hacks to allow frisbee client/server to run on the same machine.
We had made it remarkably hard to do this.  But then again, why would you
want to!  Look for SAME_HOST_HACK in the makefile.

 * lint: compile cleanly under newer GCC and on 64-bit OS
 * rename SUBBLOCKSIZE to CHUNKSIZE since that is what it is
 * collect some more stats, primarily for frisbee event collection code
 * add stupid hacky bubble-sort workaround but don't define it by default
   cuz it is stupid...and hacky...but it allows mike to make an image of
   his home machine before he dies of old age
 * tweaks to libntfs version support

Change '__packed' to '__attribute__((__packed__))'.  Linux doesn't
define the former, and FreeBSD defines it as the latter.  Just use the
longer form so that it works with both.

files (as opposed to from disk partitions).  This is almost identical to
rawmode, but also knows how to handle inputs that are not a multiple of the
sector size.  Imagezip will pad out the final sector, but uses a special
relocation entry (SHORTSECTOR) to tell imageunzip/frisbee how many bytes
to write in that final sector.

This is "pretty much" backward compatible.  If an old version of imageunzip
encounters the unknown new relocation entry, it will ignore it, resulting
in imageunzip just writing out the additional pad bytes to the output file.
This may or may not screw over the resulting output file.

Also fixed some lint found by GCC 4.2 on FBSD7.