1. Implement PREQUEST message which passes a bit map of desired blocks. We still use the REQUEST message (start block + number of blocks) for full chunk requests as that is more efficient. This message also includes a flag indicating whether it is a retry of a request we originally made or not. This gives the server more accurate loss info. 2. More stats and tracing goo. Frisbee client: 1. Add 'C' and 'W' command line options to specify amount of memory for chunk buffers (network buffering) and for write buffers (disk buffering). The Emulab frisbee startup script uses these to partition up all the available memory on a machine. Previously we were just using a fixed ~128MB even though our machines have 256 or 512MB of memory. Also add the 'M' option which specifies the overall memory, internally dividing it up between chunk buffers and write buffers. 2. Add 'S' command line option to explicitly specify the server. This allows us to make a feeb...um, "lightweight" authentication check on incoming messages. 3. Use the common BlockMap data struct to track which pieces of a chunk we have received. This is easily inverted to make PREQUESTS and it is also smaller than the older byte-per-block technique. 4. Allow partial request-ahead. Previously, we only issued request-ahead if there were enough empty chunk buffers for a maximum (2) request-ahead. Frisbee server: 1. Use BlockMap for workQ elements. An easy way to allow a complete merge of incoming requests with existing ones. 2. Check for overlap of incoming requests with the request currently being serviced. This happens surprisingly often. 3. Dubious: burst gap becomes burst interval. The latter takes into account the time required to read data, etc., in other words, we now have variable-sized gaps and put out bursts at specific times rather than having fixed gaps and putting out bursts at variable times. This gives us more accurate pacing over shorter time periods. I thought this might be important for dynamic pacing. 4. Add 'W' command line option to specify a target bandwidth. Frisbeed will use this to calculate a burst size/interval. 5. Rewrote the dynamic pacing code. It is now easily as bad as before if not worse. But it does have fewer magic constants! Needs to be redone by someone who understands the TCP-friendly rate equation. Imagezip: 1. add 'R' option to specify one or more partitions for which to force raw (naive) compression even if the FS format is understood. Useful for benchmarking. 2. add 'D' option to allow "dangerous" writes. In this mode, we don't do the fsync's or retries of failed writes. Overrides the hack we put in for NFS. Use this if writing to a local filesystem (or /dev/null). 3. Eliminate an extra copy of every chunk header. Imageunzip: 1. Eliminate extra copy of decompressed data that we were doing between the single decompression buffer and the disk buffers. Helps on slow machines (like gatech's 300Mhz machines with 66MHz memory bus). 2. Allow dynamic number of variable-sized write buffers. Total memory not to exceed the writebufmem limit. Previously we had a small number of fixed-size (256K) buffers. 3. Add debugging 'C' option to just compute a single CRC of the decompressed image. Back-ported to older imageunzip and used to make sure my write buffer changes were correct. Maybe handy for similar massive changes in the future.
Showing with 2322 additions and 608 deletions