1. 11 Oct, 2011 3 commits
    • Ryan Jackson's avatar
      Merge imagezip ext4 code into extfs code · b8d79ac6
      Ryan Jackson authored
      Merge the ext4 support for imagezip into the extfs code.  There's no
      real reason to keep it separate, since ext4 is backward-compatible
      with ext3 and ext2.
      All of the macro definitions in extfs.c were changed from EXT2_* or
      EXT3_* to EXT4_*.  The primary reason for this is that while ext4 is
      backward compatible, the data structures have been extended and some
      of the definitions needed to change to handle them.  Some things that
      were constant before (block group descriptor sizes, for example) are
      now dynamic and must be calculated from other fields in the
      Since imagezip and frisbee don't support 64-bit block numbers, ext4
      filesystems that are larger than 2 TB (assuming 512-byte sectors) are
      not supported.  These can be detected by examining the
      feature_incompat field of the superblock.  Imagezip will fail if the
      user tries to compress one of these filesystems.
    • Mike Hibler's avatar
      Switch to using version 2.0.0 of the libntfs library for Windows images. · 5d115d2c
      Mike Hibler authored
      I ran about 50 Windows images through this to "verify" that it produces the
      same results as the old 1.7.1 library.
    • Mike Hibler's avatar
      Add -Z option to zero free space included in the image due to -F. · 5a02dd31
      Mike Hibler authored
      Previously, this "internal free space" was whatever garbage happened to
      be on the disk we were imaging. By zeroing, we eliminate any leakage of
      information from the source disk and also allow the image to compress just
      a tad (1-4%) better.
      Why isn't this the default? Eh...no good reason, other than that this whole
      squish-out-small-free-ranges-to-allow-longer-writes optimization should be
      done by the client and should not be encoded in the image.
  2. 08 Oct, 2011 2 commits
    • Mike Hibler's avatar
      Revert "Adjust the set of unix gids used for a download server." · baba7478
      Mike Hibler authored
      This reverts commit fc89eb38.
      Checked in a bunch of crap that was unrelated.
    • Mike Hibler's avatar
      Adjust the set of unix gids used for a download server. · fc89eb38
      Mike Hibler authored
      When downloading an image, start the frisbeed process with the minimum set of
      gids necessary to access the image. This includes the unix gid of the
      project that the image is in and, optionally, the unix gid of the project
      subgroup if the image is part of one.
      Previously, we just use the gid set of the uid of the swapper of the
      experiment. Not only was this excessive, but it might also not include the
      gids needed in the case of a "global" image that is not in the world-readable
      /usr/testbed/images directory.
  3. 29 Sep, 2011 1 commit
  4. 28 Sep, 2011 3 commits
    • Mike Hibler's avatar
      Update with a couple of items that came up recently. · 1f54baef
      Mike Hibler authored
      Also update the DONE status on a few things.
    • Mike Hibler's avatar
      Add client '-f' option for using O_DIRECT open mode for the output device. · 46a62612
      Mike Hibler authored
      On Linux, device IO goes through the buffer cache by default. This makes
      frisbee run really fast...until it closes the output device. Then it sits
      for minutes while it flushes disk data out of the cache. This is
      technically okay, but wasteful, since frisbee allocates its own memory
      for caching disk write data. By using direct IO on the output device,
      writes do not go through the cache.
      Aren't two caches better than one? No. They can compete for memory and it
      just causes an extra data copy. Frisbee is faster when using O_DIRECT.
      Note that this change is complicated somewhat because Linux requires that
      the IO buffer for an O_DIRECT opened file be sector aligned. So we play
      some games to do this.
      This should have no effect on FreeBSD where device writes don't go through
      the buffer cache.
    • Mike Hibler's avatar
      Avoid a pre-mature close of the output fd. · 179c189f
      Mike Hibler authored
      Wait til all other threads have joined.
  5. 30 Aug, 2011 1 commit
  6. 26 Aug, 2011 1 commit
  7. 25 Aug, 2011 1 commit
  8. 16 Aug, 2011 1 commit
  9. 13 Jul, 2011 1 commit
  10. 06 Jul, 2011 1 commit
    • Mike Hibler's avatar
      Add -f option for reporting hashes. · bfaa986c
      Mike Hibler authored
      This forces imagehash to hash pieces on aligned sector boundaries rather
      than just doing it from the start of each chunk. Result now is that there
      may be a partial hash piece at the beginning of each range and another
      partial piece at the end. Ideally, we would avoid those partial pieces,
      but we are only using this as a point of comparison right now.
  11. 15 Jun, 2011 1 commit
    • Ryan Jackson's avatar
      Change rindex() to strrchr() · 4c75d2cf
      Ryan Jackson authored
      Default builds of uClibc (used for the Linux MFS) don't include
      support for index() and rindex() (gone from the POSIX standard as of
      POSIX.1-2008).  I've fixed the uClibc config to include them now, but
      I need to build frisbee for an existing build of the Linux MFS with
      the old uClibc.
  12. 02 Jun, 2011 1 commit
  13. 01 Jun, 2011 1 commit
  14. 31 May, 2011 2 commits
    • Mike Hibler's avatar
      Added compat option and assorted cleanups. · d8de4d6a
      Mike Hibler authored
      Added WITH_V3COMPAT to make sure that we can still generated V3 images
      (for other sites) if checksum/encryption is not used.
      Try to clean up the command line options some. Be more consistent by putting
      generated uuid into a file instead of just spitting it out on stderr. Make
      sure that if the decryption specifies signing and/or encryption that we
      require the image to have that info. Add some more assertions. (Re)distinguish
      checksums from signed-checksums.
    • Mike Hibler's avatar
  15. 27 May, 2011 1 commit
  16. 18 May, 2011 1 commit
  17. 04 May, 2011 2 commits
  18. 17 Jan, 2011 1 commit
    • Mike Hibler's avatar
      Random Windows (Cygwin) fixes. · 975cc86e
      Mike Hibler authored
      Client-side builds again, but haven't got node to boot correctly.
      Need to get pubsubd installed correctly as a service in place of elvind.
  19. 29 Oct, 2010 1 commit
    • Mike Hibler's avatar
      Improve regular (non-image) file transfer via frisbee. · 78f3f8d3
      Mike Hibler authored
      Basically, make it possible to transfer a non imagezip image.  Previously
      you had to wrap a regular file as an image in order to transfer it.  The
      big hang up was that the frisbee protocol could only transfer files that
      were a multiple of 1MB (the chunk size).
      This commit changes the frisbee protocol slightly to allow transfer of
      non-1MB-multiple files.  The protocol change was to add a new JOIN message
      that returns the size of the file in bytes rather than in blocks.  This
      allows the client to know that the file in question is not a multiple of 1MB
      and allows it to request the correct partial number of blocks for the
      final chunk and to extract the correct amount of data from the final 1K block
      (that block is still padded to 1K by the server).  For the server side, the
      request mostly allows it to do some sanity checking.  The fact that the
      server is started with a file that is not a multiple of 1MB is what triggers
      it to know about partial chunks.  The sanity checking is that the server will
      not acknowledge clients that attempt to join with a version 1 JOIN message,
      since nothing good would come of that pairing.
      On the client side, frisbee must be invoked with the -N (nodecompress) option
      in order to issue a v2 JOIN.  See the comment in the code for the rationale,
      but it is largely a backward compat feature.
      While I was changing the JOIN message, I added a couple of other future
      features.  One is that by passing back a 64-bit value for the size of the
      image in bytes, we can feed bigger images.  However there is still much to
      be done to realize this.  The other was to add blocksize/chunksize fields
      in the message so that the server/client can negotiate the transfer parameters,
      e.g., 1024 blocks of 1024 bytes vs. 256 blocks of 8192 bytes, the latter being
      for "jumbo" packets on a Gb ethernet.  But there is still more to be done to
      get this working too.
  20. 28 Oct, 2010 1 commit
  21. 22 Oct, 2010 2 commits
  22. 20 Jul, 2010 2 commits
  23. 07 Jul, 2010 1 commit
  24. 06 Jul, 2010 1 commit
  25. 02 Jul, 2010 3 commits
  26. 21 Jun, 2010 1 commit
  27. 17 Jun, 2010 3 commits