1. 12 May, 2015 4 commits
  2. 03 Mar, 2015 1 commit
  3. 19 Feb, 2015 1 commit
    • Mike Hibler's avatar
      Make sure partition code doesn't add skips outside the desired range. · 56b94228
      Mike Hibler authored
      The partition code (MBR, GPT) was adding skips to the list for ranges
      that were outside of any partition. That is fine if we are making a whole
      disk image, but if were are making a single-partition image it would add
      sectors that were outside of the expected range triggering an assert.
      
      This commit affects a lot of files because I modified the API to the
      partition code, doing what I should have done originally. This actually
      cleaned up some code cooties.
      56b94228
  4. 30 Jan, 2015 1 commit
  5. 18 Jan, 2015 1 commit
  6. 18 Dec, 2014 1 commit
    • Mike Hibler's avatar
      More tweaks. · af8e9149
      Mike Hibler authored
      Mostly letting GPT or MBR decide how big the disk is and let it initially
      decide on areas (outside of partitions) that can be skipped.
      af8e9149
  7. 17 Dec, 2014 2 commits
  8. 16 Dec, 2014 1 commit
  9. 04 Dec, 2014 1 commit
  10. 13 Aug, 2014 1 commit
    • Mike Hibler's avatar
      If there are no valid data to save, exit with an error. · ad5e8d9e
      Mike Hibler authored
      E.g., telling it to save an unused (type 0) partition. Previously, we
      would create a single valid range with start at the beginning of disk
      (or partition) and a size of zero. However, size zero is taken by the
      imagezip compressor to mean "til EOF" so we would wind up creating a
      really large image instead of doing nothing.
      
      It is debatable whether this is really an error. Perhaps we should just
      create a valid image file with one chunk containing no valid ranges, but
      it seems likely that they probably made a mistake and we should call it
      out immediately.
      ad5e8d9e
  11. 17 Jul, 2014 1 commit
    • Mike Hibler's avatar
      Add support for auto-detecting when to create a delta image. · bc7c6e40
      Mike Hibler authored
      Uses the new-ish imagezip -P option. But first we had to get that option
      to imagezip. So the key-value interface between server and client now has
      an IZOPTS key where you can pass options to imagezip (This is now used to
      pass the -N that we have always specified as well).
      
      Also changed imagezip to print out a less ambiguous message when the option
      is in use, if will clearly say that it is full or delta. This matters because
      we have to parse the imagezip output on the server to figure out what kind
      of image it is!
      bc7c6e40
  12. 03 Jul, 2014 3 commits
  13. 27 Jun, 2014 1 commit
    • Mike Hibler's avatar
      Resurrect/enhance delta image code. Another multi-day "one hour hack"! · 49d2f488
      Mike Hibler authored
      Resurrect: get the basic signature matching code working again.
      
      Enhance: add -U option to have imagezip update (or create) the
      signature file. Previously, the signature file was created off-line
      on boss with the imagehash command (that would be Mike's "imagehash"
      (/usr/testbed/bin/imagehash) and not Leigh's "imagehash"
      (/usr/testbed/sbin/imagehash)). Creating it as we create the image
      makes a lot of sense...except for how we do it. We actually read and
      create the hashes as a separate pass before we re-read, compress, and
      create the image--so we read the disk twice. [This is primarily because we
      are mooching off of the existing hash checking code (-H option). Doing
      this right will require re-writing The Big Loop which makes a single
      pass through the data, simultaneously dealing with disk IO, allocated
      ranges, and compression blocks all of which have different size/alignment
      criteria. But I digress...] Anyway, reading the disk data twice sucks,
      but at least it is on the client and not on boss. The take away is:
      don't create your images on pc600s.
      
      Note that -U will always create a signature file for the complete disk
      or partition even when you are creating a delta image (i.e., when combined
      with -H).
      
      Enhance: add "-P <pct>" option, used with -H, which tells imagezip that if
      a resulting delta image would be over <pct> percent the size (where size
      is number of uncompressed sectors in the image) of a full image, then just
      create a full image instead. So "-P 50" says if it would be over half the
      size, "-P 200" says if it was over twice the size, etc. If you always want
      a delta image to be produced, use -H without -P. If you always want a full
      image, don't use -H.
      
      This is part 2 of supporting images. Part 1 is the DB and user interface
      changes that Leigh is working on. Part 3 is next up and involves modifying
      the image creation MFS to download and use signatures along with the
      new imagezip when creating images. Stay tuned.
      49d2f488
  14. 18 Jun, 2013 1 commit
    • Mike Hibler's avatar
      Check that fsync is supported on output file of imagezip. · 75b72804
      Mike Hibler authored
      Part of our hack strategy to detect and correct transient NFS errors
      is to fsync after every file write and check for errors. However, some
      devices don't support fsync. So, for example, if you were trying to
      imagezip to /dev/null (to benchmark or check the resulting image size)
      fsync would always fail and we would keep retrying the write forever.
      
      There is already a "-N" option to avoid such checks and retries, but
      Mike can never remember that!
      75b72804
  15. 05 Feb, 2013 2 commits
  16. 24 Oct, 2012 1 commit
    • Mike Hibler's avatar
      Add the -x "exclude non-FS" option to imagezip. · 1e5717a8
      Mike Hibler authored
      This option says to ignore any blocks which are outside of a Linux or FreeBSD
      filesystem but still in the containing partition; i.e., in the case where the
      filesystem is smaller than the partition it is in. By default, such space is
      (and always has been) considered "allocated."
      
      This should be extended to cover space that is outside of any FreeBSD or
      DOS partition as well (though we have to be careful not to exclude the MBR
      in that case). It should also be made to work with FAT and NTFS.
      1e5717a8
  17. 24 Sep, 2012 1 commit
    • Eric Eide's avatar
      Replace license symbols with {{{ }}}-enclosed license blocks. · 6df609a9
      Eric Eide authored
      This commit is intended to makes the license status of Emulab and
      ProtoGENI source files more clear.  It replaces license symbols like
      "EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
      blocks that contain actual license statements.
      
      This change was driven by the fact that today, most people acquire and
      track Emulab and ProtoGENI sources via git.
      
      Before the Emulab source code was kept in git, the Flux Research Group
      at the University of Utah would roll distributions by making tar
      files.  As part of that process, the Flux Group would replace the
      license symbols in the source files with actual license statements.
      
      When the Flux Group moved to git, people outside of the group started
      to see the source files with the "unexpanded" symbols.  This meant
      that people acquired source files without actual license statements in
      them.  All the relevant files had Utah *copyright* statements in them,
      but without the expanded *license* statements, the licensing status of
      the source files was unclear.
      
      This commit is intended to clear up that confusion.
      
      Most Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the Affero GNU General Public License, version 3
      (AGPLv3).
      
      Most Utah-copyrighted files related to ProtoGENI are distributed under
      the terms of the GENI Public License, which is a BSD-like open-source
      license.
      
      Some Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the GNU Lesser General Public License, version 2.1
      (LGPL).
      6df609a9
  18. 21 Aug, 2012 1 commit
  19. 08 Aug, 2012 1 commit
    • Mike Hibler's avatar
      Bug Fix: fix incorrect usage of "sizeof(blockhdr_t)". · 9f5bed29
      Mike Hibler authored
      The size of the block header differs depending on the version of the image.
      sizeof(blockhdr_t) gives the size of the current version of the header when
      the binary was built (V4 currently). This only mattered in two places:
      
      In imagezip, it would have prevented us from creating a backward compatible
      V1 image. Probably nobody has done this in 10 years.
      
      In imageunzip, it could prevent relocations from being processed. This only
      affected recent builds (V4 format) when attempting to unzip an older
      single-partition FreeBSD image (FBSD5 or before). This is unlikely to occur
      outside of Utah and only very rarely here.
      9f5bed29
  20. 20 Mar, 2012 1 commit
  21. 05 Mar, 2012 1 commit
  22. 11 Oct, 2011 2 commits
    • Ryan Jackson's avatar
      Merge imagezip ext4 code into extfs code · b8d79ac6
      Ryan Jackson authored
      Merge the ext4 support for imagezip into the extfs code.  There's no
      real reason to keep it separate, since ext4 is backward-compatible
      with ext3 and ext2.
      
      All of the macro definitions in extfs.c were changed from EXT2_* or
      EXT3_* to EXT4_*.  The primary reason for this is that while ext4 is
      backward compatible, the data structures have been extended and some
      of the definitions needed to change to handle them.  Some things that
      were constant before (block group descriptor sizes, for example) are
      now dynamic and must be calculated from other fields in the
      superblock.
      
      Since imagezip and frisbee don't support 64-bit block numbers, ext4
      filesystems that are larger than 2 TB (assuming 512-byte sectors) are
      not supported.  These can be detected by examining the
      feature_incompat field of the superblock.  Imagezip will fail if the
      user tries to compress one of these filesystems.
      b8d79ac6
    • Mike Hibler's avatar
      Add -Z option to zero free space included in the image due to -F. · 5a02dd31
      Mike Hibler authored
      Previously, this "internal free space" was whatever garbage happened to
      be on the disk we were imaging. By zeroing, we eliminate any leakage of
      information from the source disk and also allow the image to compress just
      a tad (1-4%) better.
      
      Why isn't this the default? Eh...no good reason, other than that this whole
      squish-out-small-free-ranges-to-allow-longer-writes optimization should be
      done by the client and should not be encoded in the image.
      5a02dd31
  23. 08 Oct, 2011 2 commits
    • Mike Hibler's avatar
      Revert "Adjust the set of unix gids used for a download server." · baba7478
      Mike Hibler authored
      This reverts commit fc89eb38.
      
      Checked in a bunch of crap that was unrelated.
      baba7478
    • Mike Hibler's avatar
      Adjust the set of unix gids used for a download server. · fc89eb38
      Mike Hibler authored
      When downloading an image, start the frisbeed process with the minimum set of
      gids necessary to access the image. This includes the unix gid of the
      project that the image is in and, optionally, the unix gid of the project
      subgroup if the image is part of one.
      
      Previously, we just use the gid set of the uid of the swapper of the
      experiment. Not only was this excessive, but it might also not include the
      gids needed in the case of a "global" image that is not in the world-readable
      /usr/testbed/images directory.
      fc89eb38
  24. 02 Jun, 2011 1 commit
  25. 31 May, 2011 1 commit
    • Mike Hibler's avatar
      Added compat option and assorted cleanups. · d8de4d6a
      Mike Hibler authored
      Added WITH_V3COMPAT to make sure that we can still generated V3 images
      (for other sites) if checksum/encryption is not used.
      
      Try to clean up the command line options some. Be more consistent by putting
      generated uuid into a file instead of just spitting it out on stderr. Make
      sure that if the decryption specifies signing and/or encryption that we
      require the image to have that info. Add some more assertions. (Re)distinguish
      checksums from signed-checksums.
      d8de4d6a
  26. 04 May, 2011 2 commits
  27. 22 Oct, 2010 2 commits
  28. 20 Jul, 2010 1 commit
  29. 06 Jul, 2010 1 commit