Improve storage server disk usage
This comes from another ticket (#627 (closed)) but is not specific to that storage server.
The general problem is that we run out of space well before what we think is the capacity of the zpool. I discovered that the ZFS zvol volume size (-V when creating) is a space limit from the perspective of the user, and not a limit on how much space the zvol will consume. While knowing this at some level, I failed to appreciate just how much overhead ZFS can introduce on top of that, including metadata such as RAID parity, checksums, and fragmentation due to misalignment of various blocksizes. Apparently the recommendation for iSCSI on zvols is to not allocate more than 50% of the capacity. In our most extreme example, we have a 15TB dataset, which is using 21TB of disk space even though the overlayed Linux filesystem is only using 7TB. Potential improvements here:
- Use "thin" zvols. This really doesn't solve anything though, it just allows us to overbook storage and probably everyone will run out of space in a truly ugly way down the road.
- Turn on compression for the zvols. This is one way of implementing that "don't allocate the full disk capacity", but of course depends on the nature of the data and doesn't address the metadata overhead.
- Switch to using iSCSI volumes on top of ZFS filesystem files. This is what a number of people say you should do, as apparently it has fewer "misalignment" problems and performs better overall.
- Turn on the "discard" option when we create ext4 filesystems, so that they TRIM. This will help when the overlayed filesystem has a lot of free space.
- Use something other than ZFS? This would be a lot of work to implement at this point.