Changes to the blockstore model to accomodate the DriveScale boxes
It seemed early on that the two, 70+ drive disk boxes that DriveScale donated were going to be a near perfect match for the exisiting blockstore mechanism. But there is a fairly significant implementation issue we need to address.
With our current FreeNAS based storage servers, we put the experimental interfaces in trunk mode and create per-blockstore VLANs on the interface over which the iSCSI volumes are exposed. Thus on the client-side, a blockstore is just another node connected by its own experimental link/VLAN.
However, the current DriveScale API does not support creating VLANs for exporting iSCSI volumes. It is assumed that there is a single (aggregated) interface with a single IP subnet. The iSCSI configuration is used to limit which clients (IP addresses on that subnet) can access a particular iSCSI volume. This is more like what I originally thought we were going to do, having a fixed "SAN LAN" on the experiment fabric and putting blockstore clients into that.
I don't think this is a fundamental change, but I will let @kwebb weigh in since he did the original implementation.
If this does prove to be non-trivial, then maybe we want to go back and re-think the model of how we expose the DriveScale disks to users. We are still limited by the fact that iSCSI is the protocol used to expose the disks and that may limit what exactly users can do to the drives or how the iSCSI layer may insulate users from the behavior of the real disks. However, I think @ricci mosly wanted to get the boxes into use ASAP.