Handle blockstores "correctly" during experiment modify
Since @stoller got experiment modify working via the portal (have I mentioned how awesome this is?) I need to figure out how to handle blockstores in a sane manner.
First, it should be pretty easy to make this work for remote blockstores, we just have to detach from them before modify and reattach to whatever is in the experiment after the modify.
Local blockstores are the issue. The cleanest approach would just be to destroy any existing blockstores before modify and then let it recreate blockstores after the modify. However, this would likely not sit well with users unless they were explicitly making a change to the blockstore configuration as part of their modify operation. If they were just say, adding a node, then wiping out their local blockstores on all other nodes is probably not a reasonable thing to do. Unfortunately, this may be what I do for the current "reconfig" target, I had better go fix that!
There is code in place today to save the current config at boot time and on reboot read in that config, adding or removing any blockstores that appear of disappear. But that is not going to work unless the reconfig involves a reboot. It also won't work if the boot disk is reloaded as part of the modify. We either need to store the configuration info in /proj, of maybe take advantage of the fact that volume managers like LVM and ZFS can reconstruct their config from on-disk info.