Setup/teardown race in the iSCSI blockstore implementation
While trying to debug another issue, I discovered that there is some sort of race between terminating one blockstore experiment on the server and then immediately instantiating another from the same profile. We get a "Mismatched vlan" error.
What is happening is that the server, on a setup, looks up the requested VLAN tag for a vnode instance. If that vlan already exists on the server, it makes sure that the associated "blockstore ID" is the same as what was requested. If not it complains.
I think what is happening is that the serialized teardown of the previous instance is plodding along, but that we have already marked the associated VLAN tags as free in the Emulab DB. So when the new instance comes along, we reuse the VLAN tags for vnodes that haven't been torn down yet. But this is only a theory, I will need @stoller to verify/refute this.