Reservation System breaks wrt Shared Nodes
Assign allocates VMs on shared nodes based on the physical type of the node, but the shared node experiment is locked down, so those physical nodes are technically not available when admission control is done. We have missed this, cause most of time there are some other d820s available, and so the limits file has a non-zero number of d820s, and so assign will allocate them to shared VMs.
But the other day, there were no d820s available, and with the four shared nodes locked down, the limits file got a zero in it. Some of the experiments ended up with a single VM on a dedicated node (since that is the fallback) of another type. But one experiment could not start at all cause there were nodes of any type available.
We can't just make the four shared nodes "available" since that will cause an overbook if all of the other nodes of that type are spoken for.
Suggestions welcome from the peanut gallery; @mike and @ricci and @gtw ...