in the timestamps, and seeing what was taking so much time.

* Increase the overlap from two to four seconds (have vnodesetup exit
  earlier) so that the next jail can start sooner. Using getrusage it
  seemed pretty clear that there were plenty of cycles available to
  start the next jail going a little earlier. Instead of looking for
  the watchdog pid file to appear, I touch (in mkjail) a specific file
  in the jail directory , and have vnodesetup look for that file
  instead of the pid.

* Cut back the number of inodes in the filesystem! Sheesh, we ere
  creating 137,000 inodes for a 50MB vnode filesystem. Takes 1
  second to do that. Reduced to 30,000 and the time when to .1
  seconds. Could probably go even lower.

* Add -p option to fsck when restoring a jail from an existing
  filesystem. No need to force check the filesystem if its clean.

* Instead of running MAKDEV inside each jail filesystem (which was
  also taking .8 seconds to do), do it just once and stash a copy in
  /var/emulab/jails. Each jail does a "hier cp" of that master copy
  into the jail filesystem. Reduced the time to .05 seconds.

So, for the results:

Original time to boot a jail: 8.2 seconds, with the last two seconds
overlapped with the next one.

Original time to reinstate a jail after halting it (filesystem left
behind): 6.5 seconds, with the last 1.5 seconds overlapped with the next
one.

Original time to boot 16 jails on a pc850: 2 minutes and 12 seconds.

---------------------------------------------------------------------------

New time to boot a jail: 6.2 seconds, with the last 4 seconds overlapped
with the next one.

New time to reinstate a jail after halting it: 6 seconds, with the last 4
seconds overlapped with the next onw.

New time to boot 16 jails on a pc850: 88 seconds. Does this number ring any
bells? :-)