Commit 38fa67c4 authored by Leigh B. Stoller's avatar Leigh B. Stoller
Browse files

A set of small performance related changes that I made after putting

in the timestamps, and seeing what was taking so much time.

* Increase the overlap from two to four seconds (have vnodesetup exit
  earlier) so that the next jail can start sooner. Using getrusage it
  seemed pretty clear that there were plenty of cycles available to
  start the next jail going a little earlier. Instead of looking for
  the watchdog pid file to appear, I touch (in mkjail) a specific file
  in the jail directory , and have vnodesetup look for that file
  instead of the pid.

* Cut back the number of inodes in the filesystem! Sheesh, we ere
  creating 137,000 inodes for a 50MB vnode filesystem. Takes 1
  second to do that. Reduced to 30,000 and the time when to .1
  seconds. Could probably go even lower.

* Add -p option to fsck when restoring a jail from an existing
  filesystem. No need to force check the filesystem if its clean.

* Instead of running MAKDEV inside each jail filesystem (which was
  also taking .8 seconds to do), do it just once and stash a copy in
  /var/emulab/jails. Each jail does a "hier cp" of that master copy
  into the jail filesystem. Reduced the time to .05 seconds.

So, for the results:

Original time to boot a jail: 8.2 seconds, with the last two seconds
overlapped with the next one.

Original time to reinstate a jail after halting it (filesystem left
behind): 6.5 seconds, with the last 1.5 seconds overlapped with the next
one.

Original time to boot 16 jails on a pc850: 2 minutes and 12 seconds.

---------------------------------------------------------------------------

New time to boot a jail: 6.2 seconds, with the last 4 seconds overlapped
with the next one.

New time to reinstate a jail after halting it: 6 seconds, with the last 4
seconds overlapped with the next onw.

New time to boot 16 jails on a pc850: 88 seconds. Does this number ring any
bells? :-)
parent 8a2dd59d
......@@ -470,8 +470,8 @@ sub killvnode() {
"$mpid!\n";
last;
}
sleep(2);
$i += 2;
sleep(1);
$i += 1;
}
if ($doplab) {
......@@ -537,8 +537,8 @@ sub cleanup()
#
$SIG{TERM} = 'IGNORE';
kill('TERM', -$pgrp);
print "Waiting 2 seconds for process group to die off ...\n";
sleep(2);
print "Waiting 1 second for process group to die off ...\n";
sleep(1);
}
#
......@@ -713,7 +713,7 @@ sub removeconfdir($)
sub hackwaitandexit()
{
my $now = time();
my $goofy = CONFDIR() . "/root/var/run/emulab-watchdog.pid";
my $goofy = CONFDIR() . "/proceed";
my $count = 60;
while ($count--) {
......
......@@ -91,6 +91,8 @@ my $NFSMOUNT_REMOTE = 0; # too much load on NFS server
#
my $JAILPATH = "/var/emulab/jails";
my $ETCJAIL = "/etc/jail";
my $DEVCACHEMEM = "$JAILPATH/devmem";
my $DEVCACHENMEM= "$JAILPATH/devnomem";
my $LOCALFS = LOCALROOTFS();
my $LOCALMNTPNT = "/local";
my $TMCC = "$BINDIR/tmcc";
......@@ -278,6 +280,11 @@ else {
}
TBDebugTimeStamp("mkjail done with root fs");
#
# Lets tell the caller it can proceed.
#
system("touch $JAILPATH/$vnodeid/proceed");
#
# If the jail has its own IP, must insert the control network alias.
# We use a 255.255.255.255 netmask since there might be multiple
......@@ -389,8 +396,8 @@ sub mkrootfs($)
TBDebugTimeStamp("mkjail vnconfig, label, newfs, tunefs, mount");
mysystem("vnconfig -s labels vn${vndevice} root.vnode");
mysystem("disklabel -r -w vn${vndevice} auto");
mysystem("newfs -U -b 8192 -f 1024 -i 4096 -c 15 /dev/vn${vndevice}c");
mysystem("tunefs -m 2 -o time /dev/vn${vndevice}c");
mysystem("newfs -U -b 8192 -f 1024 -i 16384 -c 15 /dev/vn${vndevice}c");
mysystem("tunefs -m 5 -o time /dev/vn${vndevice}c");
mysystem("mount /dev/vn${vndevice}c root");
TBDebugTimeStamp("mkjail vnconfig, label, newfs, tunefs, mount done!");
push(@mntpoints, "$path/root");
......@@ -437,26 +444,54 @@ sub mkrootfs($)
# /dev is also special. It gets a very restricted set of entries.
# Note that we create some BPF devices since they work in our jails.
#
TBDebugTimeStamp("mkjail creating /dev directory");
my $makedevs = "bpf31";
# Set up the dev directory cache; running MAKEDEV is incredibly slow,
# but copying a directory device entries is really fast!
#
if (! -e $DEVCACHEMEM) {
TBDebugTimeStamp("mkjail creating /dev cached copy");
mkdir($DEVCACHEMEM, 0755) or
fatal("Could not mkdir $DEVCACHEMEM: $!");
mkdir($DEVCACHENMEM, 0755) or
fatal("Could not mkdir $DEVCACHENMEM: $!");
# Jails with JAIL_DEVMEM get more devices.
mysystem("cd $DEVCACHEMEM; cp -p /dev/MAKEDEV .; ".
" ./MAKEDEV bpf31 std pty0; " .
# Remove some extraneous devices.
"rm -f pci io klog console; " .
#
# "Link" /dev/console to /dev/null We actually create a
# separate special file rather than just linking in case
# anyone mucks with the permissions on /dev/console.
#
"hier cp null console; chmod 600 console");
# Otherwise no devmem.
mysystem("cd $DEVCACHENMEM; cp -p /dev/MAKEDEV .; ".
" ./MAKEDEV bpf31 jail; " .
# Remove some extraneous devices.
"rm -f pci io klog console; " .
#
# "Link" /dev/console to /dev/null We actually create a
# separate special file rather than just linking in case
# anyone mucks with the permissions on /dev/console.
#
"hier cp null console; chmod 600 console");
}
#
# Now copy contents of cached device directory.
#
TBDebugTimeStamp("mkjail copying cached /dev directory");
if ($jailflags & $JAIL_DEVMEM) {
$makedevs .= " std pty0";
mysystem("hier cp $DEVCACHEMEM $path/root/dev");
}
else {
$makedevs .= " jail";
mysystem("hier cp $DEVCACHENMEM $path/root/dev");
}
mysystem("cd $path/root/dev; cp -p /dev/MAKEDEV .; ./MAKEDEV $makedevs");
# Remove some extraneous devices.
mysystem("cd $path/root/dev; rm -f pci io klog console; ");
#
# "Link" /dev/console to /dev/null
# We actually create a separate special file rather than just linking
# in case anyone mucks with the permissions on /dev/console.
#
mysystem("cd $path/root/dev; hier cp null console; chmod 600 console");
#
# Create stub /var and create the necessary log files.
#
......@@ -633,7 +668,7 @@ sub restorerootfs($)
if ($debug);
mysystem("vnconfig -s labels vn${vndevice} root.vnode");
mysystem("fsck -y /dev/vn${vndevice}c");
mysystem("fsck -p -y /dev/vn${vndevice}c");
mysystem("mount /dev/vn${vndevice}c root");
push(@mntpoints, "$path/root");
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment