Commit d138f5cc authored by Christopher Alfeld's avatar Christopher Alfeld

First version of the testing software.

This is a functional check in.  To run the testing software do
something like:
	1. mkdir ~/testbed_test
	2. cd ~/testbed_test
	3. ~/testbed/testsuite/tbtest tbdb frontend
alternately:
	1. mkdir ~/testbed_test
	2. cd ~/testbed/testsuite
	3. tbtest -path ~/testbed_test tbdb frontend

A lot of stuff will be placed in testbed_test, including a configured
tree, an installed tree, a couple versions of the DB, and a whole
bunch of log files.

Interesting log files:

The main log file of interest is test.log which is a copy of the
output of tbtest.  The other log files are only useful if things fail.
Usually the fail message will include which log file to look in.  Log
files for tests are in tests/<test>/test.log .

What's missing:

1. Lot's of tests.  There's only one currently.

2. DB checking support.

3. A real README.  The current one is mainly a bunch of notes for the
real README.

4. Full support.  The current setup is only for testing the frontend.
Although there are some hooks for supporting full testing runs a lot
still needs to be done.
parent 10e37b2f
mktest
------
Make test is a tool to create basic tests from a NS file. Tests created
with mktest only check the exit codes, they do not check database or
node status.
Syntax:
mktest <mode> <name> <nsfile> [<description>]
mktest
<mode> is one of "frontend" or "full". If a description is not
specified on the command line then an editor will be opened for the
user to enter one.
When run with no arguments then mktest prompts the user for all
information.
mktest MUST be run in testbed/testsuite.
Test Format
-----------
A test is a directory in testsuite/tests which contains the following files:
nsfile.ns - NS file.
dbstate - Any commands to set up the DB.
test - Test file.
info - Info file.
dbstate
This is just a list of SQL commands that are applied to the DB state
before the test in run.
info
Just a description of the test.
test
This is a perl script that actually runs the test. Generally it looks
something like:
tb_prerun("tbprerun",0);
tb_compare("<SQL query>",<results>);
tb_run("tbswapin -test",0);
tb_compare("<SQL query>",<results>);
See TEST FILES below
TEST FILES
----------
Routines:
tb_prerun(<cmd>,<exitcode>)
This runs "<cmd> pid eid nsfile", and compares the exitcode. The test
fails and exit if the exit codes do not equal.
tb_run(<cmd>,<exitcode>)
This runs "<cmd> pid eid", and compares the exitcode. The test fails
and exit if the exit codes do not equal.
tb_compare(<query>,<results>)
This executes the SQL <query> and then fetches the results. <results>
is a list of list references. If the results match exactly then test
test continues otherwise results are displayed and the test fails.
tb_fail(<msg>)
Explicitely fail the test with <msg>.
Variables (do not change values!):
$pid, $eid - PID and EID of test experiment.
$test - Test name.
$dir - Directory containing test files.
Other notes:
The current directory should not be changed. It will be the directory
containing .top, .ptop, and .log files.
#!/usr/bin/perl
# The lack of -w is intentional! Several variables here are only used
# by the test file and maybe not even there. Thus -w will cause a lot
# of warnings of variables being used only once.
# tbstub
# This should not be directly called by the user. This is called from
# tbtest to provide a basic environment for the test scripts to run. The
# reason to have the tests run in a seperate process it for memory management.
# There is no way to completely clear out the memory used by a package
# other than terminating the interpreter. Thus to allow each test to
# run in it's own namespace without having a memory leak we put each
# in it's own process.
# The last last of output must be:
# PASS
# FAIL <msg>
# Syntax:
# tbstub <db> <pid> <eid> <testname> <testdir>
($db,$pid,$eid,$test,$dir) = @ARGV;
# The status of the test.
$status = "";
sub doexit {
if ($status eq "") {
$status = "PASS";
}
print "$status\n";
exit(0);
};
do {
package TEST;
$eid = $::eid;
$pid = $::pid;
$test = $::test;
$dir = $::dir;
sub tb_prerun {
my ($cmd,$exitcode) = @_;
print "$cmd $pid $eid $dir/nsfile.ns\n";
open(TBEXEC,"$cmd $pid $eid $dir/nsfile.ns 2>&1 |");
while (<TBEXEC>) {
print $_;
}
close(TBEXEC);
if (($? >> 8) != $exitcode) {
tb_fail($cmd);
}
};
sub tb_compare {
print "tb_compare not yet implemented.\n";
};
sub tb_run {
my ($cmd,$exitcode) = @_;
print "$cmd $pid $eid\n";
open(TBEXEC,"$cmd $pid $eid 2>&1 |");
while (<TBEXEC>) {
print $_;
}
close(TBEXEC);
if (($? >> 8) != $exitcode) {
tb_fail($cmd);
}
};
sub tb_fail {
$::status = "FAIL - " . $_[0];
::doexit;
};
print "Executing $dir/test\n";
do "$dir/test";
};
doexit;
#!/usr/bin/perl -w
# tbtest
# This is the toplevel of the testbed testing setup. This command
# should be invokved by a user and script. It well then create a
# separate object tree, configure it appropriately, setup a testing
# database, and then proceed to run tests.
# IMPORTANT: This command should be come from a tree configured
# for the real DB.
# tbtest [options] <db> <testdir>
# Options:
# -path <path> - Path to test tree, otherwise uses cwd.
use DBI;
use POSIX;
$| = 1;
$pwd = `/bin/pwd`;
chop $pwd;
# Figure out where the script is
if ($0 =~ m|/|) {
# Not pwd
@a = split("/",$0);
$dir = join("/",@a[0..$#a-1]);
if ($0 =~ m|^/|) {
# absolute path
$basedir = $dir;
} else {
# relative path
chdir $dir;
$basedir = `/bin/pwd`;
chop $basedir;
chdir $pwd;
}
} else {
$basedir = `/bin/pwd`;
chop $basedir;
}
$path = $pwd;
$ENV{'PATH'} = "/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/site/bin:$basedir";
# Parse argv
while ($#ARGV > 1) {
$arg = shift;
if ($arg eq "-path") {
$path = shift;
} else {
print STDERR "Unknown argument: $arg\n";
exit(1);
}
}
chdir $path;
if ($#ARGV != 1) {
print STDERR "Syntax: $0 [-path <path>] <db> <testdir>\n";
exit(1);
}
($TBDB,$testdir) = @ARGV;
if (! ($testdir =~ m|^/|)) {
$testdir = "$basedir/tests/$testdir";
}
# Initialize DB
$dbh = DBI->connect("DBI:mysql:database=$TBDB;host=localhost") ||
die "Could not connect to DB.\n";
# Having made sure we can, we now disconnect so as not to harm DB.
$dbh->disconnect;
$user=`whoami`;
chop $user;
$testdb = "tbdb_$user";
$logfile = "test.log";
open(LOG,"> $logfile");
# prints sends the string to stdout and the logfile.
sub prints {
print $_[0];
print LOG $_[0];
};
# tbexec <cmd> <file>
# Executes <cmd> storing results to <file> and displaying appropriate
# message and exiting on failure.
sub tbexec {
my ($cmd,$file) = @_;
open(EXEC,"$cmd 2>&1 |");
open(FILE,"> $file");
while (<EXEC>) {
print FILE $_;
}
close(FILE);
close(EXEC);
if (($? >> 8) == 0) {
prints " PASS\n";
$pass++;
} else {
prints " FAIL - See $file\n";
$fail++;
doexit(1);
}
};
# tbexect <cmd> <logfp>
# This is similar to tbexec except is used for running commands specific
# to tests. It runs the command appending the output to the file handle
# <logfp> and returns the exit code.
sub tbexect {
my ($cmd,$logfp) = @_;
print $logfp "$cmd\n";
open(EXEC,"$cmd 2>&1 |");
while (<EXEC>) {
print $logfp $_;
}
close(EXEC);
return ($? >> 8);
};
# doexit
# This displays exit counts and then exits with the specified exit code.
sub doexit {
printf LOG "Total Tests: %-5d Pass: %-5d Fail: %-5d Unknown: %-5d\n", $pass+$fail+$unknown, $pass, $fail, $unknown;
printf "Total Tests: %-5d Pass: %-5d Fail: %-5d Unknown: %-5d\n", $pass+$fail+$unknown, $pass, $fail, $unknown;
# Try to do the necessary cleanup
system("echo \"drop database if exists $testdb\" | mysql");
exit($_[0]);
};
# Test counts
$pass = 0;
$fail = 0;
$unknown = 0;
prints "User: $user\n";
prints "Testdir: $testdir\n";
prints "Logfile: $logfile\n";
prints "\n";
prints "Beginning test run. " . ctime(time) . "\n";
# Set up object tree
prints "Setting up object tree.\n";
if (! -d $path) {
mkdir $path,0755;
}
mkdir "tbobj",0755;
mkdir "install",0755;
chdir "tbobj";
system("cp -f $basedir/test-defs $path/defs");
open(DEFS,">>$path/defs");
print DEFS "TBDBNAME=$testdb\n";
print DEFS "TBOPSEMAIL=$user\@emulab.net\n";
print DEFS "TBLOGSEMAIL=$user\@emulab.net\n";
close(DEFS);
prints " Configuring ";
tbexec("$basedir/../configure --prefix=$path/install ".
"--with-TBDEFS=$path/defs --with-LEDA=/usr/testbed/LEDA/LEDA-4.0",
"$path/configure.log");
prints " Building ";
tbexec("gmake","$path/build.log");
prints " Installing ";
tbexec("gmake boss-install","$path/install.log");
prints " Post-Installing ";
tbexec("su1 gmake post-install","$path/postinstall.log");
# Let's add some stuff to our path
$ENV{'PATH'} .= ":$path/install/bin:$path/install/sbin";
chdir $path;
# Setup test database
prints "Setting up test database.\n";
prints " Dumping Current ";
tbexec("mysqldump $TBDB","$path/dbdump.txt");
prints " Creating DB ";
tbexec("echo \"create database $testdb\" | mysql","createdb.log");
prints " Filling DB ";
tbexec("cat $path/dbdump.txt | mysql $testdb","fill.log");
# Clear out experiments and set up test experiment.
$dbh = DBI->connect("DBI:mysql:database=$testdb;host=localhost") || do {
prints "ERROR: Could not connect to new DB.\n";
doexit(1);
};
prints " Clearing out existing experiments.\n";
open(CLEARLOG,"> clear.log");
$sth = $dbh->prepare("SELECT pid,eid from experiments");
$sth->execute;
while (($pid,$eid) = $sth->fetchrow_array) {
prints " $pid/$eid ";
print CLEARLOG "$pid $eid\n";
print CLEARLOG "tbswapout -force -test $pid $eid\n";
open(EXEC,"tbswapout -force -test $pid $eid 2>&1 |");
while (<EXEC>) {
print CLEARLOG $_;
}
close(EXEC);
if (($? >> 8) != 0) {
prints "FAIL - On tbswapout - See clear.log\n";
$fail++;
doexit(1);
}
print CLEARLOG "tbend $pid $eid\n";
open(EXEC,"tbend $pid $eid 2>&1 |");
while (<EXEC>) {
print CLEARLOG $_;
}
close(EXEC);
if (($? >> 8) != 0) {
prints " FAIL - On tbend - See clear.log\n";
$fail++;
doexit(1);
}
prints "PASS\n";
$pass++;
}
close(CLEARLOG);
prints " Setting up test experiment ";
$dbh->do("DELETE from experiments where pid=\"testbed\" and eid=\"test\"");
$dbh->do("INSERT into experiments (pid,eid) "
. "values (\"testbed\",\"test\")") || do {
prints "FAIL\n";
$fail++;
doexit(1);
};
print "PASS\n";
$pass++;
prints " Snapshotting clean state ";
tbexec("mysqldump $testdb","clean.txt");
# Time to run tests!
prints "Running Tests\n";
opendir(TESTDIR,$testdir);
@tests = readdir(TESTDIR);
closedir(TESTDIR);
mkdir "tests",0755;
foreach $test (@tests) {
if (($test eq ".") || ($test eq "..") || ($test eq "CVS")) {
next;
}
prints " $test ";
mkdir "tests/$test",0755;
chdir "tests/$test";
open(TESTLOG,"> test.log");
# Set up DB state
if (tbexect("echo \"drop database $testdb\" | mysql",TESTLOG)) {
prints "FAIL - Could not drop test DB.\n";
prints "Failure was FATAL - Exiting.\n";
$fail++;
doexit(1);
}
if (tbexect("echo \"create database $testdb\" | mysql",TESTLOG)) {
prints "FAIL - Could not recreate test DB.\n";
prints "Failure was FATAL - Exiting.\n";
$fail++;
doexit(1);
}
if (tbexect("cat $path/clean.txt | mysql $testdb",TESTLOG)) {
prints "FAIL - Could not populate test DB.\n";
prints "Failure was FATAL - Exiting.\n";
$fail++;
doexit(1);
}
if (-f "$testdir/$test/dbstate") {
if (tbexect("cat $testdir/$test/dbstate | mysql $testdb",TESTLOG)) {
prints "FAIL - Could not evaluate dbstate.\n";
$fail++;
next;
}
}
# We're now ready to run the test.
# The last line of this command will be either PASS
# or FAIL <msg>, all output before that should be sent
# to the log file.
open(TESTFP,"tbstub $testdb testbed test $test $testdir/$test |");
while (<TESTFP>) {
print TESTLOG $_;
$status = $_;
}
chop $status;
close(TESTFP);
if (($? >> 8) != 0) {
prints "UNKNOWN - Test exited with non-0 code.\n";
$unknown++;
} else {
if ($status =~ /^PASS/) {
$pass++;
} elsif ($status =~ /^FAIL/) {
$fail++;
} else {
$unknown++;
$status = "UKNONW";
}
prints "$status\n";
}
close(TESTLOG);
chdir $path;
}
prints "Cleaning up\n";
prints " Dropping test DB ";
tbexec("echo \"drop database $testdb\" | mysql","cleanup.log");
prints "End test run. " . ctime(time) . "\n";
doexit(0);
tb_prerun("tbprerun",0); tb_prerun("tbprerun",0);
tb_run("tbswapin -test",0); tb_run("tbswapin -test",0);
tb_run("tbswapout -force",0); tb_run("tbswapout -test",0);
tb_run("tbswapin -test",0); tb_run("tbswapin -test",0);
tb_run("tbswapout -force",0); tb_run("tbswapout -test",0);
tb_run("tbend",0); tb_run("tbend",0);
#
# This is the defaults definitions file for configure.
#
# Use the -with-DEFS=filename to specify your own. If you add a variable,
# be sure to go and update configure.in.
#
WWWDEFS=default
IPBASE=192.168
DELAYCAPACITY=2
Just a basic initial test.
Topology:
link1 link2
+-----node4-----+
| |
node1 node2 node3
| | |
lan1---------------------
Delays:
lan1 100Mb/0ms
link1 100Mb/50ms
link2 100Mb/100ms
TB commands:
None
Warnings:
None
Errors:
None
Log: /users/calfeld/testbed/testsuite/tests/frontend/basic/nsfile.log
Input: /users/calfeld/testbed/testsuite/tests/frontend/basic/nsfile.ns
Beginning pre run for testbed-test (/users/calfeld/testbed/testsuite/tests/frontend/basic/nsfile.ns). Wed Jun 27 15:25:54 2001
Checking sanity.
Running parser.
Marking as prerunned.
Pre run finished - Wed Jun 27 15:25:54 2001
Log: /users/calfeld/testbed/testsuite/tests/frontend/basic/nsfile.log
Input: /users/calfeld/testbed/testsuite/tests/frontend/basic/nsfile.ns
Beginning pre run for testbed-test (/users/calfeld/testbed/testsuite/tests/frontend/basic/nsfile.ns). Wed Jun 27 15:27:33 2001
Checking sanity.
Running parser.
Marking as prerunned.
Pre run finished - Wed Jun 27 15:27:33 2001
Log: /users/calfeld/testbed/testsuite/tests/frontend/basic/nsfile.log
Input: /users/calfeld/testbed/testsuite/tests/frontend/basic/nsfile.ns
Beginning pre run for testbed-test (/users/calfeld/testbed/testsuite/tests/frontend/basic/nsfile.ns). Wed Jun 27 15:28:22 2001
Checking sanity.
Running parser.
Marking as prerunned.
Pre run finished - Wed Jun 27 15:28:22 2001
Log: /users/calfeld/testbed/testsuite/tests/frontend/basic/nsfile.log
Input: /users/calfeld/testbed/testsuite/tests/frontend/basic/nsfile.ns
Beginning pre run for testbed-test (/users/calfeld/testbed/testsuite/tests/frontend/basic/nsfile.ns). Wed Jun 27 15:29:21 2001
Checking sanity.
Running parser.
Marking as prerunned.
Pre run finished - Wed Jun 27 15:29:21 2001
source tb_compat.tcl
set ns [new Simulator]
set node1 [$ns node]
set node2 [$ns node]
set node3 [$ns node]
set node4 [$ns node]
set lan1 [$ns make-lan "$node1 $node2 $node3" 100Mb 0ms]
set link1 [$ns duplex-link $node4 $node1 100Mb 50ms DropTail]
set link2 [$ns duplex-link $node4 $node3 10Mb 100ms DropTail]
$ns run
tb_prerun("tbprerun",0);
tb_run("tbswapin -test",0);
tb_run("tbswapout -test",0);
tb_run("tbswapin -test",0);
tb_run("tbswapout -test",0);
tb_run("tbend",0);
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment