Emmanuel Cecchet
emulab-devel

Repository



Testbed Testing Suite
---------------------

This directory contains the testing framework for the Testbed.

Running
-------

To run the tests, follow these steps:

1. Create a directory to store your test files and logs.
	mkdir ~/tbtest

2. Make that directory your current working directory.
	cd ~/tbtest

3. Run tbtest with the appropriate arguments.  For frontend tests this
would be:
	~/testbed/testsuite/tbtest run tbdb frontend

4. Sit back and wait.

The test directory holds the following files:
	tbobj/			A configured/built object tree.
	install/		An installed tree.
	tests/			Holds individual test files.
	tests/<test>/
		test.log	Main log of everything.
		nsfile.ns	The NS file used.
		db.txt ***	The state of the database.
		*		Any files created by tbsetup stuff.
	test.log		Master log file.  Output of tbtest.
	configure.log		Log of 'configure'
	build.log		Log of 'gmake'
	install.log		Log of 'gmake boss-install'
	postinstall.log		Log of 'gmake post-install'
	createdb.log #		Log of creating test DB.
	fill.log #		Log of filling test DB.
	clear.log *		Log of clearing experiments from test DB.
	unavailable.log **	Log of marking nodes as unavailable.
	clean.txt		Dump of clean test DB.
	dbdump.txt		Dump of real database.
	defs			Defs file used for configure.
	state			Used internally for proper cleanup.
	
* - Appears only in frontend (default) mode.
** - Appears only in full mode.
*** - Only exists for failed tests.
# - Empty except under error conditions.

Note: cleanup.log, unavailable.log, and free.log appear only in full
mode.  clear.log appears only in frontend mode.

Advanced: To run all tests with a special DB setup you can do 'tbtest
init' and then replace 'clean.txt' with a dump of the DB of your
choice.  'clean.txt' is loaded into the test DB before every test
run.  Be very careful to maintain the reserved table when doing this
in full mode.

Adding Tests
------------

There are two ways to add a test.  

The simple way:

The script 'mktest' can be used to automatically generate tests the
use the following testing approach:
	Run tbprerun and check for exit code 0.
	Run tbswapin and check for exit code 0.
	Run tbswapout and check for exit code0.
	Run tbswapin and check for exit code 0.
	Run tbswapout and check for exit code 0.
	Run tbend and check for exit code 0.

I.e. tests that should pass and only care about checking as far as
exit codes.

To make such a test run:
	mktest <mode> <testname> <nsfile>

Where <mode> is frontend or full.  You will be prompted to enter a description of the test.

Alternately you can run mktest without arguments and it will prompt
for all information.


The advanced way:

The testing framework supports a far wider range of tests than those
described above.  A test can have arbitrary DB state, run any sequence
of commands, check for fail cases, and inspect data base state for
correctness.  To create such tests read the "Test Format" section
below.

Using the Test Tree
-------------------

It is often useful to be able to do manual testing using the test
tree.  This can easily be done.  By running 
	tbtest -leavedb init tbdb
you will setup the test tree and test database, leaving the DB
intact.  You can now run all your commands from the install tree to do
manual testing just as the tests do.

The experiment 'test' under project 'testbed' exists in the test DB
and should be used for all testing.

The following command, when run from the test directory, will set up
your path correctly:

setenv PATH "/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/site/bin:.:`pwd`/install/bin:`pwd`/install/sbin"

Program Reference 
-----------------

mktest
------

mktest is a tool to create basic tests from a NS file.  Tests created
with mktest only check the exit codes, they do not check database or
node status.

Syntax:

mktest <mode> <name> <nsfile> [<description>]
mktest

<mode> is one of "frontend" or "full".  If a description is not
specified on the command line then an editor will be opened for the
user to enter one.

When run with no arguments then mktest prompts the user for all
information.

mktest MUST be run in testbed/testsuite.


tbtest
------

tbtest runs all the tests.  It is designed to do an entire testing run
from start to finish leaving a large collection of log files in its
wake.


Syntax: tbtest [-leavedb] [-path <path>] [-frontend] [-full] <mode>
Options:
  -leavedb - Do not drop the test database on exit.
  -path <path> - Path to directory to store test files.
  -frontend - Run in frontend mode.
  -full - Run in full mode.
  -exitonfail - Exit on any failure.
  -session <sessionid> - Specify alternate session ID.
  -flest - Generate flest.log for use with flest.
Mode:
  run <db> <testlist> [<pid> <eid> <num>]
  init <db> [<pid> <eid> <num>]
  test <testlist>
  single <testlist> <tests>
  finish
tbtest [-leavedb] [-path <path>] <db> <testlist>

Notes:

The arguments in [] for the modes are necessary with -full and should
not be present otherwise.

You need only use -full on init to set the type for any other
subsequent commands.

Generally tbtest is invoked in 'run' mode that does everything.  For
an more interactive approach a you can invoke 'tbtest init ...' and
then any number of 'tbtest test' and 'tbtest single' commands to run
tests.  When finished do 'tbtest finish'.

Everything is keyed to your username.  There currently is no way to
have multiple test runs in progress under the same username without
collision.

-leavedb is really only useful when running in single mode with only
one test.  Since the database is reset at the beginning of every test
if you use this mode with multiple tests you'll only get the DB state
of the last one.  You recreate the database state of any test by 
creating a new database and then loading tests/<test>/db.txt into it.

If -path is not present the current working directory is used.

In full mode you must provide a <pid>, <eid>, and the number of nodes
to reserve.  tbtest will attempt to reserve that many nodes from the
testbed under the given experiment and then use those nodes to run the
tests.

-single can take a list of tests.

-exitonfail is best used in conjunction with -leavedb.  It causes
tbtest to exit on the first test failure, thus no later tests will
corrupt the DB.  The user of this option is a matter of taste, as you
can always use tests/<test>/db.txt to restore the database state.

-session specifies an alternate session id.  The test DB will be named
tbdb_<session>.  Usually the session id is taken from the username.
Unfortunately the session ID must be globally unique.

dbdump
------

dbdump is a utility for constructing the results structure used in
tb_compare.

	dbdump <database> <query>

the output will be Perl code constructing the an array, @result, which
matches the result of <query> on DB <database>.  This should be copied
into the test script and the appropriate tb_compare command inserted
afterwards:

	tb_compare("<query>",\@result);

Notes: 

Make sure you change " to \" in <query> for the tb_compare statement.

This makes an exact copy.  As such it is inappropriate for information
that can vary from experiment to experiment such as the
virtual<->physical mapping.


Test Format
-----------

A test is a directory in testsuite/tests which contains the
following files:
	nsfile.ns - NS file.
	dbstate - Any commands to set up the DB.
	test - Test file.
	info - Info file.

dbstate 

This is just a list of SQL commands that are applied to the DB state
before the test in run.

info

Just a description of the test.

test 

This is a perl script that actually runs the test.  Generally it looks
something like:

tb_prerun("tbprerun",0);
tb_compare("<SQL query>",<results>);
tb_run("tbswapin -test",0);
tb_compare("<SQL query>",<results>);

See "Test API" below for a list of available subroutines and
variables.


Test API
--------

Routines:

tb_prerun(<cmd>,<exitcode>)

This runs "<cmd> pid eid nsfile", and compares the exitcode.  The test
fails and exit if the exit codes do not equal.

tb_run(<cmd>,<exitcode>)

This runs "<cmd> pid eid", and compares the exitcode.  The test fails
and exit if the exit codes do not equal.

tb_compare(<query>,<results>)

This executes the SQL <query> and then fetches the results.  <results>
is a list of list references.  If the results match exactly then test
test continues otherwise results are displayed and the test fails.

tb_fail(<msg>)

Explicitly fail the test with <msg>.


Variables (changing this values will have no effect besides
potentially messing your own test up).

$pid, $eid - PID and EID of test experiment.
$test - Test name.
$dir - Directory containing test files.
$dbh - Handle to DB connection.

Other notes:

The current directory should not be changed.  It will be the directory
containing .top, .ptop, and .log files.