README 8.62 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Testbed Testing Suite
---------------------

This directory contains the testing framework for the Testbed.

Running
-------

To run the tests, follow these steps:

1. Create a directory to store your test files and logs.
	mkdir ~/tbtest

2. Make that directory your current working directory.
	cd ~/tbtest

3. Run tbtest with the appropriate arguments.  For frontend tests this
would be:
19
	~/testbed/testsuite/tbtest run tbdb frontend
20 21 22 23

4. Sit back and wait.

The test directory holds the following files:
24 25 26 27 28 29
	tbobj/			A configured/built object tree.
	install/		An installed tree.
	tests/			Holds individual test files.
	tests/<test>/
		test.log	Main log of everything.
		nsfile.ns	The NS file used.
30
		db.txt ***	The state of the database.
31 32 33 34 35 36
		*		Any files created by tbsetup stuff.
	test.log		Master log file.  Output of tbtest.
	configure.log		Log of 'configure'
	build.log		Log of 'gmake'
	install.log		Log of 'gmake boss-install'
	postinstall.log		Log of 'gmake post-install'
37 38 39 40
	createdb.log #		Log of creating test DB.
	fill.log #		Log of filling test DB.
	clear.log *		Log of clearing experiments from test DB.
	unavailable.log **	Log of marking nodes as unavailable.
41 42 43
	clean.txt		Dump of clean test DB.
	dbdump.txt		Dump of real database.
	defs			Defs file used for configure.
44
	state			Used internally for proper cleanup.
45
	
46 47
* - Appears only in frontend (default) mode.
** - Appears only in full mode.
48
*** - Only exists for failed tests.
49
# - Empty except under error conditions.
50

51 52
Note: cleanup.log, unavailable.log, and free.log appear only in full
mode.  clear.log appears only in frontend mode.
53 54 55

Advanced: To run all tests with a special DB setup you can do 'tbtest
init' and then replace 'clean.txt' with a dump of the DB of your
56 57 58
choice.  'clean.txt' is loaded into the test DB before every test
run.  Be very careful to maintain the reserved table when doing this
in full mode.
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95

Adding Tests
------------

There are two ways to add a test.  

The simple way:

The script 'mktest' can be used to automatically generate tests the
use the following testing approach:
	Run tbprerun and check for exit code 0.
	Run tbswapin and check for exit code 0.
	Run tbswapout and check for exit code0.
	Run tbswapin and check for exit code 0.
	Run tbswapout and check for exit code 0.
	Run tbend and check for exit code 0.

I.e. tests that should pass and only care about checking as far as
exit codes.

To make such a test run:
	mktest <mode> <testname> <nsfile>

Where <mode> is frontend or full.  You will be prompted to enter a description of the test.

Alternately you can run mktest without arguments and it will prompt
for all information.


The advanced way:

The testing framework supports a far wider range of tests than those
described above.  A test can have arbitrary DB state, run any sequence
of commands, check for fail cases, and inspect data base state for
correctness.  To create such tests read the "Test Format" section
below.

96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112
Using the Test Tree
-------------------

It is often useful to be able to do manual testing using the test
tree.  This can easily be done.  By running 
	tbtest -leavedb init tbdb
you will setup the test tree and test database, leaving the DB
intact.  You can now run all your commands from the install tree to do
manual testing just as the tests do.

The experiment 'test' under project 'testbed' exists in the test DB
and should be used for all testing.

The following command, when run from the test directory, will set up
your path correctly:

setenv PATH "/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/site/bin:.:`pwd`/install/bin:`pwd`/install/sbin"
113 114 115

Program Reference 
-----------------
116 117 118 119

mktest
------

120
mktest is a tool to create basic tests from a NS file.  Tests created
121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138
with mktest only check the exit codes, they do not check database or
node status.

Syntax:

mktest <mode> <name> <nsfile> [<description>]
mktest

<mode> is one of "frontend" or "full".  If a description is not
specified on the command line then an editor will be opened for the
user to enter one.

When run with no arguments then mktest prompts the user for all
information.

mktest MUST be run in testbed/testsuite.


139 140 141 142 143 144 145 146
tbtest
------

tbtest runs all the tests.  It is designed to do an entire testing run
from start to finish leaving a large collection of log files in its
wake.


147 148 149 150 151
Syntax: tbtest [-leavedb] [-path <path>] [-frontend] [-full] <mode>
Options:
  -leavedb - Do not drop the test database on exit.
  -path <path> - Path to directory to store test files.
  -frontend - Run in frontend mode.
152
  -full - Run in full mode.
153
  -exitonfail - Exit on any failure.
154
  -session <sessionid> - Specify alternate session ID.
155
  -flest - Generate flest.log for use with flest.
156
Mode:
157
  run <db> <testlist> [<pid> <eid> <num>]
158
  init <db> [<pid> <eid> <num>]
159 160
  test <testlist>
  single <testlist> <tests>
161
  finish
162
tbtest [-leavedb] [-path <path>] <db> <testlist>
163 164 165

Notes:

166 167 168
The arguments in [] for the modes are necessary with -full and should
not be present otherwise.

169 170
You need only use -full on init to set the type for any other
subsequent commands.
171

172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194
Generally tbtest is invoked in 'run' mode that does everything.  For
an more interactive approach a you can invoke 'tbtest init ...' and
then any number of 'tbtest test' and 'tbtest single' commands to run
tests.  When finished do 'tbtest finish'.

Everything is keyed to your username.  There currently is no way to
have multiple test runs in progress under the same username without
collision.

-leavedb is really only useful when running in single mode with only
one test.  Since the database is reset at the beginning of every test
if you use this mode with multiple tests you'll only get the DB state
of the last one.  You recreate the database state of any test by 
creating a new database and then loading tests/<test>/db.txt into it.

If -path is not present the current working directory is used.

In full mode you must provide a <pid>, <eid>, and the number of nodes
to reserve.  tbtest will attempt to reserve that many nodes from the
testbed under the given experiment and then use those nodes to run the
tests.

-single can take a list of tests.
195

196 197 198 199 200
-exitonfail is best used in conjunction with -leavedb.  It causes
tbtest to exit on the first test failure, thus no later tests will
corrupt the DB.  The user of this option is a matter of taste, as you
can always use tests/<test>/db.txt to restore the database state.

201 202 203
-session specifies an alternate session id.  The test DB will be named
tbdb_<session>.  Usually the session id is taken from the username.
Unfortunately the session ID must be globally unique.
204

205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228
dbdump
------

dbdump is a utility for constructing the results structure used in
tb_compare.

	dbdump <database> <query>

the output will be Perl code constructing the an array, @result, which
matches the result of <query> on DB <database>.  This should be copied
into the test script and the appropriate tb_compare command inserted
afterwards:

	tb_compare("<query>",\@result);

Notes: 

Make sure you change " to \" in <query> for the tb_compare statement.

This makes an exact copy.  As such it is inappropriate for information
that can vary from experiment to experiment such as the
virtual<->physical mapping.


229 230 231
Test Format
-----------

232
A test is a directory in testsuite/tests which contains the
233
following files:
234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257
	nsfile.ns - NS file.
	dbstate - Any commands to set up the DB.
	test - Test file.
	info - Info file.

dbstate 

This is just a list of SQL commands that are applied to the DB state
before the test in run.

info

Just a description of the test.

test 

This is a perl script that actually runs the test.  Generally it looks
something like:

tb_prerun("tbprerun",0);
tb_compare("<SQL query>",<results>);
tb_run("tbswapin -test",0);
tb_compare("<SQL query>",<results>);

258 259
See "Test API" below for a list of available subroutines and
variables.
260 261


262 263
Test API
--------
264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284

Routines:

tb_prerun(<cmd>,<exitcode>)

This runs "<cmd> pid eid nsfile", and compares the exitcode.  The test
fails and exit if the exit codes do not equal.

tb_run(<cmd>,<exitcode>)

This runs "<cmd> pid eid", and compares the exitcode.  The test fails
and exit if the exit codes do not equal.

tb_compare(<query>,<results>)

This executes the SQL <query> and then fetches the results.  <results>
is a list of list references.  If the results match exactly then test
test continues otherwise results are displayed and the test fails.

tb_fail(<msg>)

285
Explicitly fail the test with <msg>.
286 287


288 289
Variables (changing this values will have no effect besides
potentially messing your own test up).
290 291 292 293

$pid, $eid - PID and EID of test experiment.
$test - Test name.
$dir - Directory containing test files.
294
$dbh - Handle to DB connection.
295

296 297 298 299 300 301 302 303
Other notes:

The current directory should not be changed.  It will be the directory
containing .top, .ptop, and .log files.




304