README 8.06 KB
Newer Older
1 2 3 4 5 6 7
Image test is a utility with takes a specified image, and then runs a
set of standard experiments on the image.  Each experiment consists of
an Emulab experiment which is designed to test specific functionally
on an image, such as static routing.  After an experiment is swapped
in it runs a number tests on the experiment to check that everything
is working.

8 9
SETUP

10 11 12 13 14 15 16 17 18 19 20 21
Checkout a copy of the image-test directory into the /proj/PID/ directory,
for example:

  cd /proj/PID/
  cvs -d bas.flux.utah.edu:/usr/flux/CVS co -d image-test testbed/image-test

The directory tree can be located anywhere as long as it is under
/proj/PID.  Also the directory doesn't necessary need to be named
"image-test".

You might want to also change the variable $eid_prefix at the start of
"image-test" to something different, your username might be a good
22 23
choice.

24

25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
BASIC USAGE

From the base directory run:

  ./image-test OS

where <OS> is an exiting image name such as RHL90-STD.

This will run each of the experiments in series.  The project to run
them in will automatically be determined based on the current working
directory.

The results will be in results-DATE.  It will print the directory
where the results will can be found to stdout.  The results of each
experiment can than be found in RESULTSDIR/EXP

Some of the files in this directory include:

  log: all output to stdout and stderr is redirected here
  nsfile.ns: the nsfile used
45 46
  parms: the value of the different parmaters used in the experiment.
    This is different from the one found in the test directory described 
47 48 49 50 51 52 53 54 55
    latter.
  failed-tests: a list of tests that failed
  exp-data/: a copy of the experiment data as found in /proj/PID/exp/EID
  tiplog-NODE: a copy of tiplog for NODE


Running each of the experiments in series can take a long time.  For this
reason it is possible to run more than one tests at once.  To do this use

56
  ./image-test -p [-m NUM] OS
57

58 59 60 61 62 63 64
where NUM is the maxim number of nodes to use at once.  If the "-m"
flag is not given NUM defaults to 5.  If a particular experiment uses
more than NUM nodes it will be run in series after all the experiments
that can be run in parallel have finished.  Thus setting NUM to 0 will
force all the experiments to run in series.

Setting -m sets a soft limit to the number of nodes to use.  To set a
65
hard limit use "-M NUM".  If an experiment will use more nodes than NUM
66 67
it will simply not be run.

68 69 70 71 72 73
NOTE: Image test does not currently handle temporary resource shortage
(ie not enough pcs) in an intelligent manner, thus it is best to set
NUM to something smaller than the number of PCs currently available.
Otherwise, you will have failed tests due to not being able to swap
the experient in.

74 75 76 77 78 79 80 81 82 83
If an experiment fails to swap out, the number of available nodes will
decrees, but other experiments will continue to run as long as there
are enough nodes left.  If there is only a soft limit all the
experiments will still run, but some of them may not be in parallel.
If a hard limit is specified than some experiments may be skipped if
there are no longer enough nodes available.  

To prevent new experiments from starting on a failed swap out by use the
"-h" option.  Any existing experiments already running will be allowed
to continue, however.
84

85 86 87 88
To prevent an experiment from being swapped out when a tests fails use
the "-u" option.  Since there are some test which are expected to fail
occasionally, this option is most useful when an exceptions file is
given via the "-e" option.  See "EXCEPTIONS FILE" for more info.
89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131


To only run particular experiment use:

  ./image-test OS EXPS

which will only run the experiments listed in EXPS.  For example:

  ./image-test RHL90-STD single-pc850 single-pc3000

will only run the experiments single-pc850 and single-pc3000.


Certain experiments can be run with different parameters.  These
parameters will be appended to the base experiment name.  For example
the experiment "single-pc850" is really the experiment "single" with
the first parameter being "pc850".  To run a particular experiment in
all the possible combinations just specify the base name, for example:

  ./image-test RHL90-STD single


Each experiment consists of 5 stages:

  (c) Create the experiment
  (s) Swap it in
  (t) Perform a series of tests of the experiment
  (o) Swap it out
  (e) End the experiment

To only run particular stages of an experiment use:

  ./image-test -s STAGES ...

where STAGES consists of the letters above.  Note that 'e' implies 'o'.
For example to avoid swapping an experiment out use:

  ./image-test -s cst ...

Or to just run the tests on a already swapped in experiment use:

  ./image-test -s t ...

132

133

134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172
TESTING FRAMEWORK

Each individual experiment is expected to be the directory "tests/EXP".
This directory must contain the following files

  num-nodes
  nsfile.ns
  
and may also contain:

  parms
  tests.pl

and any other experiment specific files.

The file "num-nodes" consists of a single line which is the number of
nodes the experiment will use.

The file "nsfile.ns" in the template ns file for the experiment. Any
strings of the form "@PARM@" will be substituted for the value of the
parameter.  The following parameters are available to all experiments:

  OS: OS to use as given in the command line
  DATADIR: BASE/tests/EXP

The file "parms" can be used to specify additional parameters for the
experiment.  all possible combinations of the parameters specified will
be tried with each combination being run with the experiment name
"EXP-PARM1-PARM2-...".

The file "tests.pl" can be used to specify additional experiment
specific tests.  This code will be run inside the "ImageTest"
package.  See the file "ImageTest.pm" and existing tests for more
info.

For every experiment a number of standard tests will be run.

For each node in the experiment the following tests will be run:

173 174 175
  ssh-NODE: try to ssh into the node, this also makes sure that the 
    users home directory is mounted, otherwise public-key authorization
    will fail
176 177 178 179
  sudo-NODE: make sure sudo is working correctly
  hostname-NODE: make sure host name is what it is expected to be
  login_prompt-NODE: make sure that the login prompt appears
    in the console
180 181
  proj_mount-NODE: make sure that the appropriate "/proj" directory is 
    mounted and readable
182 183 184

For experiments with more than one node:

185
  linktest?: run linktest levels 1 - 4
186 187 188

Additional standard tests will be added over time.

189 190 191 192 193

NOTE ON SIGNAL HANDLING

image-test works by forking and having the child do the real work.
All the parent does is monitor the log file and pass the signals
194 195 196
SIGINT and SIGTERM to the parent.  The child calls setsid() so that it
does not have a controlling terminal.  This unfortunately also means
that it is in a separate process group.
197 198 199 200 201 202 203 204

In addition each experiment runs in a separate process.  The PID of
this process is in the file "pid" in the experiment results directory.

Sending a SIGINT or a SIGTERM to the parent (or child) will terminate
all experiments ASAP.  Sending the SIGINT or a SIGTERM to an experiment
process will terminate only that experiment.

205 206 207
Sending a SIGHUP to the the parent will only terminate the parent.
The test will proceed normally in the background.  Similarly, sending
a SIGTSTP will only suspend the parent.
208

209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236

EXCEPTIONS FILE

An exceptions file may be specified with the "-e <FILE>" option.  For
example:
  ./image-test -e FILE IMAGE

The exceptions provides allows you to specify 1) experiments to skip 2)
individual tests within an experiment to skip, and 3) tests to ignore
failures for.

(1) and (2) are useful if there are experiments or individual tests
which are not relevant for the image being tested.  (3) is primary
useful with the "-u" option.

Each line of the exceptions file has is one of the following.  Blank lines
and extra whitespace are ignored, as well as anything after the "#"

  skip <exps>
  in <exp> skip <tests>
  in <exp> ignore <tests>

<exps> is a list of one or more experiments separated by spaces.
<exp> is a single experiment name.  A "*" may be used as a wildcard.
<tests> is a list of one or more tests.

Like with the experiments specified on the command line a base name of
an experiment may also be specified.
237