README-howto.txt 15.4 KB
Newer Older
1 2 3
# -*- mode: text; indent-tabs-mode: nil -*-
#
# Copyright (c) 2007 University of Utah and the Flux Group.
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
# 
# {{{EMULAB-LICENSE
# 
# This file is part of the Emulab network testbed software.
# 
# This file is free software: you can redistribute it and/or modify it
# under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or (at
# your option) any later version.
# 
# This file is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public
# License for more details.
# 
# You should have received a copy of the GNU Affero General Public License
# along with this file.  If not, see <http://www.gnu.org/licenses/>.
# 
# }}}
23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
#
sec-check/README-howto.txt - Details of running and incremental development.

See README-concepts.txt for a description of the design and methods employed.
See README-FIRST.txt for a top-level outline.


 - General

   . Directories

     - Sec-check is run via configure/gmake in an Emulab obj-devel/www/sec-check
       directory.  Below, it is assumed you're cd'ed there.

     - Control files come from testbed/www/sec-check in the user's source tree.

     - Generated intermediate files and output are checked into source subdir
       testbed/www/sec-check/results, so they can be compared over time with CVS
       as things change.

     - These variables, relative to obj-devel/www/sec-check, are used below:
         set tws=../../../testbed/www/sec-check
         set twsr=$tws/results

   . Inner Emulab-in-Emulab experiment
   
     You need an inner Elab swapped in to do almost everything below.  This is
     documented in https://www.emulab.net/tutorial/elabinelab.php3 .

     There is a simple ElabInElab experiment ns file in $tws/vulnElab.ns .
53

54
     Change EinE_proj and EinE_exp in the $tws/GNUmakefile.in to match the
55 56
     Emulab project and experiment names you create.  Edit the email addresses to
     match your mailers.  Notice that two *REAL* email addresses are required.
57 58 59 60 61 62


 - High-level targets

   . all: src_forms spider forms_coverage input_coverage normal probe

63
     Do everything after activation.  Ya gotta activate below first!
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86

         gmake all |& tee all.log

   . msgs: src_msg site_msg forms_msg input_msg analyze probes_msg
     Show all of the summaries.

         gmake msgs

     - I log a copy of the results to CVS once in a while:

           gmake msgs | egrep -v gmake | tee $twsr/msgs.log


 - Stages of operation (makefile targets)

   ----------------
   . src_forms: src_list src_msg

     Grep the sources for <form and make up a list of php form files.

         gmake src_forms
         gmake src_msg

87
     - Output goes in $twsr/src_{forms,files}.list, bare filenames and raw
88 89
       <form grep lines respectively.

90 91 92 93 94 95 96
     - gmake src_msg reports something like:

       ** Sources: 107 separate forms are on 89 code pages. **
       ** (See src_forms.list and src_files.list
       **  in ../../../testbed/www/sec-check/results .) **
       **

97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124
   ----------------
   . activate: activate.wget $(activate_tasks) analyze_activate

     Sets up the newly swapped-in ElabInElab site in the makefile to create
     "one of everything" (or sometimes two in different states), thus turning
     on as many forms as we can for spidering.

     - Don't forget to first log in to the inner Elab in a web browser and
       change your password in Edit Profile in the inner Elab to match the
       string given in the GNUmakefile.in as $(pswd).  

       The initial password is your outer Elab password, imported into the
       inner Elab.  There's no reason to put your real Elab password into the
       makefile!  Don't do it!

       For me, the URL of the inner Elab apache httpd server on "myboss" is
       https://myboss.vulnelab.testbed.emulab.net .

       Check that it's working:

           gmake login
           gmake admin

       Browse to the logins/*.html files to see the results.  I use Opera for
       this purpose, since it doesn't get upset with constant changes to the
       security certificates as the inner Elab is swapped in and out over
       time.  Firefox doesn't like the certificate changes.

125 126 127 128 129
       Login cookies go stale after a while.  Nothing will work until you have
       logged in again as admin.  You'll have to repeat this when that happens:

           gmake admin

130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145
     - Run activate:

           gmake activate |& tee activate.wget/activate.log

     - Output of activate actions goes into an activate.wget directory as
         *.html - Pages from wget.
         *.html.cmd - The wget command that was executed.
         *.html.log - The wget log for that command.

       The activate.wget directory gets moved aside to activate.wget.prev .
       If you want more context than that, move them to numbered versions.

     - analyze_activate applies the success/failure pattern files after the run.
       Results go in $twsr/analyze_activate.txt .

           gmake analyze_activate
146 147 148
           ** Activation analysis: success 10, failure 1, problem 1, UNKNOWN 1 **
           ** (See analyze_activate.txt in ../../../../testbed/www/sec-check/results .) **
           **
149 150 151 152 153
     - The success/failure.txt pattern files are also used action-by-action to
       look at the html results in the wget_post makefile function.  When a
       failure occurs, the make stops.  This is a Good Thing, because there
       are a lot of dependencies in the activation sequence.  Most early
       objects must exist for later ones to be created.  When UNKNOWN results
154
       occur, browse to the file to see what was returned and update the patterns.
155

156 157 158 159
       There's no dependency sequence in the activation tasks in the makefile.
       If activate fails, fix the problem and make the remaining targets in the
       activate_tasks list.

160 161 162 163 164 165 166 167 168 169 170 171 172
     - You can browse to the .html files using File/Open, or pop up pages in
       your browser directly from Emacs using this handy keyboard macro:

           (fset 'browse-to-file
              [?\C-a ?\C-b ?\C-\M-s ?^ ?\\ ?w ?\C-a ?\C-c ?\C-o 
               ?\C-a ?\M-d ?\C-d ?\C-d ?\C-\M-y ?d return return ?\C-e ?\C-f])
           (global-set-key "" 'browse-to-file)
           (progn (switch-to-buffer "d") (kill-region (point-min) (point-max))
             (insert-string ; Or in the parent dir for setup/teardown.
               "file://localhost/home/fish/flux/obj-devel/www/sec-check/activate.wget"))
           (setq browse-url-generic-program "opera")
           (global-set-key "\^C\^O" 'browse-url-generic)

173 174 175 176 177 178 179 180 181
       To define it, select the above lines and type M-X eval-region .  It
       will take you to buffer "d", so just ^X-b (switch-to-buffer) to return.
       Remove the *.wget directory suffix if you are in the main directory
       (setup, show, and teardown actions below.)

       Usage: The file path to your current obj-devel directory containing
       html files is in buffer "d".  Put the cursor on a filename line in an
       analyze*.txt file and type ^C-^F to pop up the page in the current
       Opera tab.
182

183 184 185
       Notice that if the resulting .html file contains a javascript
       PageReplace, you will need to comment that out to see what was actually
       returned by PHP, adding "!--" and "--", like this:
186

187 188 189
           <!-- script type='text/javascript' language='javascript'>
           PageReplace('shownode.php3?node_id=pc120');
           </script -->
190

191 192 193 194 195 196 197 198 199 200 201 202 203 204
     - Debugging: You can gmake individual activation tasks when things are
       broken, working through the sequence by hand.  The actions should
       handle the dance of logging in and out and toggling admin privileges.

       Notice that since all of the integer object GUID's were added to the
       Emulab DB schema, there's a lot of ssh'ing mysql commands to myboss to
       fetch _idx values, that are substituted into later page arguments.
       This may be fragile if it keeps changing.

   ----------------
   . spider: clear_wget_dirs do_spider site_list site_msg

     Recursively wget a copy of the ElabInElab site and extract a <forms list.

205 206 207 208
     Logging in and out is handled automatically, to get both the public and
     the private (admin) versions of pages.  As with activate, .prev versions
     of the output directories are kept.

209 210
         gmake spider |& tee spider.log

211 212 213 214 215
     This takes a while to run, since it's getting multiple 10's of megabytes
     of stuff.  It isn't smart enough to get just one copy of forms that are
     linked in multiple times, and have GET arg lists that become part of the
     URL's and hence part of the output filenames as well.

216 217 218 219 220 221 222 223
     - HTML output goes into public.wget and admin.wget subtrees, with a log
       file at the top and a tree named for the server,
       myboss.vulnelab.testbed.emulab.net for me, underneath.  You can browse
       into those files as well.

     - Grepped forms lines go into $tws/{public,admin,site}_{forms,files}.list,
       similar to the src_forms output above.

224 225 226 227 228 229 230
     - gmake site_msg reports something like:

       ** Spider: 1814 ( 3 + 1811 ) forms instances are in 55 ( 3 + 55 ) web pages. **
       ** (See *_{forms,files}.list in ../../../testbed/www/sec-check/results .) **
       **


231 232 233 234 235 236 237 238 239
   ----------------
   . forms_coverage: files_missing forms_msg

     Compare the two lists to find uncovered (unlinked) forms.

         gmake forms_coverage

     - Output goes in $twsr/files_missing.list .

240 241 242 243 244 245
     - gmake forms_msg reports src_msg and site_msg, plus something like:

       ** Forms: 40 out of 89 forms files are not covered. **
       ** (See ../../../testbed/www/sec-check/results/files_missing.list .) **
       **

246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269
     - Generally, unlinked forms are a symptom of an object type (or state)
       that is not yet activated.  Iterate on the activation logic.

       . Look in the sources to find where the missing links should be.

       . Connect to the ElabInElab site from a browser through Spike Proxy.

       . Interactively create DB state that will elicit the uncovered forms.
         - Projects/users awaiting approval, 
         - Experiments swapped in with active nodes, and so on.

       . Capture a list of URL's along with Get or Post inputs for automation.

       . Add steps to the activate: list in the GNUmakefile.in .

       . Re-spider and compare until everything is covered (no more missing forms.)

   ----------------
   . input_coverage: input_list input_msg

     Extract <input fields from spidered forms.

         gmake input_coverage

270 271 272 273 274 275 276 277
     - This reports something like:

       ** Inputs: 10203 input fields, 331 unique, 127 over-ridden. **
       ** (See site_inputs.list and input_names.list
       **  in ../../../testbed/www/sec-check/results,
       ** and input_values.list in ../../../testbed/www/sec-check .) **
       **

278 279
     - This is automatic, but there are a few little special cases in the
       makefile.  It leads into iterating on the input_values.list dictionary.
280
       See the comments about this in README-concepts.txt .
281 282 283

     - Input is the spidered public.wget and admin.wget subtrees.

284 285
     - Output is $twsr/site_inputs.list and $twsr/input_names.list .

286
       . When the inputs to a form are changed (fields, names, or defaults) you can
287 288 289 290
         edit site_inputs.list rather than having to do another whole spider run.

       . input_names.list isn't used directly, but is a source for copying strings
         to add to the $tws/input_values.list control file.
291 292 293 294 295 296 297 298 299 300 301 302 303

       See the comments in $twsr/forms-to-urls.gawk about features controlled
       by special annotations on the input_values.list entries.

   ----------------
   . normal: gen_all run_all analyze

     Create, run, and categorize "normal operations" test cases.

         gmake normal |& tee normal.log

     - Subtasks:

304 305
       "run" subtasks drive the "gen" subtasks by dependencies.  You can
       also do the "gen"s to get the scripts they make.
306 307 308 309

       . Update the $twsr/{setup,show,teardown}_cases.{urls,wget} files.

             gmake gen_all
310
         or:
311 312 313 314 315 316 317 318
                 gmake gen_setup
                 gmake gen_show
                 gmake gen_teardown

         - Inputs are $twsr/site_inputs.list, $tws/input_values.list .
           $tws/{setup,teardown}_forms.list are used for separating out the
           setup and teardown case sequences.

319 320 321 322 323 324
         - There are a couple of files mentioned in the teardown_forms.list
           that aren't actually covered yet, so making teardown_cases.urls
           will complain a bit:
             *** Missing: 24 deleteimageid.php3
             *** Missing: 56 delmmlist.php3

325 326
       . Run the generated {setup,show,teardown}_cases.wget scripts.

327 328 329 330 331 332
         Setup/teardown are sequences creating/deleting ephemeral objects,
         typically with a "3" appended to their names.  Teardown undoes setup,
         and vice-versa of course.

         Show (everything else) is separate and works on the objects made by
         activation.
333 334

             gmake run_all |& tee run_all.log
335
         or:
336 337 338 339 340 341 342 343 344 345 346 347 348 349
                 gmake run_setup |& tee run_setup.log
                 gmake run_show |& tee run_show.log
                 gmake run_teardown |& tee run_teardown.log

       . HTML output goes into individual .html files in the top-level directory.

       . run_all does analyze after each sub-task.  Detailed output goes into
         the $twsr/analyze_output.txt file, and a summary is printed.

         When you edit the $tws/{success,failure,problem}.txt files, you can
         just update the analysis file.

             gmake analyze

350 351 352 353 354 355 356 357 358
         Output, for example, is:

             ** Run analysis: success 16, failure 3, problem 1, UNKNOWN 0 **
             ** (See analyze_output.txt in ../../../testbed/www/sec-check/results .) **
             **

         Look into failures and problems, of course.  See the description of
         browsing to the output files above in the "activate:" section.

359 360 361 362 363 364 365 366 367 368
       . Here, you pretty much iterate on the makefile and all of the scripts
         and control files until everything works smoothly.  (This is a goal
         that is asymptotically approached.)  

         See the comments in README-concepts.txt about what to change.

       . Debugging: It's often useful to regenerate the .wget files and
         copy-paste individual commands into a shell window to "single-step"
         the sequence.

369
         You can just refresh the browser window as the .html output changes.
370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404

         Except, when an Emulab "backgrounded" task finishes, the page gets a
         little bit of "PageReplace" Javascript to redirect to the "show"
         page.  You can edit the .html file to remove or comment out the
         PageReplace, adding "!--" and "--", like this:

           <!-- script type='text/javascript' language='javascript'>
           PageReplace('shownode.php3?node_id=pc120');
           </script -->

   ----------------
   . probe: gen_probes probe_all probes_msg

     Create and run probes to test the checking code of all input fields.

         gmake probe_all |& tee probe_all.log

     This can get really complicated, for reasons described in this section of
     README-concepts.txt .

     - Subtasks:

       . Update the $twsr/{setup,show,teardown}_probe.{urls,wget} files.

             gmake gen_probes

       . probe_setup first does run_teardown to make sure debris is cleaned up.

             gmake probe_setup |& tee probe_setup.log
             gmake probe_show |& tee probe_show.log
             gmake probe_teardown |& tee probe_teardown.log

       . Analysis

             gmake probes_msg