Commit e43fddea authored by Russ Fish's avatar Russ Fish

Checkpoint from before the NSDI crunch.

parent 4df8b160
This diff is collapsed.
......@@ -47,15 +47,25 @@ sec-check/README-howto.txt - Documentation outline.
- Forms coverage
. Grep the sources for <form and make up a list of php form files.
gmake src_forms
Creates: src_forms.list, src_files.list
- 105 separate forms are on 95 php code pages (plus 7 "extras" on Boss.)
gmake src_msg
. Spider a copy of the EinE site with wget and extract its forms list.
Have to edit the EinE experiment details into the makefile.
It's better to change your password in the EinE than put it in the makefile.
See GNUmakefile.in for details.
gmake login
gmake spider
gmake site_forms
Creates: admin.wget subdir, site_forms.list, site_files.list
- 40 "base" forms are visible once logged in as user, 47 with admin on.
gmake site_msg
. Compare the two lists to find uncovered (unlinked) forms.
gmake forms_coverage
Creates: files_missing.list
gmake forms_msg
. Create a script to activate the EinE site to turn on all forms.
- Look in the sources to find where the missing links should be.
......@@ -64,7 +74,7 @@ sec-check/README-howto.txt - Documentation outline.
. Projects/users awaiting approval,
. Experiments swapped in with active nodes, and so on.
- Capture a list of URL's along with Get or Post inputs for automation.
- Convert the list into an wget script and/or WebInject test cases.
- Add steps to the activate: list in the GNUmakefile.in .
. Re-spider and compare until everything is covered (no more missing forms.)
gmake spider
......@@ -73,11 +83,23 @@ sec-check/README-howto.txt - Documentation outline.
- Input fields coverage
. Grep spidered forms for <input definitions and devise acceptable values.
gmake input_coverage
Creates: site_inputs.list, input_names.list
You make: input_values.list
At first, Copy input_names.list to input_values.list,
then edit default values onto the lines for auto-form-fill-in.
After the first time, you can merge new ones into input_values.list .
- 1631 <input lines in admin-base, 511 unique, with 156 unique field names.
gmake input_msg
- But only 78 of the unique field names are text fields.
. Convert the list to WebInject XML test cases submitting input field values.
. Test using WebInject until "normal" input tests work properly in all forms.
- "normal" test cases
. Convert the list to test cases submitting input field values.
gmake gen_normal
Creates: site_normal.urls, normal_cases.xml
. Test until "normal" input tests work properly in all forms.
gmake run_normal
Creates: normal_output.xml
- Probe the checking code of all input fields for SQL injection holes
. Generate WebInject cases with SQL injection probes in individual fields.
......
archive_view.php3
beginexp_html.php3
boot.php3
delaycontrol.php3
deletegroup.php3
deleteimageid.php3
deleteosid.php3
deletepubkey.php3
deleteuser.php3
editexp.php3
editgroup_form.php3
editimageid.php3
endexp.php3
feedback.php3
freenode.php3
freezeuser.php3
gensslcert.php3
joinproject.php3
kb-manage.php3
kb-search.php3
linktest.php3
loadimage.php3
modifyexp.php3
moduserinfo.php3
newgroup_form.php3
newimageid_ez.php3
newmmlist.php3
newnodelog_form.php3
newnodes_list.php3
newosid_form.php3
newproject.php3
nodecontrol_form.php3
nscheck_form.php3
nsgen.php3
plab_ez.php3
powertime.php3
replayexp.php3
showpubkeys.php3
showsfskeys.php3
showsumstats.php3
showuser_list.php3
swapexp.php3
template_create.php
updateaccounts.php3
This diff is collapsed.
This diff is collapsed.
================ success ================
beginexp_html.php3.html:<b>Starting experiment configuration!</b> Since you are
boot.php3.html:reboot (pc4): Attempting to reboot ...
gensslcert.php3.html:Your new SSL certificate has been created. You can
kb-manage.php3.html:<center><b>Knowledge Base Entry: 261 test_tag</b><br>(<a hre
loadimage.php3.html: <title>MyEmulab.Net - Snapshot Node Disk into Existing Ima
loadimage.php3.html:Snapshot Node Disk into Existing Image Descriptor</h2>
newnodelog.php3.html: Log for node pc4.
================ failure ================
archive_missing.php3.html:Could not continue. Please contact <a href="mailto:tes
editexp.php3.html: &nbsp;Oops, please fix the following errors!&nbsp;
editimageid.php3.html: &nbsp;Oops, please fix the following errors!&nbsp;
editimageid.php3.html: <font color=red>Must select at least one type</font></td>
joinproject.php3.html: &nbsp;Oops, please fix the following errors!&nbsp;
kb-search.php3.html: Invalid page arguments: /kb-search.php3?submit=Submit%20Que
modifyexp.php3.html: The experiment T10000-1 is not a valid experiment in projec
moduserinfo.php3.html: &nbsp;Oops, please fix the following errors!&nbsp;
newgroup.php3.html: Missing field; Please go back and fill out the "Group Name"
newimageid_ez.php3.html: &nbsp;Oops, please fix the following errors!&nbsp;
newimageid_ez.php3.html: <font color=red>Must select at least one type</font></t
newmmlist.php3.html: &nbsp;Oops, please fix the following errors!&nbsp;
newosid.php3.html: Missing field; Please go back and fill out the "Descriptor Na
newproject.php3.html: &nbsp;Oops, please fix the following errors!&nbsp;
nodecontrol.php3.html:Could not continue. Please contact <a href="mailto:testbed
swapexp.php3.html: The experiment T10000-1 is not a valid experiment in project
template_analyze.php.html: The experiment template 10000/1 is not a valid experi
template_export.php.html: The experiment template 10000/1 is not a valid experim
template_modify.php.html: The experiment template 10000/1 is not a valid experim
template_swapin.php.html: The experiment template 10000/1 is not a valid experim
================ UNKNOWN ================
admin.html
approveuser.php3.html
archive_missing.php3.html
archive_tag.php3.html
archive_tags.php3.html
beginexp_html.php3.html
boot.php3.html
editexp.php3.html
editimageid.php3.html
feedback.php3.html
gensslcert.php3.html
joinproject.php3.html
kb-manage.php3.html
kb-search.php3.html
loadimage.php3.html
login.php3.html
login_fish.html
modifyexp.php3.html
moduserinfo.php3.html
newgroup.php3.html
newimageid_ez.php3.html
newmmlist.php3.html
newnodelog.php3.html
newosid.php3.html
newproject.php3.html
nodecontrol.php3.html
nscheck.php3.html
nsgen.php3.html
plab_ez.php3.html
powertime.php3.html
showpubkeys.php3.html
showsfskeys.php3.html
showsumstats.php3.html
showuser_list.php3.html
swapexp.php3.html
template_analyze.php.html
template_create.php.html
template_export.php.html
template_metadata.php.html
template_modify.php.html
template_swapin.php.html
updateaccounts.php3.html
Invalid form arguments
Invalid page arguments
No tags
Could not continue
You are not authorized
please fix the following errors
Must select at least one type
You do not appear to be logged in
Improper query type
is not a valid
Missing field
< approveproject_form.php3
< approveuser_form.php3
< approvewauser_form.php3
< archive_missing.php3
< archive_tag.php3
< cdromnewkey.php
< cdromqueue.php3
< cdromrequest.php3
< chpasswd.php3
< deletenodelog.php3
< deleteproject.php3
< deletesfskey.php3
< delmmlist.php3
< editnodetype.php3
< editsitevars.php3
< expaccess_form.php3
< floormap.php3
< menu.php3
< modnodeattributes_form.php3
< newimageid.php3
< newnode_edit.php3
< news.php3
< nodemon.php3
< nodemon_all.php3
< panicbutton.php3
< prereserve_node.php3
< remapexp.php3
< request_idleinfo.php3
< request_swapexp.php3
< robotmap.php3
< search.php3
< setnodeloc.php3
< showstuff.php3
< survey.php3
< template_analyze.php
< template_commit.php
< template_export.php
< template_exprun.php
< template_metadata.php
< template_modify.php
< template_swapin.php
< verifyusr_form.php3
< widearea_nodeinfo.php3
< widearea_register.php
< wideareakeys.php3
#! /usr/bin/awk -f
FNR == 1 {
form=0;
# Exempt forms in twik and flyspray files.
exempt = FILENAME ~ "/(twiki|flyspray)/";
if ( exempt ) next;
if (NR != 1) printf "\n";
print FILENAME;
}
/<form/ && ! exempt && !/action=[^ ]*\/search.php3/ {
form=1;
sub(".*<form", "<form"); # Put <form at beginning of line.
sub("[ \t]on[a-zA-Z]+=.*['\"]", "", $0 ); # Skip Javascript.
while ( !match($0, ">") ) { # Multi-line <form statements.
sub("[ \t]*$", " "); # Single space at end of line.
getline ln;
sub("[ \t]on[a-zA-Z]+=.*['\"]", "", ln ); # Skip Javascript.
sub("^[ \t]*", "", ln); # No space on start of new line.
$0 = $0 ln;
}
sub(">.*", ">"); # Leave only <form ... > on the line.
print;
}
form && /<input/ {
sub(".*<input", "<input"); # Put <input at beginning of line.
sub("[ \t]on[a-zA-Z]+=.*['\"]", "", $0 ); # Skip Javascript.
while ( !match($0, ">") ) { # Multi-line <input statements.
sub("[ \t]*$", " "); # Single space at end of line.
getline ln;
sub("[ \t]on[a-zA-Z]+=.*['\"]", "", ln ); # Skip Javascript.
sub("^[ \t]*", "", ln); # No space on start of new line.
$0 = $0 ln;
}
sub(">.*", ">"); # Leave only <input ... > on the line.
# Canonicalize.
sub("type=readonly", "type=text"); # There is no readonly type, text is default.
# Convert single-quoted type and name values to double quotes.
$0 = gensub("(name|type)='([^']+)'", "\\1=\"\\2\"", "g");
# Quote unquoted values.
$0 = gensub("(name|type|value)=([^'\"][^ >]+)", "\\1=\"\\2\"", "g");
# Reorder: <input type=.* name=.* value=.* .*>
$0 = gensub("<input (.*)value=('[^']+'|\"[^\"]+\")", "<input value=\\2 \\1", 1);
$0 = gensub("<input (.*)name=('[^']+'|\"[^\"]+\")", "<input name=\\2 \\1", 1);
$0 = gensub("<input (.*)type=('[^']+'|\"[^\"]+\")", "<input type=\\2 \\1", 1);
gsub(" *", " "); # Collapse extra spaces.
print;
}
/<\/form/ { form=0 }
{next}
#! /usr/bin/gawk -f
#
# EMULAB-COPYRIGHT
# Copyright (c) 2000-2006 University of Utah and the Flux Group.
# All rights reserved.
#
# form-input.gawk - Scan a lot of spidered HTML files, extracting forms.
#
# Output format:
# - Spidered filename lines are from wget: ./host.path/page.php?getargs
# - Files may have multiple form sections, terminated by blank lines.
# - Each form section has one <form line, possibly many <input lines.
# - Attributes of <input elements are canonicalized and reordered:
# <input type="..." name="..." value=... ...>
# - <textarea become <input type="textarea" for uniformity.
# - <select become <input type="select" for uniformity.
# Beginning of file.
FNR == 1 {
end_form(); # In case we missed a </form>.
form=0;
# Exempt forms in twik and flyspray files. Can't get to the DB.
if ( FILENAME ~ "/(twiki|flyspray)/" ) nextfile;
print FILENAME;
}
{ gsub("<textarea", "<input type=textarea"); }
{ gsub("<select", "<input type=select"); }
# Rarely, a mix of <input and <form elements on a single line within a form.
form && /<input.*<form/ {
while ( $0 ~ "<input.*<form" || $0 ~ "</form.*<form" ) {
##print "INPUT/FORM", $0;
if ( index($0, "<input") < index($0, "<form") )
do_input();
else {
if ( index($0, "</form") < index($0, "<form") )
end_form();
else {
do_form();
}
}
}
}
# Start of form.
/<form/ && ! form { do_form(); }
function do_form() {
# Handle the searchbox form that's on every page, elsewhere.
if ( $0 ~ "action=[^ ]*\\/search.php3" ) return
form=1;
$0 = substr($0, index($0, "<form") ); # Put <form at beginning of line.
while ( $0 !~ ">" ) { # Multi-line <form elements.
sub("[ \t]*$", " "); # Single space at end of line.
getline ln;
sub("^[ \t]*", "", ln); # No space on start of new line.
$0 = $0 ln;
}
# Skip Javascript "on...=..." and "style=...".
sub("[ \t]on[a-zA-Z]+=('[^']+'|\"[^\"]+\")", "", $0 );
sub("[ \t]style=('[^']+'|\"[^\"]+\")", "", $0 );
end = index($0, ">");
rest = substr($0, end+1); # Save rest of line.
##print "'" rest "'";
$0 = substr($0, 1, end); # Leave only <form ... > on the line.
# Canonicalize.
# Convert single-quoted action and method values to double quotes.
$0 = gensub("(method|action)='([^']+)'", "\\1=\"\\2\"", "g");
# Quote unquoted values.
$0 = gensub("(method|action)=([^'\"][^ >]+)", "\\1=\"\\2\"", "g");
print;
# May have more elements on the same line.
$0 = rest;
##print "FORM", $0;
}
form && /<input/ { while ( $0 ~ "<input" ) do_input(); }
# May be multiple <input elements on a line.
function do_input() {
##print "INPUT", $0;
$0 = substr($0, index($0, "<input")); # Put <input at beginning of line.
sub("[ \t]on[a-zA-Z]+=.*['\"]", "", $0 ); # Skip Javascript.
# Collect multi-line <input elements.
while ( $0 !~ ">" ||
# <select blocks need a value from an <option.
$0 ~ "type=select" && $0 !~ "value=" ) {
sub("[ \t]*$", " "); # Single space at end of line.
getline ln;
sub("[ \t]on[a-zA-Z]+=.*['\"]", "", ln ); # Skip Javascript.
sub("^[ \t]*", "", ln); # No space on start of new line.
$0 = $0 ln;
}
# <select blocks need a value from an <option.
$0 = gensub("(<input type=select.*name=[^ >]+).*(value=[^ >]+).*>",
"\\1 \\2>", 1);
##print "FULL INPUT", $0;
end = index($0, ">");
rest = substr($0, end+1); # Save rest of line.
##print "'" rest "'";
$0 = substr($0, 1, end); # Leave only <input ... > on the line.
# Canonicalize.
sub("type=readonly", "type=text"); # There is no readonly type, text is default.
# Convert single-quoted type and name values to double quotes.
$0 = gensub("(name|type)='([^']+)'", "\\1=\"\\2\"", "g");
# Quote unquoted values.
$0 = gensub("(name|type|value)=([^'\"][^ >]*)", "\\1=\"\\2\"", "g");
# Reorder: <input type=.* name=.* value=.* .*>
$0 = gensub("<input (.*)value=('[^']+'|\"[^\"]+\")", "<input value=\\2 \\1", 1);
$0 = gensub("<input (.*)name=('[^']+'|\"[^\"]+\")", "<input name=\\2 \\1", 1);
$0 = gensub("<input (.*)type=('[^']+'|\"[^\"]+\")", "<input type=\\2 \\1", 1);
gsub(" *", " "); # Collapse extra spaces.
# Filter out Delay Control entries.
if ( $0 !~ "DC::" )
print;
# May have more elements on the same line.
$0 = rest;
}
/<\/form/ { # End of form.
end_form();
}
function end_form() {
# Blank-line terminator on each form section.
if ( form ) {
printf "\n";
end = index($0, ">");
$0 = substr($0, end+1); # Save rest of line.
}
form = 0;
}
{next}
#! /usr/bin/gawk -f
#
# EMULAB-COPYRIGHT
# Copyright (c) 2000-2006 University of Utah and the Flux Group.
# All rights reserved.
#
# forms-to-urls - Generate URL's for accessing the site.
#
# form-input.gawk's output format is the input format for this script.
#
# A site_values.list file path is provided by a -v VALUES= awk arg.
# Contents are 'name="..." value'. Optional value (to end of line)
# is used for auto-form-fill-in.
#
# Output is a set of page URL's including appended ?args.
# The Get arg method is default. Post is indicated by a post: prefix.
#
# A -v MAX_TIMES= awk arg specifies how many times to target a form.
#
BEGIN {
if ( ! MAX_TIMES ) MAX_TIMES = 1; # Default.
while ( getline <VALUES ) {
arg_name = $1;
arg_name = gensub("name=\"([^\"]*)\"", "\\1", 1, arg_name);
###arg_name = gensub("formfields\\[(.*)\\]", "\\1", 1, arg_name);
if (NF > 1)
defaults[arg_name] = substr($0, index($0, $2));
##printf "defaults %s=%s.\n", arg_name, defaults[arg_name];
}
}
/^[.][\/]/ { # A page file section starts with a filename.
# Remember the host path from filenames. Often not on <form action= .
host_path = gensub("^[.][/](.*)/.*", "\\1", 1);
##print "host_path", host_path;
}
/^<form/ { # <form action="..." method="..."
action = gensub(".* action=\"([^\"]*)\".*", "\\1", 1);
method = gensub(".* method=\"([^\"]*)\".*", "\\1", 1);
# Action= URL can have args specified. Use the values over anything else.
url = action;
delete args;
if ( q = index(action, "?") ) {
url = substr(action, 1, q-1);
# The "&" arg separator is escaped in HTML.
n = split(substr(action, q+1), url_args, "&amp;");
for (i = 1; i <= n; i++) {
name_val = url_args[i];
eq = index(name_val, "=");
nm = substr(name_val, 0, eq-1);
vl = substr(name_val, eq+1);
args[nm] = vl;
# A default with a ! prefix over-rides an action= arg.
df = defaults[nm];
if ( df ~ "!" )
args[nm] = substr(df, 2);
##printf "name_val %s, nm %s, vl %s, df %s\n", name_val, nm, vl, df;
}
}
# Add host path to relative url's.
if (! index(url, ":") ) url = "https://" host_path "/" url;
##printf "url %s, method %s, args", url, method;
##for (i in args) printf " %s", args[i]; printf "\n";
target[url]++;
form = target[url] <= MAX_TIMES; # Limit target hits.
arg_vals = 0; # Count arguments with user provided values.
}
form && /^<input/ { # <input type="..." name="..." value=... ...>
# Gotta have a name to be an arg.
if ( $0 !~ " name=" ) next;
# Type and name have been double-quoted. Value can be single- or double-.
type = gensub(".* type=\"([^\"]*)\".*", "\\1", 1);
name = gensub(".* name=\"([^\"]*)\".*", "\\1", 1);
if ( $0 ~ " value=\"" )
value = gensub(".* value=\"([^\"]*)\".*", "\\1", 1);
else if ( $0 ~ " value='" )
value = gensub(".* value='([^']*)'.*", "\\1", 1);
else value = "";
##printf "type %s, name %s, value %s\n", type, name, value;
val_arg = (type=="text" || type=="textarea" || type=="password" ||
type=="hidden" || type=="checkbox" || type=="select");
# Follow just the positive submit controls, not cancel, etc.
sub_arg = (type=="submit" &&
(value ~ "Submit" || value ~ "Create" ||
value=="Confirm" || value=="Go!"));
if ( val_arg || sub_arg ) {
arg_name = name; ### gensub("formfields\\[(.*)\\]", "\\1", 1, name);
##printf "arg_name %s, default=%s, value=%s.\n",
## arg_name, defaults[arg_name], value;
df = defaults[arg_name];
if ( df != "" ) {
# Default value from VALUES file. May have ! prefix.
if ( df ~ "!" )
args[arg_name] = substr(df, 2);
else
args[arg_name] = df;
}
else if ( value != "" )
# Value from <input field default.
args[arg_name] = value;
else
args[arg_name] = "";
if ( args[arg_name] ) arg_vals++;
}
}
form && /^$/ { # Blank line terminates each form section.
arg_str = "";
for (arg in args) {
###if ( args[arg] != "" )
if ( arg_str == "" ) arg_str = arg "=" args[arg];
else arg_str = arg_str "&" arg "=" args[arg];
}
post = (method=="post" ? "post:" : "");
if (arg_vals) # Ignore if no argument values to supply.
print post url "?" arg_str;
}
name="MAX_FILE_SIZE"
name="OS"
name="action"
name="add_testuser"
name="addnumber"
name="beginexp"
name="change_testuser"
name="clear_bootstrap"
name="clear_last"
name="def_boot_cmd_line"
name="def_boot_osid"
name="description"
name="dochange"
name="eid"
name="email"
name="eventrestart"
name="exp_pideid"
name="exptidx"
name="formfields[autoswap]"
name="formfields[autoswap_timeout]"
name="formfields[batchmode]"
name="formfields[body]"
name="formfields[canfail]"
name="formfields[copyid]"
name="formfields[count]"
name="formfields[cpu_usage]"
name="formfields[description]"
name="formfields[exp_autoswap]"
name="formfields[exp_autoswap_timeout]"
name="formfields[exp_batched]"
name="formfields[exp_branch]"
name="formfields[exp_description]"
name="formfields[exp_gid]"
name="formfields[exp_id]"
name="formfields[exp_idleswap]"
name="formfields[exp_idleswap_timeout]"
name="formfields[exp_linktest]"
name="formfields[exp_localnsfile]"
name="formfields[exp_noidleswap_reason]"
name="formfields[exp_noswap_reason]"
name="formfields[exp_pid]"
name="formfields[exp_preload]"
name="formfields[exp_savedisk]"
name="formfields[exp_swappable]"
name="formfields[faq_entry]"
name="formfields[fullname]"
name="formfields[gid]"
name="formfields[idleswap]"
name="formfields[idleswap_timeout]"
name="formfields[imagename]"
name="formfields[joining_uid]"
name="formfields[linktest_level]"
name="formfields[listname]"
name="formfields[loadpart]"
name="formfields[localnsfile]"
name="formfields[max_concurrent]"
name="formfields[mem_usage]"
name="formfields[mtype_pc600]"
name="formfields[new_section]"
name="formfields[node]"
name="formfields[noidleswap_reason]"
name="formfields[notes]"
name="formfields[op_mode]"
name="formfields[os_feature_ipod]"
name="formfields[os_feature_isup]"
name="formfields[os_feature_linktest]"
name="formfields[os_feature_ping]"
name="formfields[os_feature_ssh]"
name="formfields[os_name]"
name="formfields[os_version]"
name="formfields[path]"
name="formfields[pid]"
name="formfields[proj_URL]"
name="formfields[proj_funders]"
name="formfields[proj_head_uid]"
name="formfields[proj_linked]"
name="formfields[proj_members]"
name="formfields[proj_name]"
name="formfields[proj_pcs]"
name="formfields[proj_plabpcs]"
name="formfields[proj_public]"
name="formfields[proj_ronpcs]"
name="formfields[proj_why]"
name="formfields[proj_whynotpublic]"
name="formfields[resusage]"
name="formfields[section]"
name="formfields[shared]"
name="formfields[target_uid]"
name="formfields[tid]"
name="formfields[title]"
name="formfields[type]"
name="formfields[units]"
name="formfields[user_interface]"
name="formfields[usr_URL]"
name="formfields[usr_addr2]"
name="formfields[usr_addr]"
name="formfields[usr_affil]"
name="formfields[usr_city]"
name="formfields[usr_country]"
name="formfields[usr_email]"
name="formfields[usr_key]"
name="formfields[usr_name]"
name="formfields[usr_phone]"
name="formfields[usr_shell]"
name="formfields[usr_state]"
name="formfields[usr_title]"
name="formfields[usr_zip]"
name="formfields[w_password1]"
name="formfields[w_password2]"
name="formfields[when]"
name="formfields[wholedisk]"
name="formfields[wikiname]"
name="formfields[xref_tag]"
name="group_description"
name="group_id"
name="group_leader"
name="group_pid"
name="imageid"
name="level"
name="log_entry"
name="log_type"
name="modbase"
name="mode"
name="newprefix"
name="newtype"
name="node"
name="node_id"
name="node_type"
name="nodeid"
name="nodes[]"
name="nodetype"
name="nsdata"
name="op_mode"
name="os_feature_ipod"
name="os_feature_isup"
name="os_feature_linktest"
name="os_feature_ping"
name="os_feature_ssh"
name="os_magic"
name="os_path"
name="os_version"
name="osid"
name="osname"
name="phone"
name="pid"
name="poweron"
name="query"
name="range"
name="reboot"
name="referrer"
name="remap[0]"
name="remap[1]"
name="remap[2]"
name="remap[3]"
name="remap[4]"
name="remap[5]"
name="remap[6]"
name="rpms"
name="searchfor"
name="showby"
name="showtype"
name="sortby"
name="startupcmd"
name="tarballs"
name="target_pid"
name="target_uid"
name="template"
name="templatevalues[Count]"
name="templatevalues[HWType]"