Commit 7a093b06 authored by Robert Ricci's avatar Robert Ricci

Lecture notes for lecture #12

parent 8468812a
......@@ -22,6 +22,86 @@
\1 Left over from lat time
\1 Overall goal of experiment design:
\2 Learn as much as possible from as few experiments as possible
\1 Some terminology
\2 Response variable: outcome
\3 \textit{Why call it a variable?}
\3 \textit{Examples of non-performance variables?}
\2 Factors: things you change
\3 \textit{Why call them predictor variables?}
\2 Primary / secondary factors
\3 \textit{How to decide which ones to use?}
\2 Replication: How many reactions
\1 Important properties of experiment design
\2 Every experiment should get you closer to answering one of the questions
\2 You should be able to explain all behavior in the results---if not, you
may need more experiments
\2 Control all variables you can
\2 Measure the variables you can't control
\1 Interacting factors
\2 Understand which of your factors interact, and which are independent
\2 Saves you a lot of time not running experiments that don't reveal more
information
\2 May take a few experiments to determine
\2 A good (negative) example the FV paper
\2 If you know for sure they are independent, make sure to say so in the
paper
\1 Common mistakes
\2 Ignoring variation in experimental error
\2 Not controlling params
\2 One factor at a time experiments
\2 Not isolating effects
\2 Too many experiments
\3 Break into several sub-evals to answer questions, evaluate particular
pieces of the SUT
\1 Discussing design of lab 1
\2 Our goal: Which variant of TCP should I run on my webserver?
\2 Congestion control: cubic vs. newreno
\2 SACK or no SACK? (orthogonal to CC algo)
\2 ``Doesn't make a difference'' is an okay answer, but have to prove it.
\1 Questions to answer
\2 Which provides my clients the best experience?
\3 Low latency to load page
\3 High throughput
\2 Which allows me to server more users?
\3 Resources on server
\3 Fairness between clients
\1 Metrics
\2 Time to download one page
\2 Error rate
\2 Throughput
\2 Jain's fairness index
\2 Resource usage on server (eg. CPU)
\1 Parameters and factors
\2 TCP parameters
\2 Number of simultaneous clients
\2 File size distribution
\2 Which webserver?
\2 Client load generator?
\2 Packet loss
\2 Client RTT
\2 Client bandwidth
\1 Tools to use
\2 Webserver: which one (apache 2.2?)
\2 \texttt{tc}
\2 Client workload generator (\texttt{httperf?})
\1 Experiments to to run
\2 Determine whether SACK and CC are interacting factors
\2 Max out number of clients
\1 How to present results
\1 For next time
\2 Finish paper analysis before class
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment