Commit f7cba233 authored by Robert Ricci's avatar Robert Ricci

Some more big lessons from the class

parent 51ac0ab4
......@@ -20,13 +20,18 @@
\2 Overall point of systems evaluation
\3 Providing data necessary to make a decision
\3 Presenting it on a convincing manner
\3 Performance evaluation is not the only relevant data, but it's
important
\2 Understand what you are measuring
\3 Understand what make convincing evidence
\3 Understand what you need to measure and how to measure it
\3 Do some baseline tests of your workload generator and monitors to
make sure they exhibit expected results
\2 Understand the system under test
\3 Where are the boundaries?
\3 What is inside those boundaries that you are actually measuring?
\3 How to the SUT boundaries relate to a deployment environment?
\3 How do the SUT boundaries relate to a deployment environment?
\3 How do the SUT boundaries relate to the claims of the paper?
\2 Evaluation should be a part of the research and development process
\3 Convince yourself with data, not just bias
\3 Often need preliminary evaluations
......@@ -34,12 +39,14 @@
\3 The more evaluation you do along the way, the less biased you are
likely to be
\2 Recognize the strengths and weaknesses in evaluations that you read
\3 There is a huge difference between active and passive reading
\3 Think actively about what you need to see to be convinced
\3 Look for biases or basic mistakes
\2 Common mistakes in systems evaluation
\3 No goals or biased goals
\3 Ignoring significant factors
\3 Analysis without understanding the problem
\3 Analysis without understanding the problem---a reason to have a
concrete problem statement
\3 No sensitivity analysis
\3 Ignoring variability
\2 Use the statistical tools available to you
......@@ -47,13 +54,19 @@
\3 For understanding the confidence in your results
\3 For showing difference or sameness in results
\3 The question is ``have you found something real?''
\2 Use the tools available to you
\3 Reproducibility is a big deal---not just for others, but for
yourself too
\3 Assuming your environment is fragile forces you to build more
reproducible research
\3 Don't just eyeball graphs
\4 Compute CIs
\4 Do linear regressions
\4 Compares means, percentiles, etc.
\2 Work done to make work repeatable helps everyone
\3 Not just for others, but for yourself too
\3 Lets you re-run things when something changes
\3 Makes it easier for others to build on your work
\3 These things bitrot incredibly fast
\3 The closer you can get to ``experiment as function call'', the
happier you will be
\3 Assuming your environment is fragile forces you to build more
repeatable research
\3 Keep track of everything
\2 You are running experiments all the time, the only question is whether
you are learning anything from them
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment