Commit 607d72d9 authored by Robert Ricci's avatar Robert Ricci

Second draft of final lecture notes

parent 85c4be28
......@@ -21,8 +21,40 @@
\begin{outline}
\1 Review of topics covered
\2 TODO fill in
\2 Overall point of systems evaluation
\3 Providing data necessary to make a decision
\3 Presenting it on a convincing manner
\2 Understand what you are measuring
\3 Understand what make convincing evidence
\3 Understand what you need to measure and how to measure it
\2 Understand the system under test
\3 Where are the boundaries?
\3 What is inside those boundaries that you are actually measuring?
\3 How to the SUT boundaries relate to a deployment environment?
\2 Evaluation should be a part of the research process
\3 Convince yourself with data, not just bias
\2 Recognize the strengths and weaknesses in evaluations that you read
\3 Think actively about what you need to see to be convinced
\3 Look for biases or baisic mistakes
\2 Common mistakes in systems evaluation
\3 No goals or biased goals
\3 Ignoring significant factors
\3 Analysis without understanding the problem
\3 No sensitivity analysis
\3 Ignoring variability
\2 Use the statistical tools available to you
\3 For selecting the number of runs
\3 For understanding the confidence in your results
\3 For showing difference or sameness in results
\3 The question is ``have you found something real?''
\2 Use the tools available to you
\3 Reproducibility is a big deal---not just for others, but for
yourself too
\3 Assuming your environment is fragile forces you to build more
reproducible research
\3 The closer you can get to ``experiment as function call'', the
happier you will be
\3 Keep track of everything
\1 Go through example of a nice report
\2 By no means perfect, but better than most
......@@ -46,8 +78,10 @@
\2 What did we to do much of?
\2 What did we not do enough of?
\2 How was the balance between the book, papers, and labs?
\2 What do you see yourself potentially using
\2 What do you see yourself potentially using?
\2 Not using ever?
\2 How about the book?
\2 Submission system
\end{outline}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment