Commit 85c5d474 authored by Robert Ricci's avatar Robert Ricci

Expand on lecture notes

parent a5a2fa01
...@@ -28,6 +28,8 @@ ...@@ -28,6 +28,8 @@
\1 \textit{What mechanisms have you used as monitors?} \1 \textit{What mechanisms have you used as monitors?}
\1 Event driven vs timer driven (tracing vs. sampling) \1 Event driven vs timer driven (tracing vs. sampling)
\2 Event driven: log every time certain types of events occur
\2 Sampling: Wake up every once in a while to see what's going on
\2 \textit{How to decide between them?} \2 \textit{How to decide between them?}
\2 \texttt{gprof}, \texttt{top} \2 \texttt{gprof}, \texttt{top}
\2 Self-time vs. inherited time \textit{When to look at each?} \2 Self-time vs. inherited time \textit{When to look at each?}
...@@ -46,44 +48,62 @@ ...@@ -46,44 +48,62 @@
\1 What do you do when you have to drop measurements? \1 What do you do when you have to drop measurements?
\2 Best to know when you are actually doing so \2 Best to know when you are actually doing so
\2 Put in markers on both ends of event \2 Put in markers on both ends of event
\2 Have to be careful with bias: for example, dropping measurements about
disk I/O when disk is overwhelemed
\1 Common mistakes in benchmarking \1 Common mistakes in benchmarking
\begin{itemize} \2 \emph{Which ones have you made yourself?}
\item Only report average behavior, not variance \2 Only report average behavior, not variance
\item Ignore skewness - certain things get hit harder \2 Ignore skewness - certain things get hit harder
\item Controlling workload inappropriately \2 Controlling workload inappropriately
\item Ignoring caching - are you measuring what you think you are? \2 * Ignoring caching - are you measuring what you think you are?
\item Setting buffer sizes \2 Setting buffer sizes
\item Sampling inaccuracies \2 Sampling inaccuracies
\item Ignore monitoring overhead \2 Ignore monitoring overhead
\item Not validating measurements \2 * Not validating measurements
\item Different initial conditions \2 * Different initial conditions
\item Not measuring transient performance \2 Not measuring transient performance
\item Using device utilizations for performance comparisons \2 Using device utilizations for performance comparisons
\item Too much data, not enough analysis \2 Too much data, not enough analysis
\end{itemize}
\1 I'd add: \1 I'd add:
\2 Strawman baselines \2 Strawman baselines
\3 Compare to something no one would ever do \3 Compare to something no one would ever do
\3 These actually make your paper weaker, but people still do it all
the time
\3 Examples: Brute force search on an ordered list, random
ordering/assignment, ignoring parallel processing, comparing to
code compiled without optimizations turned on, etc.)
\3 \textit{other examples?}
\3 Justify your baselines based on previous work, common practice, or \3 Justify your baselines based on previous work, common practice, or
common sense common sense
\2 Comparing optimized vs. non-optimized cases (tuned vs. not tuned) \2 Comparing optimized vs. non-optimized cases (tuned vs. not tuned)
\3 You know your system a lot better \3 You know your system a lot better
\3 Ideally both should be set up by an expert
\3 \textit{What can you do defensively about this} (make exact
environment and/or tuning parameters available)
\3 \textit{What can you do when comparing to another system?}
\2 Cherry-picking
\3 Taking evidence (eg. specific benchmarks) that support your claim
and ignoring or hiding those that don't
\3 One way out: scope your claims and make a convincing argument that
the scope is important - eg. we perform better under high node
churn, we perform better on a write-heave workload, etc.
\3 You are actually learning something valuable!
\2 Not measuring what you think you're measuring \2 Not measuring what you think you're measuring
\3 Caching just a special case \3 Caching just a special case
\3 Making bad assumptions about where bottlenecks are \3 Making bad assumptions about where bottlenecks are
\3 Making bad assumptions about what resources are being used \3 Making bad assumptions about what resources are being used
\3 Example: TCP window size / BDP, not actual available bandwidth
\3 \textit{How can you tell?} (Intuition, check against analytical
analysis, check resource usage on supposed bottleneck, look for
suspicious thresholds)
\2 \textit{Any other ideas?} \2 \textit{Any other ideas?}
\1 For next time \1 For next time
\2 Read Chapter 10 (data presentation) \2 Read Chapter 10 (data presentation)
\2 Finish HW \#4 \2 Finish HW \#4
\2 First bundle of papers posted - clone the assignment repo \2 Keep working on the papers
\3 Answer the early questions as you go through the paper---write your
expectations before you finish reading the paper
\3 Start early, it will be a lot of work
\3 You may use my LaTeX form if you want, instructions in the handout
\end{outline} \end{outline}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment