Commit 85c5d474 authored by Robert Ricci's avatar Robert Ricci

Expand on lecture notes

parent a5a2fa01
......@@ -28,6 +28,8 @@
\1 \textit{What mechanisms have you used as monitors?}
\1 Event driven vs timer driven (tracing vs. sampling)
\2 Event driven: log every time certain types of events occur
\2 Sampling: Wake up every once in a while to see what's going on
\2 \textit{How to decide between them?}
\2 \texttt{gprof}, \texttt{top}
\2 Self-time vs. inherited time \textit{When to look at each?}
......@@ -46,44 +48,62 @@
\1 What do you do when you have to drop measurements?
\2 Best to know when you are actually doing so
\2 Put in markers on both ends of event
\2 Have to be careful with bias: for example, dropping measurements about
disk I/O when disk is overwhelemed
\1 Common mistakes in benchmarking
\begin{itemize}
\item Only report average behavior, not variance
\item Ignore skewness - certain things get hit harder
\item Controlling workload inappropriately
\item Ignoring caching - are you measuring what you think you are?
\item Setting buffer sizes
\item Sampling inaccuracies
\item Ignore monitoring overhead
\item Not validating measurements
\item Different initial conditions
\item Not measuring transient performance
\item Using device utilizations for performance comparisons
\item Too much data, not enough analysis
\end{itemize}
\2 \emph{Which ones have you made yourself?}
\2 Only report average behavior, not variance
\2 Ignore skewness - certain things get hit harder
\2 Controlling workload inappropriately
\2 * Ignoring caching - are you measuring what you think you are?
\2 Setting buffer sizes
\2 Sampling inaccuracies
\2 Ignore monitoring overhead
\2 * Not validating measurements
\2 * Different initial conditions
\2 Not measuring transient performance
\2 Using device utilizations for performance comparisons
\2 Too much data, not enough analysis
\1 I'd add:
\2 Strawman baselines
\3 Compare to something no one would ever do
\3 These actually make your paper weaker, but people still do it all
the time
\3 Examples: Brute force search on an ordered list, random
ordering/assignment, ignoring parallel processing, comparing to
code compiled without optimizations turned on, etc.)
\3 \textit{other examples?}
\3 Justify your baselines based on previous work, common practice, or
common sense
\2 Comparing optimized vs. non-optimized cases (tuned vs. not tuned)
\3 You know your system a lot better
\3 Ideally both should be set up by an expert
\3 \textit{What can you do defensively about this} (make exact
environment and/or tuning parameters available)
\3 \textit{What can you do when comparing to another system?}
\2 Cherry-picking
\3 Taking evidence (eg. specific benchmarks) that support your claim
and ignoring or hiding those that don't
\3 One way out: scope your claims and make a convincing argument that
the scope is important - eg. we perform better under high node
churn, we perform better on a write-heave workload, etc.
\3 You are actually learning something valuable!
\2 Not measuring what you think you're measuring
\3 Caching just a special case
\3 Making bad assumptions about where bottlenecks are
\3 Making bad assumptions about what resources are being used
\3 Example: TCP window size / BDP, not actual available bandwidth
\3 \textit{How can you tell?} (Intuition, check against analytical
analysis, check resource usage on supposed bottleneck, look for
suspicious thresholds)
\2 \textit{Any other ideas?}
\1 For next time
\2 Read Chapter 10 (data presentation)
\2 Finish HW \#4
\2 First bundle of papers posted - clone the assignment repo
\3 Answer the early questions as you go through the paper---write your
expectations before you finish reading the paper
\3 Start early, it will be a lot of work
\3 You may use my LaTeX form if you want, instructions in the handout
\2 Keep working on the papers
\end{outline}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment