From 2a25ef9ae205e2c9654e24c1b9a84c06fe5cacf4 Mon Sep 17 00:00:00 2001 From: Robert Ricci Date: Wed, 12 Feb 2014 16:00:13 -0700 Subject: [PATCH] Lecture notes for Lecture 11 --- lectures/lecture11/Makefile | 3 + lectures/lecture11/lecturenotes.tex | 123 ++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+) create mode 100644 lectures/lecture11/Makefile create mode 100644 lectures/lecture11/lecturenotes.tex diff --git a/lectures/lecture11/Makefile b/lectures/lecture11/Makefile new file mode 100644 index 0000000..14373ab --- /dev/null +++ b/lectures/lecture11/Makefile @@ -0,0 +1,3 @@ +DOCUMENTS= lecturenotes + +include ../../Makerules diff --git a/lectures/lecture11/lecturenotes.tex b/lectures/lecture11/lecturenotes.tex new file mode 100644 index 0000000..9353f46 --- /dev/null +++ b/lectures/lecture11/lecturenotes.tex @@ -0,0 +1,123 @@ +\documentclass{article}[12pt] + +\usepackage[no-math]{fontspec} +\usepackage{sectsty} +\usepackage[margin=1.25in]{geometry} +\usepackage{outlines} + +\setmainfont[Numbers=OldStyle,Ligatures=TeX]{Equity Text A} +\setmonofont{Inconsolata} +\newfontfamily\titlefont[Numbers=OldStyle,Ligatures=TeX]{Equity Caps A} +\allsectionsfont{\titlefont} + +\title{CS6963 Lecture \#11} +\author{Robert Ricci} +\date{February 13, 2014} + +\begin{document} + +\maketitle + +\begin{outline} + +\1 From last time + +\1 Big idea for the day: all statements we make from evals are probabilistic + +\1 Quick refresher - sample vs. population + \2 Parameters of prob distribution vs. statistics of the sample + +\1 We measure a sample mean, but it is really just an estimate of population + mean + \2 We can get a confidence interval that the true mean is within some range: + significance level / confidence level + \2 Book explanation of way to get confidence level + \3 Get multiple samples (multiple trials per sample), compute stats on the means, treat that as a sample set and take confidence intervals + \2 Again, iid comes up, and this is why you need to be careful in experiment design + \3 \textit{When might you not meet identically distributed criteria?} + \2 Standard error --- not to be confused with standard deviation or STDERR + +\1 Confidence interval for sample mean + \2 Lower: $\overline{x} - \frac{z_{1-\alpha/2}s}{\sqrt{n}}$ + \2 Upper: $\overline{x} + \frac{z_{1-\alpha/2}s}{\sqrt{n}}$ + \2 $\overline{x}$ is sample mean + \2 $s$ is sample stddev + \2 $z_{1-\alpha/2}$ is $(1 - \alpha/2)$ quantile of unit normal dist ($\mu = 0$ and $\sigma = 1$) - note, you are picking $\alpha$ + \2 $n$ is the sample size + \2 \textit{So, what does this tell us?} + \3 We are x\% certain that the population mean is between $x$ and $y$ + \2 \textit{What do we need to apply this result?} + \3 iid sample + \3 Large samples (30 or greater) + \3 Or sample itself is normally distribted + \2 \textit{When is it not worth computing this?} + \3 When the means are extremely far apart + \2 \textit{When is it important?} + \3 Close enough that it's possible that means lie within each others' + confidence intervals + \2 Testing for mean of particular value - does it lie within the CI? + \2 \textit{When might you want your mean to be the same as another mean?} + \3 Showing insignificant overhead + +\1 Showing significance: Paired samples (eg. same benchmarks) + \2 Take samples for two systems under the same workload + \2 Compute statistics of the difference + \2 Compute CI of mean of the difference + \2 If CI contains zero, not statistically different: The hypothesis ``the two + systems are the same'' is supported by the data + +\1 Showing significance: t-test (eg. truly random samples) + \2 Best to leave the implementation of this up to someone else + \2 Degrees of freedom: number of independent sources of data that go into + the model: number of samples minus steps that go into the estimation + \2 eg. R includes this as a module + \2 Fun fact: t-test invented as a way of measuring the quality of beer + (Guinness Stout) + +\1 Showing significance: visual check + \2 Draw both confidence intervals and means + \2 If CIs don't overlap, one is clearly better + \2 If CIs do overlap, both means fall inside CI of the other: effectively + the same + \2 If the mean of one is in the CI of the other, but this is not true for + both, t-test required + +\1 Picking CIs + \2 As discussed before, degree of confidence has to do with the gain/loss of + being outside the range + \2 Reiterate plane example, you don't want to fly on a plane built with + only 99\% confidence intervals + +\1 The value of hypothesis testing + \2 State your goal, test whether or not you achieved it + \2 ``On Bullshit'' + \2 eg. a good thesis statement is a testable hypothesis + +\1 Proportions + \2 Similar, but for categorical outcomes (range not domain) + \2 What proportion of the population consists of category X? + \2 Sample proportion: $\frac{n_1}{n}$ + \2 CI for sample proportion $p \mp z_{1-\alpha/2} \sqrt{\frac{p(1-p)}{n}}$ + \2 $np > 10$ required + \2 \textit{Why is this symmetric?} + +\1 Picking a sample size + \2 Sample size being too big is rarely a problem + \2 It's just that it can take too much time to get that many samples + \2 All dependent on the variance, which is intuitive + \2 $n = \left(\frac{100zs}{r\overline{z}}\right)^2$ + \2 $n = z^2\frac{p(1-p)}{r^2}$ + \2 For comparing two, upper edge of lower must be below lower edge of upper + \2 $x \mp z \frac{s}{\sqrt{n}} $ + \2 Leave $n$ unbound, set the plus and minus versions with the appropriate + comparison operator, solve for $n$ + + +\1 For next time + \2 Bring your laptop + \2 Sign up for GENI account + \2 Read GENI paper posted on Canvas + +\end{outline} + +\end{document} -- GitLab