All new accounts created on Gitlab now require administrator approval. If you invite any collaborators, please let Flux staff know so they can approve the accounts.

lecturenotes.tex 3.3 KB
 Robert Ricci committed Mar 03, 2015 1 2 \documentclass{article}[12pt]  Robert Ricci committed Apr 16, 2015 3 4 \input{../../texstuff/fonts.sty} \input{../../texstuff/notepaper.sty}  Robert Ricci committed Mar 03, 2015 5 6 7 8 \usepackage{outlines} \title{CS6963 Lecture \#20} \author{Robert Ricci}  Robert Ricci committed Apr 16, 2015 9 \date{April 16, 2015}  Robert Ricci committed Mar 03, 2015 10 11 12 13 14 15 16  \begin{document} \maketitle \begin{outline}  Robert Ricci committed Apr 16, 2015 17 18 \1 Project status updates  Robert Ricci committed Mar 03, 2015 19 20 21 22 \1 Review of topics covered \2 Overall point of systems evaluation \3 Providing data necessary to make a decision \3 Presenting it on a convincing manner  Robert Ricci committed Apr 16, 2015 23 24  \3 Performance evaluation is not the only relevant data, but it's important  Robert Ricci committed Mar 03, 2015 25 26 27  \2 Understand what you are measuring \3 Understand what make convincing evidence \3 Understand what you need to measure and how to measure it  Robert Ricci committed Apr 16, 2015 28 29  \3 Do some baseline tests of your workload generator and monitors to make sure they exhibit expected results  Robert Ricci committed Mar 03, 2015 30 31 32  \2 Understand the system under test \3 Where are the boundaries? \3 What is inside those boundaries that you are actually measuring?  Robert Ricci committed Apr 16, 2015 33 34  \3 How do the SUT boundaries relate to a deployment environment? \3 How do the SUT boundaries relate to the claims of the paper?  Robert Ricci committed Apr 16, 2015 35  \2 Evaluation should be a part of the research and development process  Robert Ricci committed Mar 03, 2015 36 37 38 39 40 41  \3 Convince yourself with data, not just bias \3 Often need preliminary evaluations \3 Understand the scope and limitations of your work \3 The more evaluation you do along the way, the less biased you are likely to be \2 Recognize the strengths and weaknesses in evaluations that you read  Robert Ricci committed Apr 16, 2015 42  \3 There is a huge difference between active and passive reading  Robert Ricci committed Mar 03, 2015 43  \3 Think actively about what you need to see to be convinced  Robert Ricci committed Apr 16, 2015 44  \3 Look for biases or basic mistakes  Robert Ricci committed Mar 03, 2015 45 46 47  \2 Common mistakes in systems evaluation \3 No goals or biased goals \3 Ignoring significant factors  Robert Ricci committed Apr 16, 2015 48 49  \3 Analysis without understanding the problem---a reason to have a concrete problem statement  Robert Ricci committed Mar 03, 2015 50 51 52 53 54 55 56  \3 No sensitivity analysis \3 Ignoring variability \2 Use the statistical tools available to you \3 For selecting the number of runs \3 For understanding the confidence in your results \3 For showing difference or sameness in results \3 The question is have you found something real?''  Robert Ricci committed Apr 16, 2015 57 58 59 60 61 62 63 64 65  \3 Don't just eyeball graphs \4 Compute CIs \4 Do linear regressions \4 Compares means, percentiles, etc. \2 Work done to make work repeatable helps everyone \3 Not just for others, but for yourself too \3 Lets you re-run things when something changes \3 Makes it easier for others to build on your work \3 These things bitrot incredibly fast  Robert Ricci committed Mar 03, 2015 66 67  \3 The closer you can get to experiment as function call'', the happier you will be  Robert Ricci committed Apr 16, 2015 68 69  \3 Assuming your environment is fragile forces you to build more repeatable research  Robert Ricci committed Mar 03, 2015 70  \3 Keep track of everything  Robert Ricci committed Apr 16, 2015 71 72  \2 You are running experiments all the time, the only question is whether you are learning anything from them  Robert Ricci committed Mar 03, 2015 73   Robert Ricci committed Apr 16, 2015 74 75 76 77 78 79 \1 For next time \2 Weekly 3 due Friday midnight \2 Papers 4 due next Tuesday \2 Talks next Thursday (Anil Kumar, Jithu, George, Jonathon) \3 Schedule a time to meet with me \3 Any questions about what's expected?  Robert Ricci committed Mar 03, 2015 80 81 82 83  \end{outline} \end{document}