repeatable-research.scrbl 2.15 KB
Newer Older
1 2 3
#lang scribble/manual
@(require "defs.rkt")

4
@title[#:tag "repeatable-research" #:version apt-version]{Apt and Repeatable Research}
5

6
One of @(tb)'s key goals is to enable @italic{repeatable research}---we aim to
7 8 9
make it easier for researchers to get the same software and hardware
environment so that they can repeat or build upon each others' work.

10
@seclink["virtual-machines"]{Virtual machines} in @(tb) do @bold{not} provide
Robert Ricci's avatar
Robert Ricci committed
11 12
strong resource guarantees or performance isolation from other concurrent
users. Therefore, they are suitable primarily under the following conditions:
13 14

@itemlist[
Robert Ricci's avatar
Robert Ricci committed
15
    @item{During initial exploration, development, gathering of preliminary
16
        performance results, etc.}
Robert Ricci's avatar
Robert Ricci committed
17 18
    @item{When it is the output of the software, rather than the software's
    performance, that is of interest.}
19 20 21 22 23 24
]

We therefore recommend that researchers who want to provide a repeatable 
environment do the following:

@itemlist[#:style 'ordered
Robert Ricci's avatar
Robert Ricci committed
25 26 27 28 29 30
    @item{Conduct initial development and gather initial numbers using
    @seclink["virtual-machines"]{virtual machines}. Because much of the time in
    this phase is often spent on debugging, development, etc., using a full
    physical machine is often an inefficient use of resources.}
    @item{Switch to @seclink["physical-machines"]{physical machines} to collect
    numbers of publication. This ensures that published numbers are not
31
    affected by interference from other users of @(tb).}
32 33 34 35 36 37 38
]

Similarly, for those who are repeating or building on the work of others, we
recommend:

@itemlist[#:style 'ordered
    @item{During the initial, exploratory phase of figuring out how to run
Robert Ricci's avatar
Robert Ricci committed
39 40 41 42 43
        the code, examining its configuration and parameters, etc., use
        @seclink["virtual-machines"]{virtual machines}.}
    @item{Switch to @seclink["physical-machines"]{physical machines} when it's
    time to do real experiments, compare performance with published
    results, etc.}
44 45
]

Robert Ricci's avatar
Robert Ricci committed
46 47
@future-work["planned-virt-switching"]

48
Because the @seclink["disk-images"]{disk images} that @(tb) provides boot on both
Robert Ricci's avatar
Robert Ricci committed
49 50
virtual and physical machines, in most cases, switching means simply modifying
one's profile to request the other type of resource.