repeatable-research.scrbl 3.38 KB
Newer Older
1 2 3
#lang scribble/manual
@(require "defs.rkt")

@title[#:tag "repeatable-research" #:version apt-version]{@(tb) and Repeatable Research}

One of @(tb)'s key goals is to enable @italic{repeatable research}---we aim to
7 8 9
make it easier for researchers to get the same software and hardware
environment so that they can repeat or build upon each others' work.

10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72

    @seclink["virtual-machines"]{Virtual machines} in @(tb) do @bold{not}
    provide strong resource guarantees or performance isolation from other
    concurrent users. Therefore, they are suitable primarily under the
    following conditions:

        @item{During initial exploration, development, gathering of preliminary
            performance results, etc.}
        @item{When it is the output of the software, rather than the software's
        performance, that is of interest.}

    We therefore recommend that researchers who want to provide a repeatable
    environment do the following:

    @itemlist[#:style 'ordered
        @item{Conduct initial development and gather initial numbers using
            @seclink["virtual-machines"]{virtual machines}. Because much of the
            time in this phase is often spent on debugging, development, etc.,
            using a full physical machine is often an inefficient use of
        @item{Switch to @seclink["physical-machines"]{physical machines} to
            collect numbers of publication. This ensures that published numbers
            are not affected by interference from other users of @(tb).}

    Similarly, for those who are repeating or building on the work of others,
    we recommend:

    @itemlist[#:style 'ordered
        @item{During the initial, exploratory phase of figuring out how to run
            the code, examining its configuration and parameters, etc., use
            @seclink["virtual-machines"]{virtual machines}.}
        @item{Switch to @seclink["physical-machines"]{physical machines} when
            it's time to do real experiments, compare performance with
            published results, etc.}


    Because the @seclink["disk-images"]{disk images} that @(tb) provides boot
    on both virtual and physical machines, in most cases, switching means
    simply modifying one's profile to request the other type of resource.


    @(tb) is designed as a @italic{scientific instrument}. It gives full
    visibility into every aspect of the facility, and it's designed to minimize
    the impact that simultaneous slices have on each other. This means that
    researchers using @(tb) can fully understand why their systems behave the
    way they do, and can have confidence that the results that they gather are
    not artifacts of competition for shared hardware resources. @(tb)
    @seclink["profiles"]{profiles} can also be published, giving other
    researchers the exact same environment—hardware and software—on which to
    repeat experiments and compare results.

    @(tb) gives exclusive access to compute resources to one experiment at
    a time. (That experiment may involve re-exporting those resources to
    other users, for example, by running cloud services.) Storage resources
    attached to them (eg. local disk) are also used by a single experiment
73 74
    at a time, and it is possible to run experiments that have 
    exclusive access to switches.
75 76