hardware.scrbl 18.9 KB
Newer Older
Robert Ricci's avatar
Robert Ricci committed
1
#lang scribble/manual
2
@(require "defs.rkt")
Robert Ricci's avatar
Robert Ricci committed
3

4
@title[#:tag "hardware" #:version apt-version]{Hardware}
Robert Ricci's avatar
Robert Ricci committed
5

6
@(tb) can allocate experiments on any one of several federated clusters.
7 8

@clab-only{
9 10 11
    @(tb) has the ability to dispatch experiments to several clusters: three
    that belong to CloudLab itself, plus several more that belong to federated
    projects.
12

13 14
    Additional hardware expansions are planned, and descriptions of them
    can be found at @url[(@apturl "hardware.php")]
15
}
16

17
@clab-only{
Robert Ricci's avatar
Robert Ricci committed
18
    @; Reminder! This is duplicated below for Apt
19
    @section[#:tag "cloudlab-utah"]{CloudLab Utah}
20

Robert Ricci's avatar
Robert Ricci committed
21
    The CloudLab cluster at the University of Utah is being built in partnership
22 23 24 25
    with HP. It consists of 200 Intel Xeon E5 servers, 270 Xeon-D servers, and
    315 64-bit ARM servers and 270 Intel Xeon-D severs, for a total of 6,680
    cores. The cluster is housed in the University of Utah's Downtown Data
    Center in Salt Lake City.
26

27
    @(nodetype "m400" 315 "64-bit ARM"
28 29 30 31 32 33
        (list "CPU"  "Eight 64-bit ARMv8 (Atlas/A57) cores at 2.4 GHz (APM X-GENE)")
        (list "RAM"  "64GB ECC Memory (8x 8 GB DDR3-1600 SO-DIMMs)")
        (list "Disk" "120 GB of flash (SATA3 / M.2, Micron M500)")
        (list "NIC"  "Dual-port Mellanox ConnectX-3 10 GB NIC (PCIe v3.0, 8 lanes")
    )

34 35 36
    @(nodetype "m510" 270 "Intel Xeon-D"
        (list "CPU"  "Eight-core Intel Xeon D-1548 at 2.0 GHz")
        (list "RAM"  "64GB ECC Memory (4x 16 GB DDR4-2133 SO-DIMMs)")
37
        (list "Disk" "256 GB NVMe flash storage")
38 39 40
        (list "NIC"  "Dual-port Mellanox ConnectX-3 10 GB NIC (PCIe v3.0, 8 lanes")
    )

Robert Ricci's avatar
Robert Ricci committed
41
    There are 45 nodes in a chassis, and this cluster consists of thirteen
42 43 44 45 46 47 48 49 50 51 52
    chassis. Each chassis has two 45XGc switches; each node is connected to
    both switches, and each chassis switch has four 40Gbps uplinks, for a total
    of 320Gbps of uplink capacity from each chassis. One switch is used for
    control traffic, connecting to the Internet, etc. The other is used to
    build experiment topologies, and should be used for most experimental
    purposes.

    All chassis are interconnected through a large HP FlexFabric 12910 switch
    which has full bisection bandwidth internally.

    We have plans to enable some users to allocate entire chassis; when
Robert Ricci's avatar
Robert Ricci committed
53
    allocated in this mode, it will be possible to have complete administrator
54
    control over the switches in addition to the nodes.
55 56 57 58 59 60 61 62 63 64 65

    In phase two we added 50 Apollo R2200 chassis each with four HPE ProLiant
    XL170r server modules. Each server has 10 cores for a total of 2000 cores.

    @(nodetype "xl170" 200 "Intel Broadwell, 10 core, 1 disk"
        (list "CPU"  "Ten-core Intel E5-2640v4 at 2.4 GHz")
        (list "RAM"  "64GB ECC Memory (4x 16 GB DDR4-2400 DIMMs)")
        (list "Disk" "Intel DC S3520 480 GB 6G SATA SSD")
        (list "NIC"  "Two Dual-port Mellanox ConnectX-4 25 GB NIC (PCIe v3.0, 8 lanes")
    )

66 67
    Each server is connected via a 10Gbps control link (Dell switches) and a
    25Gbps experimental link to Mellanox 2410 switches in groups of 40 servers.
68 69 70 71 72 73 74
    Each of the five groups' experimental switches are connected to a Mellanox
    2700 spine switch at 5x100Gbps. That switch in turn interconnects with the
    rest of the Utah CloudLab cluster via 6x40Gbps uplinks to the HP FlexFabric
    12910 switch.

    A unique feature of the phase two nodes is the addition of eight ONIE
    bootable "user allocatable" switches that can run a variety of Open Network
75 76 77 78 79
    OSes: six Dell S4048-ONs and two Mellanox MSN2410-BB2Fs. These switches and
    all 200 nodes are connected to two NetScout 3903 layer-1 switches, allowing
    flexible combinations of nodes and switches in an experiment.

    @margin-note{The layer one network is still being deployed.}
80 81
}

82
@clab-only{
83
    @section[#:tag "cloudlab-wisconsin"]{CloudLab Wisconsin}
84

85 86
    The CloudLab cluster at the University of Wisconsin is built in
    partnership with Cisco, Seagate, and HP. The cluster, which is in
87 88 89
    Madison, Wisconsin, has 270 servers with a total of 5,000 cores connected
    in a CLOS topology with full bisection bandwidth. It has 1,070 TB of
    storage, including SSDs on every node. 
90

91 92
    @bold{Note: As of the time of writing, Wiscosin is working on upgrading nodes, and some may still be listed under their old ``type''}

Robert Ricci's avatar
Robert Ricci committed
93
    More technical details can be found at @url[(@apturl "hardware.php#wisconsin")]
94

95
    @(nodetype "c220g1" 90 "Haswell, 16 core, 3 disks"
96
        (list "CPU"  "Two Intel E5-2630 v3 8-core CPUs at 2.40 GHz (Haswell w/ EM64T)")
97 98
        (list "RAM"  "128GB ECC Memory (8x 16 GB DDR4 1866 MHz dual rank RDIMMs)")

99 100

        (list "Disk" "Two 1.2 TB 10K RPM 6G SAS SFF HDDs")
101 102
        (list "Disk" "One Intel DC S3500 480 GB 6G SATA SSDs")
        (list "NIC"  "Dual-port Intel X520-DA2 10Gb NIC (PCIe v3.0, 8 lanes)")
103 104
        (list "NIC"  "Onboard Intel i350 1Gb"))

105
    @(nodetype "c240g1" 10 "Haswell, 16 core, 14 disks"
106 107
        (list "CPU"  "Two Intel E5-2630 v3 8-core CPUs at 2.40 GHz (Haswell w/ EM64T)")

108 109 110
        (list "RAM"  "128GB ECC Memory (8x 16 GB DDR4 1866 MHz dual rank RDIMMs)")

        (list "Disk" "Two Intel DC S3500 480 GB 6G SATA SSDs")
111
        (list "Disk" "Twelve 3 TB 3.5\" HDDs donated by Seagate")
112
        (list "NIC"  "Dual-port Intel X520-DA2 10Gb NIC (PCIe v3.0, 8 lanes)")
113 114
        (list "NIC"  "Onboard Intel i350 1Gb"))

115
    @(nodetype "c220g2" 163 "Haswell, 20 core, 3 disks"
116
        (list "CPU"  "Two Intel E5-2660 v3 10-core CPUs at 2.60 GHz (Haswell EP)")
117
        (list "RAM"  "160GB ECC Memory (10x 16 GB DDR4 2133 MHz dual rank RDIMMs)")
118
        (list "Disk" "One Intel DC S3500 480 GB 6G SATA SSDs")
119 120 121 122
        (list "Disk" "Two 1.2 TB 10K RPM 6G SAS SFF HDDs")
        (list "NIC"  "Dual-port Intel X520 10Gb NIC (PCIe v3.0, 8 lanes")
        (list "NIC"  "Onboard Intel i350 1Gb"))

123
    @(nodetype "c240g2" 7 "Haswell, 20 core, 14 disks"
124
        (list "CPU"  "Two Intel E5-2660 v3 10-core CPUs at 2.60 GHz (Haswell EP)")
125
        (list "RAM"  "160GB ECC Memory (10x 16 GB DDR4 2133 MHz dual rank RDIMMs)")
126
        (list "Disk" "Two Intel DC S3500 480 GB 6G SATA SSDs")
127 128 129 130
        (list "Disk" "Twelve 3 TB 3.5\" HDDs donated by Seagate")
        (list "NIC"  "Dual-port Intel X520 10Gb NIC (PCIe v3.0, 8 lanes")
        (list "NIC"  "Onboard Intel i350 1Gb"))

131 132
    All nodes are connected to two networks:

133 134 135
    @margin-note{The experiment network at Wisconsin is transitioning to using
        HP switches in order to provide OpenFlow 1.3 support.}

136 137 138 139 140 141 142 143 144 145
    @itemlist[
        @item{A 1 Gbps Ethernet @bold{``control network''}---this network 
            is used for remote access, experiment management, etc., and is
            connected to the public Internet. When you log in to nodes in your
            experiment using @code{ssh}, this is the network you are using.
            @italic{You should not use this network as part of the experiments
            you run in @(tb).}
        }

        @item{A 10 Gbps Ethernet @bold{``experiment network''}--each node has
Robert Ricci's avatar
Robert Ricci committed
146
            @bold{two interfaces} on this network. Twelve leaf switches are
147 148 149 150 151 152 153 154
            Cisco Nexus C3172PQs, which have 48 10Gbps ports for the nodes and 
            six 40Gbps uplink ports. They are connected to six spine switches 
            (Cisco Nexus C3132Qs); each leaf has one 40Gbps link to each
            spine switch. Another C3132Q switch acts as a core; each spine
            switch has one 40Gbps link to it, and it has upstream
            links to Internet2.
        }
    ]
155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179

    Phase II added 260 new nodes, 36 with one or more GPUs:

    @(nodetype "c220g5" 224 "Intel Skylake, 20 core, 2 disks"
        (list "CPU"  "Two Intel Xeon Silver 4114 10-core CPUs at 2.20 GHz")
        (list "RAM"  "192GB ECC DDR4-2666 Memory")
        (list "Disk" "One 1 TB 7200 RPM 6G SAS HDs")
        (list "Disk" "One Intel DC S3500 480 GB 6G SATA SSD")
        (list "NIC"  "Dual-port Intel X520-DA2 10Gb NIC (PCIe v3.0, 8 lanes)")
        (list "NIC"  "Onboard Intel i350 1Gb"))

    @(nodetype "c240g5" 32 "Intel Skylake, 20 core, 2 disks, GPU"
        (list "CPU"  "Two Intel Xeon Silver 4114 10-core CPUs at 2.20 GHz")
        (list "RAM"  "192GB ECC DDR4-2666 Memory")
        (list "Disk" "One 1 TB 7200 RPM 6G SAS HDs")
        (list "Disk" "One Intel DC S3500 480 GB 6G SATA SSD")
        (list "GPU"  "One NVIDIA 12GB PCI P100 GPU")
        (list "NIC"  "Dual-port Intel X520-DA2 10Gb NIC (PCIe v3.0, 8 lanes)")
        (list "NIC"  "Onboard Intel i350 1Gb"))

    @(nodetype "c4130" 4 "Intel Broadwell, 16 core, 2 disks, 4 GPUs"
        (list "CPU"  "Two Intel Xeon E5-2667 8-core CPUs at 3.20 GHz")
        (list "RAM"  "128GB ECC Memory")
        (list "Disk" "Two 960 GB 6G SATA SSD")
        (list "GPU"  "Four NVIDIA 16GB Tesla V100 SMX2 GPUs"))
180 181
}

182
@clab-only{
183
    @section[#:tag "cloudlab-clemson"]{CloudLab Clemson}
184

185
    The CloudLab cluster at Clemson University has been built
186 187 188
    partnership with Dell. The cluster so far has 260
    servers with a total of 6,736 cores, 1,272TB of disk space, and
    73TB of RAM. All nodes have 10GB Ethernet and most have QDR
189
    Infiniband.  It is located in Clemson, South Carolina.
190 191 192

    More technical details can be found at @url[(@apturl "hardware.php#clemson")]

193
    @(nodetype "c8220" 96 "Ivy Bridge, 20 core"
194 195 196 197 198 199 200
        (list "CPU"  "Two Intel E5-2660 v2 10-core CPUs at 2.20 GHz (Ivy Bridge)")
        (list "RAM"  "256GB ECC Memory (16x 16 GB DDR4 1600MT/s dual rank RDIMMs")

        (list "Disk" "Two 1 TB 7.2K RPM 3G SATA HDDs")
        (list "NIC"  "Dual-port Intel 10Gbe NIC (PCIe v3.0, 8 lanes")
        (list "NIC"  "Qlogic QLE 7340 40 Gb/s Infiniband HCA (PCIe v3.0, 8 lanes)"))

201
    @(nodetype "c8220x" 4 "Ivy Bridge, 20 core, 20 disks"
202 203 204 205
        (list "CPU"  "Two Intel E5-2660 v2 10-core CPUs at 2.20 GHz (Ivy Bridge)")
        (list "RAM"  "256GB ECC Memory (16x 16 GB DDR4 1600MT/s dual rank RDIMMs")

        (list "Disk" "Eight 1 TB 7.2K RPM 3G SATA HDDs")
206
        (list "Disk" "Twelve 4 TB 7.2K RPM 3G SATA HDDs")
207 208 209
        (list "NIC"  "Dual-port Intel 10Gbe NIC (PCIe v3.0, 8 lanes")
        (list "NIC"  "Qlogic QLE 7340 40 Gb/s Infiniband HCA (PCIe v3.0, 8 lanes)"))

210
    @(nodetype "c6320" 84 "Haswell, 28 core"
211 212 213
        (list "CPU"  "Two Intel E5-2683 v3 14-core CPUs at 2.00 GHz (Haswell)")
        (list "RAM"  "256GB ECC Memory")
        (list "Disk" "Two 1 TB 7.2K RPM 3G SATA HDDs")
214
        (list "NIC"  "Dual-port Intel 10Gbe NIC (X520)")
215
        (list "NIC"  "Qlogic QLE 7340 40 Gb/s Infiniband HCA (PCIe v3.0, 8 lanes)"))
216

217
    @(nodetype "c4130" 2 "Haswell, 28 core, two GPUs"
218 219 220 221 222
        (list  "CPU"  "Two Intel E5-2680 v3 12-core processors at 2.50 GHz (Haswell)")
        (list "RAM"  "256GB ECC Memory")
        (list "Disk" "Two 1 TB 7.2K RPM 3G SATA HDDs")
        (list "GPU"  "Two Tesla K40m GPUs")
        (list "NIC"  "Dual-port Intel 1Gbe NIC (i350)")
223
        (list "NIC"  "Dual-port Intel 10Gbe NIC (X710)")
224
        (list "NIC"  "Qlogic QLE 7340 40 Gb/s Infiniband HCA (PCIe v3.0, 8 lanes)"))
225

Mike Hibler's avatar
Mike Hibler committed
226 227 228 229 230 231 232 233 234 235 236
    There are also two, storage intensive (270TB each!) nodes
    that should only be used if you need a huge amount of volatile
    storage. These nodes have only 10GB Ethernet.

    @(nodetype "dss7500" 2 "Haswell, 12 core, 270TB disk"
        (list "CPU"  "Two Intel E5-2620 v3 6-core CPUs at 2.40 GHz (Haswell)")
        (list "RAM"  "128GB ECC Memory")
        (list "Disk" "Two 120 GB 6Gbps SATA SSDs")
        (list "Disk" "45 6 TB 7.2K RPM 6Gbps SATA HDDs")
        (list "NIC"  "Dual-port Intel 10Gbe NIC (X520)"))

237
    There are three networks at the Clemson site:
238 239 240 241 242 243 244 245 246 247 248

    @itemlist[
        @item{A 1 Gbps Ethernet @bold{``control network''}---this network 
            is used for remote access, experiment management, etc., and is
            connected to the public Internet. When you log in to nodes in your
            experiment using @code{ssh}, this is the network you are using.
            @italic{You should not use this network as part of the experiments
            you run in @(tb).}
        }

        @item{A 10 Gbps Ethernet @bold{``experiment network''}--each node has
249 250 251 252
            @bold{one interface} on this network.  This network is implemented
            using three Force10 S6000 and three Force10 Z9100 switches.  Each
            S6000 switch is connected to a companion Z9100 switch via a 480Gbps
            link aggregate.
253 254
        }

255 256
        @item{A 40 Gbps QDR Infiniband @bold{``experiment network''}--each
            node has one connection to this
257 258 259
            network, which is implemented using a large Mellanox chassis switch
            with full bisection bandwidth.
        }
260
    ]
261 262 263 264 265 266

    Phase two added 18 Dell C6420 chassis each with four dual-socket Skylake-based
    servers. Each of the 72 servers has 32 cores for a total of 2304 cores.

    @(nodetype "c6420" 72 "Intel Skylake, 32 core, 2 disk"
        (list "CPU"  "Two Sixteen-core Intel Xeon Gold 6142 CPUs at 2.6 GHz")
267
        (list "RAM"  "384GB ECC DDR4-2666 Memory")
268 269 270 271 272 273 274 275
        (list "Disk" "Two Seagate 1TB 7200 RPM 6G SATA HDs")
        (list "NIC"  "Dual-port Intel X710 10Gbe NIC")
    )

    Each server is connected via a 1Gbps control link (Dell D3048 switches) and a
    10Gbps experimental link (Dell S5048 switches).

    These Phase II machines do not include Infiniband.
276 277
}

278 279
@section[#:tag "apt-cluster"]{Apt Cluster}

280 281 282
@apt-only{@margin-note{This is the cluster that is currently used by default
    for all experiments on Apt.}}

283 284 285
@clab-only{@margin-note{This cluster is not owned by CloudLab, but is federated
and available to CloudLab users.}}

286 287 288
The main Apt cluster is housed in the University of Utah's Downtown Data
Center in Salt Lake City, Utah. It contains two classes of nodes:

289
@(nodetype "r320" 128 "Sandy Bridge, 8 cores"
290 291 292 293 294 295 296
    (list "CPU"   "1x Xeon E5-2450 processor (8 cores, 2.1Ghz)")
    (list "RAM"   "16GB Memory (4 x 2GB RDIMMs, 1.6Ghz)")
    (list "Disks" "4 x 500GB 7.2K SATA Drives (RAID5)")
    (list "NIC"   "1GbE Dual port embedded NIC (Broadcom)")
    (list "NIC"   "1 x Mellanox MX354A Dual port FDR CX3 adapter w/1 x QSA adapter")
)

297
@(nodetype "c6220" 64 "Ivy Bridge, 16 cores"
298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341
    (list "CPU"   "2 x Xeon E5-2650v2 processors (8 cores each, 2.6Ghz)")
    (list "RAM"   "64GB Memory (8 x 8GB DDR-3 RDIMMs, 1.86Ghz)")
    (list "Disks" "2 x 1TB SATA 3.5” 7.2K rpm hard drives")
    (list "NIC"   "4 x 1GbE embedded Ethernet Ports (Broadcom)")
    (list "NIC"   "1 x Intel X520 PCIe Dual port 10Gb Ethernet NIC")
    (list "NIC"   "1 x Mellanox FDR CX3 Single port mezz card")
)

All nodes are connected to three networks with @bold{one interface each}:

@itemlist[
    @item{A 1 Gbps @italic{Ethernet} @bold{``control network''}---this network
        is used for remote access, experiment management, etc., and is
        connected to the public Internet. When you log in to nodes in your
        experiment using @code{ssh}, this is the network you are using.
        @italic{You should not use this network as part of the experiments you
        run in Apt.}
    }

    @item{A @bold{``flexible fabric''} that can run up to 56 Gbps and runs
        @italic{either FDR Infiniband or Ethernet}. This fabric uses NICs and
        switches with @hyperlink["http://www.mellanox.com/"]{Mellanox's}
        VPI technology. This means that we can, on demand, configure each
        port to be either FDR Inifiniband or 40 Gbps (or even non-standard
        56 Gbps) Ethernet. This fabric consists of seven edge switches
        (Mellanox SX6036G) with 28 connected nodes each. There are two core
        switches (also SX6036G), and each edge switch connects to both cores
        with a 3.5:1 blocking factor. This fabric is ideal if you need
        @bold{very low latency, Infiniband, or a few, high-bandwidth Ethernet
        links}.
     }

    @item{A 10 Gbps @italic{Ethernet} @bold{``commodity fabric''}.  One the
        @code{r320} nodes, a port on the Mellanox NIC (permanently set to
        Ethernet mode) is used to connect to this fabric; on the @code{c6220}
        nodes, a dedicated Intel 10 Gbps NIC is used. This fabric is built 
        from two Dell Z9000 switches, each of which has 96 nodes connected
        to it. It is idea for creating @bold{large LANs}: each of the two
        switches has full bisection bandwidth for its 96 ports, and there is a
        3.5:1 blocking factor between the two switches.
    }
        
]

Robert Ricci's avatar
Robert Ricci committed
342
@section[#:tag "ig-ddc"]{IG-DDC Cluster}
343

344 345 346
@clab-only{@margin-note{This cluster is not owned by CloudLab, but is federated
and available to CloudLab users.}}

347
This small cluster is an @hyperlink["http://www.instageni.net"]{InstaGENI
348
Rack} housed in the University of Utah's Downtown Data Center. It
349 350
has nodes of only a single type:

351
@nodetype["dl360" 33 "Sandy Bridge, 16 cores"
352 353 354 355 356
    (list "CPU"   "2x Xeon E5-2450 processors (8 cores each, 2.1Ghz)")
    (list "RAM"   "48GB Memory (6 x 8GB RDIMMs, 1.6Ghz)")
    (list "Disk"  "1 x 1TB 7.2K SATA Drive")
    (list "NIC"   "1GbE 4-port embedded NIC")
]
357 358 359 360

It has two network fabrics:

@itemlist[
361 362
    @item{A 1 Gbps @bold{``control network''}. 
        This is used for remote access,
Robert Ricci's avatar
Robert Ricci committed
363
        and should not be used for experiments.}
364 365 366 367
    @item{A 1 Gbps @bold{``experiment network''}. Each node has @italic{three}
        interfaces on this network, which is built from a single HP Procurve
        5406 switch. OpenFlow is available on this network.}
]
Robert Ricci's avatar
Robert Ricci committed
368 369 370

@; Total hack, I don't want this cluster listed first for Apt
@apt-only{
371
    @section[#:tag "cloudlab-utah"]{CloudLab Utah}
Robert Ricci's avatar
Robert Ricci committed
372 373 374 375 376 377 378 379 380 381 382 383 384 385

    @margin-note{This cluster is part of
    @link["https://www.cloudlab.us"]{CloudLab}, but is also available to Apt
    users.}

    The CloudLab cluster at the University of Utah is being built in partnership
    with HP. The first phase of this cluster consists of 315 64-bit ARM
    servers with 8 cores each, for a total of 2,520 cores. The servers are
    built on HP's Moonshot platform using X-GENE system-on-chip designs from
    Applied Micro. The cluster is hosted in the University of Utah's Downtown
    Data Center in Salt Lake City.

    More technical details can be found at @url[(@apturl "hardware.php#utah")]

386
    @(nodetype "m400" 315 "64-bit ARM"
Robert Ricci's avatar
Robert Ricci committed
387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407
        (list "CPU"  "Eight 64-bit ARMv8 (Atlas/A57) cores at 2.4 GHz (APM X-GENE)")
        (list "RAM"  "64GB ECC Memory (8x 8 GB DDR3-1600 SO-DIMMs)")
        (list "Disk" "120 GB of flash (SATA3 / M.2, Micron M500)")
        (list "NIC"  "Dual-port Mellanox ConnectX-3 10 GB NIC (PCIe v3.0, 8 lanes")
    )

    There are 45 nodes in a chassis, and this cluster consists of seven
    chassis. Each chassis has two 45XGc switches; each node is connected to
    both switches, and each chassis switch has four 40Gbps uplinks, for a total
    of 320Gbps of uplink capacity from each chassis. One switch is used for
    control traffic, connecting to the Internet, etc. The other is used to
    build experiment topologies, and should be used for most experimental
    purposes.

    All chassis are interconnected through a large HP FlexFabric 12910 switch
    which has full bisection bandwidth internally.

    We have plans to enable some users to allocate entire chassis; when
    allocated in this mode, it will be possible to have complete administrator
    control over the switches in addition to the nodes.
}