#lang scribble/manual @(require "defs.rkt") @title[#:tag "hardware" #:version apt-version]{Hardware} @(tb) can allocate experiments on any one of several federated clusters. @clab-only{ @(tb) has the ability to dispatch experiments to several clusters: three that belong to CloudLab itself, plus several more that belong to federated projects. Additional hardware expansions are planned, and descriptions of them can be found at @url[(@apturl "hardware.php")] } @clab-only{ @; Reminder! This is duplicated below for Apt @section[#:tag "cloudlab-utah"]{CloudLab Utah} The CloudLab cluster at the University of Utah is being built in partnership with HP. The first phase of this cluster consists of 315 64-bit ARM servers with 8 cores each, for a total of 2,520 cores. The servers are built on HP's Moonshot platform using X-GENE system-on-chip designs from Applied Micro. The cluster is housed in the University of Utah's Downtown Data Center in Salt Lake City. More technical details can be found at @url[(@apturl "hardware.php#utah")] @(nodetype "m400" 315 (list "CPU" "Eight 64-bit ARMv8 (Atlas/A57) cores at 2.4 GHz (APM X-GENE)") (list "RAM" "64GB ECC Memory (8x 8 GB DDR3-1600 SO-DIMMs)") (list "Disk" "120 GB of flash (SATA3 / M.2, Micron M500)") (list "NIC" "Dual-port Mellanox ConnectX-3 10 GB NIC (PCIe v3.0, 8 lanes") ) There are 45 nodes in a chassis, and this cluster consists of seven chassis. Each chassis has two 45XGc switches; each node is connected to both switches, and each chassis switch has four 40Gbps uplinks, for a total of 320Gbps of uplink capacity from each chassis. One switch is used for control traffic, connecting to the Internet, etc. The other is used to build experiment topologies, and should be used for most experimental purposes. All chassis are interconnected through a large HP FlexFabric 12910 switch which has full bisection bandwidth internally. We have plans to enable some users to allocate entire chassis; when allocated in this mode, it will be possible to have complete administrator control over the switches in addition to the nodes. } @clab-only{ @section[#:tag "cloudlab-wisconsin"]{CloudLab Wisconsin} The CloudLab cluster at the University of Wisconsin is built in partnership with Cisco, Seagate, and HP. The cluster, which is in Madison, Wisconsin, has 100 servers with a total of 1,600 cores connected in a CLOS topology with full bisection bandwidth. It has 525 TB of storage, including SSDs on every node. More technical details can be found at @url[(@apturl "hardware.php#wisconsin")] @(nodetype "C220M4" 90 (list "CPU" "Two Intel E5-2630 v3 8-core CPUs at 2.40 GHz (Haswell w/ EM64T)") (list "RAM" "128GB ECC Memory (8x 16 GB DDR4 2133 MHz PC4-17000 dual rank RDIMMs") (list "Disk" "Two 1.2 TB 10K RPM 6G SAS SFF HDDs") (list "Disk" "One 480 GB 6G SAS SSD") (list "NIC" "Dual-port Cisco VIC1227 10Gb NIC (PCIe v3.0, 8 lanes") (list "NIC" "Onboard Intel i350 1Gb")) @(nodetype "C240M4" 10 (list "CPU" "Two Intel E5-2630 v3 8-core CPUs at 2.40 GHz (Haswell w/ EM64T)") (list "RAM" "128GB ECC Memory (8x 16 GB DDR4 2133 MHz PC4-17000 dual rank RDIMMs") (list "Disk" "One 1 TB 7.2K RPM SAS 3.5\" HDD") (list "Disk" "One 480 GB 6G SAS SSD") (list "Disk" "Twelve 3 TB 3.5\" HDDs donated by Seagate") (list "NIC" "Dual-port Cisco VIC1227 10Gb NIC (PCIe v3.0, 8 lanes") (list "NIC" "Onboard Intel i350 1Gb")) @(nodetype "C240M4" 10 (list "CPU" "Two Intel E5-2630 v3 8-core CPUs at 2.40 GHz (Haswell w/ EM64T)") (list "RAM" "128GB ECC Memory (8x 16 GB DDR4 2133 MHz PC4-17000 dual rank RDIMMs") (list "Disk" "One 1 TB 7.2K RPM SAS 3.5\" HDD") (list "Disk" "One 480 GB 6G SAS SSD") (list "Disk" "Twelve 3 TB 3.5\" HDDs donated by Seagate") (list "NIC" "Dual-port Cisco VIC1227 10Gb NIC (PCIe v3.0, 8 lanes") (list "NIC" "Onboard Intel i350 1Gb")) @(nodetype "c220g2" 163 (list "CPU" "Two Intel E5-2660 v3 10-core CPUs at 2.60 GHz (Haswell EP)") (list "RAM" "160GB ECC Memory (10x 16 GB DDR4 2133 MHz PC4-17000 dual rank RDIMMs - 5 memory channels)") (list "Disk" "Two 1.2 TB 10K RPM 6G SAS SFF HDDs") (list "Disk" "One 480 GB 6G SAS SSD") (list "NIC" "Dual-port Intel X520 10Gb NIC (PCIe v3.0, 8 lanes") (list "NIC" "Onboard Intel i350 1Gb")) @(nodetype "c240g2" 7 (list "CPU" "Two Intel E5-2660 v3 10-core CPUs at 2.60 GHz (Haswell EP)") (list "RAM" "160GB ECC Memory (10x 16 GB DDR4 2133 MHz PC4-17000 dual rank RDIMMs - 5 memory channels)") (list "Disk" "Two 1.2 TB 10K RPM 6G SAS SFF HDDs") (list "Disk" "One 480 GB 6G SAS SSD") (list "Disk" "Twelve 3 TB 3.5\" HDDs donated by Seagate") (list "NIC" "Dual-port Intel X520 10Gb NIC (PCIe v3.0, 8 lanes") (list "NIC" "Onboard Intel i350 1Gb")) All nodes are connected to two networks: @itemlist[ @item{A 1 Gbps Ethernet @bold{``control network''}---this network is used for remote access, experiment management, etc., and is connected to the public Internet. When you log in to nodes in your experiment using @code{ssh}, this is the network you are using. @italic{You should not use this network as part of the experiments you run in @(tb).} } @item{A 10 Gbps Ethernet @bold{``experiment network''}--each node has @bold{two interfaces} on this network. Twelve leaf switches are Cisco Nexus C3172PQs, which have 48 10Gbps ports for the nodes and six 40Gbps uplink ports. They are connected to six spine switches (Cisco Nexus C3132Qs); each leaf has one 40Gbps link to each spine switch. Another C3132Q switch acts as a core; each spine switch has one 40Gbps link to it, and it has upstream links to Internet2. } ] } @clab-only{ @section[#:tag "cloudlab-clemson"]{CloudLab Clemson} The CloudLab cluster at Clemson University has been built partnership with Dell. The cluster so far has 186 servers with a total of 4,400 cores, 596TB of disk space, and 48TB of RAM. All nodes have 10GB Ethernet, and about half have QDR Infiniband as well. It is located in Clemson, South Carolina. More technical details can be found at @url[(@apturl "hardware.php#clemson")] @(nodetype "c8220" 96 (list "CPU" "Two Intel E5-2660 v2 10-core CPUs at 2.20 GHz (Ivy Bridge)") (list "RAM" "256GB ECC Memory (16x 16 GB DDR4 1600MT/s dual rank RDIMMs") (list "Disk" "Two 1 TB 7.2K RPM 3G SATA HDDs") (list "NIC" "Dual-port Intel 10Gbe NIC (PCIe v3.0, 8 lanes") (list "NIC" "Qlogic QLE 7340 40 Gb/s Infiniband HCA (PCIe v3.0, 8 lanes)")) @(nodetype "c8220x" 4 (list "CPU" "Two Intel E5-2660 v2 10-core CPUs at 2.20 GHz (Ivy Bridge)") (list "RAM" "256GB ECC Memory (16x 16 GB DDR4 1600MT/s dual rank RDIMMs") (list "Disk" "Eight 1 TB 7.2K RPM 3G SATA HDDs") (list "Disk" "Twelve 4 TB 7.2K RPM 3G SATA HDDs") (list "NIC" "Dual-port Intel 10Gbe NIC (PCIe v3.0, 8 lanes") (list "NIC" "Qlogic QLE 7340 40 Gb/s Infiniband HCA (PCIe v3.0, 8 lanes)")) @(nodetype "c6320" 84 (list "CPU" "Two Intel E5-2683 v3 14-core CPUs at 2.00 GHz (Haswell)") (list "RAM" "256GB ECC Memory") (list "Disk" "Two 1 TB 7.2K RPM 3G SATA HDDs") (list "NIC" "Dual-port Intel 10Gbe NIC (X520)")) @(nodetype "c4130" 2 (list "CPU" "Two Intel E5-2680 v3 12-core processors at 2.50 GHz (Haswell)") (list "RAM" "256GB ECC Memory") (list "Disk" "Two 1 TB 7.2K RPM 3G SATA HDDs") (list "GPU" "Two Tesla K40m GPUs") (list "NIC" "Dual-port Intel 1Gbe NIC (i350)") (list "NIC" "Dual-port Intel 10Gbe NIC (X710)")) There are three networks at the Clemson site: @itemlist[ @item{A 1 Gbps Ethernet @bold{``control network''}---this network is used for remote access, experiment management, etc., and is connected to the public Internet. When you log in to nodes in your experiment using @code{ssh}, this is the network you are using. @italic{You should not use this network as part of the experiments you run in @(tb).} } @item{A 10 Gbps Ethernet @bold{``experiment network''}--each node has @bold{one interface} on this network. Three Dell Force10 S6000 switches are used to implent it: two are used to connect nodes directly, while the third is used as an aggregator connecting the two leaf switches. Each S6000 has 32 40Gbps ports. On the leaf switches, these are broken out as 96 10Gbps ports, plus 8 40 Gbps uplink ports. This gives a 3:1 blocking factor between the two leaf switches. } @item{A 40 Gbps QDR Infiniband @bold{``experiment network''}--for nodes with an Infinband NIC, each has one connection to this network, which is implemented using a large Mellanox chassis switch with full bisection bandwidth. } ] } @section[#:tag "apt-cluster"]{Apt Cluster} @apt-only{@margin-note{This is the cluster that is currently used by default for all experiments on Apt.}} @clab-only{@margin-note{This cluster is not owned by CloudLab, but is federated and available to CloudLab users.}} The main Apt cluster is housed in the University of Utah's Downtown Data Center in Salt Lake City, Utah. It contains two classes of nodes: @(nodetype "r320" 128 (list "CPU" "1x Xeon E5-2450 processor (8 cores, 2.1Ghz)") (list "RAM" "16GB Memory (4 x 2GB RDIMMs, 1.6Ghz)") (list "Disks" "4 x 500GB 7.2K SATA Drives (RAID5)") (list "NIC" "1GbE Dual port embedded NIC (Broadcom)") (list "NIC" "1 x Mellanox MX354A Dual port FDR CX3 adapter w/1 x QSA adapter") ) @(nodetype "c6220" 64 (list "CPU" "2 x Xeon E5-2650v2 processors (8 cores each, 2.6Ghz)") (list "RAM" "64GB Memory (8 x 8GB DDR-3 RDIMMs, 1.86Ghz)") (list "Disks" "2 x 1TB SATA 3.5” 7.2K rpm hard drives") (list "NIC" "4 x 1GbE embedded Ethernet Ports (Broadcom)") (list "NIC" "1 x Intel X520 PCIe Dual port 10Gb Ethernet NIC") (list "NIC" "1 x Mellanox FDR CX3 Single port mezz card") ) All nodes are connected to three networks with @bold{one interface each}: @itemlist[ @item{A 1 Gbps @italic{Ethernet} @bold{``control network''}---this network is used for remote access, experiment management, etc., and is connected to the public Internet. When you log in to nodes in your experiment using @code{ssh}, this is the network you are using. @italic{You should not use this network as part of the experiments you run in Apt.} } @item{A @bold{``flexible fabric''} that can run up to 56 Gbps and runs @italic{either FDR Infiniband or Ethernet}. This fabric uses NICs and switches with @hyperlink["http://www.mellanox.com/"]{Mellanox's} VPI technology. This means that we can, on demand, configure each port to be either FDR Inifiniband or 40 Gbps (or even non-standard 56 Gbps) Ethernet. This fabric consists of seven edge switches (Mellanox SX6036G) with 28 connected nodes each. There are two core switches (also SX6036G), and each edge switch connects to both cores with a 3.5:1 blocking factor. This fabric is ideal if you need @bold{very low latency, Infiniband, or a few, high-bandwidth Ethernet links}. } @item{A 10 Gbps @italic{Ethernet} @bold{``commodity fabric''}. One the @code{r320} nodes, a port on the Mellanox NIC (permanently set to Ethernet mode) is used to connect to this fabric; on the @code{c6220} nodes, a dedicated Intel 10 Gbps NIC is used. This fabric is built from two Dell Z9000 switches, each of which has 96 nodes connected to it. It is idea for creating @bold{large LANs}: each of the two switches has full bisection bandwidth for its 96 ports, and there is a 3.5:1 blocking factor between the two switches. } ] @section[#:tag "ig-ddc"]{IG-DDC Cluster} @clab-only{@margin-note{This cluster is not owned by CloudLab, but is federated and available to CloudLab users.}} This small cluster is an @hyperlink["http://www.instageni.net"]{InstaGENI Rack} housed in the University of Utah's Downtown Data Center. It has nodes of only a single type: @nodetype["dl360" 33 (list "CPU" "2x Xeon E5-2450 processors (8 cores each, 2.1Ghz)") (list "RAM" "48GB Memory (6 x 8GB RDIMMs, 1.6Ghz)") (list "Disk" "1 x 1TB 7.2K SATA Drive") (list "NIC" "1GbE 4-port embedded NIC") ] It has two network fabrics: @itemlist[ @item{A 1 Gbps @bold{``control network''}. This is used for remote access, and should not be used for experiments.} @item{A 1 Gbps @bold{``experiment network''}. Each node has @italic{three} interfaces on this network, which is built from a single HP Procurve 5406 switch. OpenFlow is available on this network.} ] @; Total hack, I don't want this cluster listed first for Apt @apt-only{ @section[#:tag "cloudlab-utah"]{CloudLab Utah} @margin-note{This cluster is part of @link["https://www.cloudlab.us"]{CloudLab}, but is also available to Apt users.} The CloudLab cluster at the University of Utah is being built in partnership with HP. The first phase of this cluster consists of 315 64-bit ARM servers with 8 cores each, for a total of 2,520 cores. The servers are built on HP's Moonshot platform using X-GENE system-on-chip designs from Applied Micro. The cluster is hosted in the University of Utah's Downtown Data Center in Salt Lake City. More technical details can be found at @url[(@apturl "hardware.php#utah")] @(nodetype "m400" 315 (list "CPU" "Eight 64-bit ARMv8 (Atlas/A57) cores at 2.4 GHz (APM X-GENE)") (list "RAM" "64GB ECC Memory (8x 8 GB DDR3-1600 SO-DIMMs)") (list "Disk" "120 GB of flash (SATA3 / M.2, Micron M500)") (list "NIC" "Dual-port Mellanox ConnectX-3 10 GB NIC (PCIe v3.0, 8 lanes") ) There are 45 nodes in a chassis, and this cluster consists of seven chassis. Each chassis has two 45XGc switches; each node is connected to both switches, and each chassis switch has four 40Gbps uplinks, for a total of 320Gbps of uplink capacity from each chassis. One switch is used for control traffic, connecting to the Internet, etc. The other is used to build experiment topologies, and should be used for most experimental purposes. All chassis are interconnected through a large HP FlexFabric 12910 switch which has full bisection bandwidth internally. We have plans to enable some users to allocate entire chassis; when allocated in this mode, it will be possible to have complete administrator control over the switches in addition to the nodes. }