|
|
|
|
|
The Utah Emulab cluster has 160 **d430** PC nodes (**pc701-pc860**) consisting of:
|
|
|
|
|
|
* [Dell Poweredge R430](http://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell-PowerEdge-R430-Spec-Sheet.pdf) 1U servers
|
|
|
* Two 2.4 GHz 64-bit 8-Core [E5-2630 "Haswell"
|
|
|
processors](http://ark.intel.com/products/83356/), 8.0 GT/s bus speed,
|
|
|
20 MB cache, and VT (VT-x, EPT, and VT-d) support
|
|
|
* 64 GB 2133 MT/s DDR4 RAM (8 x 8GB modules)
|
|
|
* 4 Broadcom NetXtreme BCM5720 GbE NICs builtin to motherboard (only one in use)
|
|
|
* 1 dual-port or 1 quad-port Intel I350 1GbE PCI-Express NICs
|
|
|
* 1 dual-port or 1 quad-port Intel X710 10GbE PCI-Express NICs
|
|
|
* 1 200GB Intel SATA SSD, 2 x 1TB 7200 rpm SATA disks
|
|
|
|
|
|
The nodes are arranged in four racks, 40 nodes per rack. Each rack has its
|
|
|
own management and control network switches along with three experimental
|
|
|
switches. The only cross-rack wiring is for the management and control
|
|
|
network interconnects and the 10Gb interfaces (both on the nodes and for
|
|
|
the experimental switches).
|
|
|
|
|
|
### Switch Configuration
|
|
|
|
|
|
Each node has six experimental interfaces. 64 of the 160 machines (16 per
|
|
|
rack) have four 10Gb interfaces and two 1Gb interfaces. The other 96
|
|
|
machines (24 per rack) have two 10Gb interfaces and four 1Gb interfaces.
|
|
|
|
|
|
All of the 10Gb interfaces connect to a single
|
|
|
[Dell Z9500](http://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell_Networking_Z9500_SpecSheet.pdf)
|
|
|
132-port 40Gb switch using 10Gb to 40Gb breakout cables.
|
|
|
|
|
|
The 1Gb interfaces connect to a set of 12
|
|
|
[Dell S3048-ON](http://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell-S3048-ON-Spec-Sheet.pdf)
|
|
|
48-port 1Gb switches (three per rack) with each of those switches uplinking
|
|
|
via 4 x 10Gb to the Z9500.
|
|
|
|
|
|
The specific node configurations are:
|
|
|
|
|
|
* **pc701-716, pc741-756, pc781-796, pc821-836:** 4 x 10Gb, 2 x 1Gb
|
|
|
* **pc717-740, pc757-780, pc797-820, pc837-860:** 2 x 10Gb, 4 x 1Gb
|
|
|
|
|
|
**Important**: The 10Gb interfaces in the d430 nodes are not capable of
|
|
|
operating at speeds lower than 10Gb, so make sure to request link speeds of
|
|
|
"10Gb" or "*" in your NS file to get one.
|
|
|
|
|
|
### Images and Kernel Support
|
|
|
|
|
|
Currently, the following Emulab standard images are supported on these machines:
|
|
|
|
|
|
* Ubuntu: 64-bit: UBUNTU14-64-STD, UBUNTU16-64-STD
|
|
|
* CentOS: 64-bit: CENTOS7-64-STD
|
|
|
* FreeBSD: 64-bit: FBSD102-64-STD
|
|
|
* Xen 4.6 with Ubuntu14 "dom0": 64-bit: XEN46-64-STD
|
|
|
|
|
|
Many of the other standard Emulab images will likely work as well, though
|
|
|
older (more than 2-3 years) FreeBSD and Linux images will not work. Also,
|
|
|
see the caveats below.
|
|
|
|
|
|
### Caveats
|
|
|
|
|
|
* **No support for 32-bit OS images.** No technical reason here, just low
|
|
|
priority. 32-bit OSes would allow access to only 4GB of the 64GB of
|
|
|
RAM, so there is no good reason to use them.
|
|
|
* **No support for Windows.** There are no current plans to support
|
|
|
Windows on these nodes.
|
|
|
* **Older versions of the 10Gb NIC driver are buggy.** If you are
|
|
|
planning on using your existing custom image, be aware that the Intel
|
|
|
X710 NICs are relatively new and older versions of the driver (in both
|
|
|
Linux and FreeBSD) have issues. **We strongly encourage you to remake**
|
|
|
**your custom images**, basing them on the standard images above. If
|
|
|
that is infeasible, then you should at least update the driver ("i40e"
|
|
|
for Linux, "ixl" for FreeBSD) from the Intel web site.
|
|
|
* **The system (boot) disk is an SSD.** Generally this is a good thing,
|
|
|
but the performance of SSDs can degrade over time unless unused blocks
|
|
|
are erased (TRIM). While Emulab attempts to periodically TRIM the disk
|
|
|
to keep performance high, you might also need to occasionally do this
|
|
|
if you are writing the root filesystem or an extra filesystem on this
|
|
|
disk (the root filesystem takes about 16GB, while the rest is available
|
|
|
for an extra user filesystem). On Linux, you can run fstrim on an
|
|
|
existing filesystem. On FreeBSD, you would need to turn on the trim
|
|
|
feature with tunefs and then create/remove a large file.
|
|
|
* **Traffic shaping has not been calibrated on these nodes!** Using one
|
|
|
of these nodes as a traffic shaping node should work fine at 1Gb/second
|
|
|
or less, but shaping bandwidth above that, or applying delay or loss to
|
|
|
a 10Gb link, will likely give you erratic (aka, "bad") results. We plan
|
|
|
to work on this.
|
|
|
* **Limited bandwidth to other node types.** Currently, the d430 node
|
|
|
data plane switch connects to the d710 data plane via a single 10Gb
|
|
|
link. By March 2016, we should have 160Gb of bandwidth directly to the
|
|
|
d820 data plane switch and 120Gb to the d710 data plane
|
|
|
switches. pc3000s will continue to be reached via hopping through the
|
|
|
d710 data plane. The control network connects at 10Gb to the d710
|
|
|
control plane. See [this
|
|
|
diagram](http://www.emulab.net/doc/docwrapper.php3?docname=topo.html)
|
|
|
for the Emulab data plane layout. |