The Utah Emulab cluster has 160 d430 PC nodes (pc701-pc860) consisting of:
- Dell Poweredge R430 1U servers
- Two 2.4 GHz 64-bit 8-Core E5-2630 "Haswell" processors, 8.0 GT/s bus speed, 20 MB cache, and VT (VT-x, EPT, and VT-d) support
- 64 GB 2133 MT/s DDR4 RAM (8 x 8GB modules)
- 4 Broadcom NetXtreme BCM5720 GbE NICs builtin to motherboard (only one in use)
- 1 dual-port or 1 quad-port Intel I350 1GbE PCI-Express NICs
- 1 dual-port or 1 quad-port Intel X710 10GbE PCI-Express NICs
- 1 200GB Intel SATA SSD, 2 x 1TB 7200 rpm SATA disks
The nodes are arranged in four racks, 40 nodes per rack. Each rack has its own management and control network switches along with three experimental switches. The only cross-rack wiring is for the management and control network interconnects and the 10Gb interfaces (both on the nodes and for the experimental switches).
Switch Configuration
Each node has six experimental interfaces. 64 of the 160 machines (16 per rack) have four 10Gb interfaces and two 1Gb interfaces. The other 96 machines (24 per rack) have two 10Gb interfaces and four 1Gb interfaces.
All of the 10Gb interfaces connect to a single Dell Z9500 132-port 40Gb switch using 10Gb to 40Gb breakout cables.
The 1Gb interfaces connect to a set of 12 Dell S3048-ON 48-port 1Gb switches (three per rack) with each of those switches uplinking via 4 x 10Gb to the Z9500.
The specific node configurations are:
- pc701-716, pc741-756, pc781-796, pc821-836: 4 x 10Gb, 2 x 1Gb
- pc717-740, pc757-780, pc797-820, pc837-860: 2 x 10Gb, 4 x 1Gb
Important: The 10Gb interfaces in the d430 nodes are not capable of operating at speeds lower than 10Gb, so make sure to request link speeds of "10Gb" or "*" in your NS file to get one.
Images and Kernel Support
Currently, the following Emulab standard images are supported on these machines:
- Ubuntu: 64-bit: UBUNTU14-64-STD, UBUNTU16-64-STD
- CentOS: 64-bit: CENTOS7-64-STD
- FreeBSD: 64-bit: FBSD103-64-STD, FBSD111-64-STD
- Xen 4.6 with Ubuntu16 "dom0": 64-bit: XEN46-64-STD
Many of the other standard Emulab images will likely work as well, though older (more than 2-3 years) FreeBSD and Linux images will not work. Also, see the caveats below.
Caveats
- No support for 32-bit OS images. No technical reason here, just low priority. 32-bit OSes would allow access to only 4GB of the 64GB of RAM, so there is no good reason to use them.
- No support for Windows. There are no current plans to support Windows on these nodes.
- Older versions of the 10Gb NIC driver are buggy. If you are planning on using your existing custom image, be aware that the Intel X710 NICs are relatively new and older versions of the driver (in both Linux and FreeBSD) have issues. We strongly encourage you to remake your custom images, basing them on the standard images above. If that is infeasible, then you should at least update the driver ("i40e" for Linux, "ixl" for FreeBSD) from the Intel web site.
- The system (boot) disk is an SSD. Generally this is a good thing, but the performance of SSDs can degrade over time unless unused blocks are erased (TRIM). While Emulab attempts to periodically TRIM the disk to keep performance high, you might also need to occasionally do this if you are writing the root filesystem or an extra filesystem on this disk (the root filesystem takes about 16GB, while the rest is available for an extra user filesystem). On Linux, you can run fstrim on an existing filesystem. On FreeBSD, you would need to turn on the trim feature with tunefs and then create/remove a large file.
- Traffic shaping has not been calibrated on these nodes! Using one of these nodes as a traffic shaping node should work fine at 1Gb/second or less, but shaping bandwidth above that, or applying delay or loss to a 10Gb link, will likely give you erratic (aka, "bad") results. We plan to work on this.
- Limited bandwidth to other node types. Currently, the d430 node data plane switch connects to the d710 data plane via a single 10Gb link. By March 2016, we should have 160Gb of bandwidth directly to the d820 data plane switch and 120Gb to the d710 data plane switches. pc3000s will continue to be reached via hopping through the d710 data plane. The control network connects at 10Gb to the d710 control plane. See this diagram for the Emulab data plane layout.