4 Broadcom NetXtreme BCM5720 GbE NICs builtin to motherboard (only one in use)
1 dual-port or 1 quad-port Intel I350 1GbE PCI-Express NICs
1 dual-port or 1 quad-port Intel X710 10GbE PCI-Express NICs
1 200GB Intel SATA SSD, 2 x 1TB 7200 rpm SATA disks
The nodes are arranged in four racks, 40 nodes per rack. Each rack has its
own management and control network switches along with three experimental
switches. The only cross-rack wiring is for the management and control
network interconnects and the 10Gb interfaces (both on the nodes and for
the experimental switches).
Each node has six experimental interfaces. 64 of the 160 machines (16 per
rack) have four 10Gb interfaces and two 1Gb interfaces. The other 96
machines (24 per rack) have two 10Gb interfaces and four 1Gb interfaces.
All of the 10Gb interfaces connect to a single
132-port 40Gb switch using 10Gb to 40Gb breakout cables.
The 1Gb interfaces connect to a set of 12
48-port 1Gb switches (three per rack) with each of those switches uplinking
via 4 x 10Gb to the Z9500.
The specific node configurations are:
pc701-716, pc741-756, pc781-796, pc821-836: 4 x 10Gb, 2 x 1Gb
pc717-740, pc757-780, pc797-820, pc837-860: 2 x 10Gb, 4 x 1Gb
Important: The 10Gb interfaces in the d430 nodes are not capable of
operating at speeds lower than 10Gb, so make sure to request link speeds of
"10Gb" or "*" in your NS file to get one.
Images and Kernel Support
Currently, the following Emulab standard images are supported on these machines:
Ubuntu: 64-bit: UBUNTU14-64-STD, UBUNTU16-64-STD
CentOS: 64-bit: CENTOS7-64-STD
FreeBSD: 64-bit: FBSD103-64-STD, FBSD111-64-STD
Xen 4.6 with Ubuntu16 "dom0": 64-bit: XEN46-64-STD
Many of the other standard Emulab images will likely work as well, though
older (more than 2-3 years) FreeBSD and Linux images will not work. Also,
see the caveats below.
No support for 32-bit OS images. No technical reason here, just low
priority. 32-bit OSes would allow access to only 4GB of the 64GB of
RAM, so there is no good reason to use them.
No support for Windows. There are no current plans to support
Windows on these nodes.
Older versions of the 10Gb NIC driver are buggy. If you are
planning on using your existing custom image, be aware that the Intel
X710 NICs are relatively new and older versions of the driver (in both
Linux and FreeBSD) have issues. We strongly encourage you to remakeyour custom images, basing them on the standard images above. If
that is infeasible, then you should at least update the driver ("i40e"
for Linux, "ixl" for FreeBSD) from the Intel web site.
The system (boot) disk is an SSD. Generally this is a good thing,
but the performance of SSDs can degrade over time unless unused blocks
are erased (TRIM). While Emulab attempts to periodically TRIM the disk
to keep performance high, you might also need to occasionally do this
if you are writing the root filesystem or an extra filesystem on this
disk (the root filesystem takes about 16GB, while the rest is available
for an extra user filesystem). On Linux, you can run fstrim on an
existing filesystem. On FreeBSD, you would need to turn on the trim
feature with tunefs and then create/remove a large file.
Traffic shaping has not been calibrated on these nodes! Using one
of these nodes as a traffic shaping node should work fine at 1Gb/second
or less, but shaping bandwidth above that, or applying delay or loss to
a 10Gb link, will likely give you erratic (aka, "bad") results. We plan
to work on this.
Limited bandwidth to other node types. Currently, the d430 node
data plane switch connects to the d710 data plane via a single 10Gb
link. By March 2016, we should have 160Gb of bandwidth directly to the
d820 data plane switch and 120Gb to the d710 data plane
switches. pc3000s will continue to be reached via hopping through the
d710 data plane. The control network connects at 10Gb to the d710
control plane. See this
for the Emulab data plane layout.