The Utah Emulab cluster has 160 PC nodes (pc401-pc560) consisting of:
- Dell r710 2U servers
- One 2.4 GHz 64-bit Quad Core Xeon E5530 "Nehalem" processor, 5.86 GT/s bus speed, 8 MB L3 cache, VT (VT-x, EPT and VT-d) support
- 12 GB 1066 MHz DDR2 RAM (6 x 2GB modules)
- 4 Broadcom NetXtreme II BCM5709 rev C GbE NICs builtin to motherboard
- 2 Broadcom NetXtreme II BCM5709 rev C GbE NICs in dual-port PCIe x4 expansion card (one NIC is the control net)
- 1 500GB 7200 rpm Western Digital SATA disk (drive 0)
- 1 250GB 7200 rpm Seagate SATA disk (drive 1)
All interfaces on these machines are connected to gigabit ports on ProCurve 5412zl switches (specifically, J8702A modules). There are three experimental-net switches (procurve3-procurve5) for these nodes, each with approximately 240 ports. A fourth switch (procurve1) acts as the center of a 'hub and spoke' topology for the experimental network: it is connected to the other three ProCurve switches at 24.4 Gbps each (via a 4 x 10Gb module, but the backplane can only manage 24.4 Gbps aggregate) and to the rest of the testbed via 4 Gbps and 8 Gbps trunks.
Most nodes have 4 experimental-net interfaces, but some have 5. The d710 nodes have their interfaces 'striped' across the switches: each PC has at least one interface on each of the three hub switches, so that large LANs can be constructed of all nodes, without needing to use any inter-switch bandwidth. The layout is:
- pc401-pc440: 4 interfaces: 2 x procurve3, 1 x procurve4, 1 x procurve5
- pc441-pc480: 4 interfaces: 1 x procurve3, 2 x procurve4, 1 x procurve5
- pc481-pc520: 4 interfaces: 2 x procurve3, 1 x procurve4, 1 x procurve5
- pc521-pc560: 5 interfaces: 1 x procurve3, 2 x procurve4, 2 x procurve5
A fifth switch (procurve2) acts as the control network for the nodes.
Images and Kernel Support
Currently, the following Emulab standard images work on the machines. Officially supported images are highlighted:
- 32-bit: FBSD72-STD, FBSD73-STD, FBSD83-STD, FBSD91-STD
- 64-bit: FBSD73-64-STD, FBSD83-64-STD, FBSD91-64-STD, FBSD103-64-STD, FBSD113-64-STD, FBSD120-64-STD
- 32-bit: CENTOS55-STD
- 64-bit: CENTOS7-64-STD
- 64-bit: UBUNTU14-64-STD, UBUNTU16-64-STD, UBUNTU18-64-STD
- 32-bit: WIN7-STD
Older FreeBSD and Fedora images will not work on these machines because they require a newer NIC driver that supports the Broadcom 5709 chipset. FreeBSD 7 and up, and Linux kernels >= 2.6.22 (although 2.6.20 and up may work), should support these NICs.
- Limited bandwidth to other node types. The experimental interconnect is limited to 10Gb to the pc3000 nodes via a link from cisco8 to procurve1, 4 x 10Gb to the d820 nodes via a link from arista1 to procurve1, and 4 x 10Gb to the d430 nodes via a link from z9500 to procurve1. The control network has a single Gb link between cisco6 and procurve2 (pc3000s) and a 10Gb link between dellexpt0 and procurve2 (d430s).
- No control net firewall support. For one utterly obscure reason, these nodes cannot be control net firewalls. This will be fixed in the future.