Hardware Recommendations

Contents


Purpose of this document

The purpose of this document is to share our experience in selecting the hardware in Emulab, to aid others who are considering similar projects. Rather than a recipe for bulding testbeds, this document gives a set of recommendations, and outlines the consequences of each choice.

Nodes

NICs
At least 2 - one for control net, one for the experimental network. Our attempts to use nodes with only 1 NIC have not met with much success. 3 interfaces lets you have delay nodes, but only linear topolgies (no branching.) 4 lets you have (really simple) routers. We opted to go with 5. Control net interface should have PXE capability. All experimental interfaces should be the same, unless you are purposely going for a heterogenous environment. Control net interface can be different. Intel EtherExpress (fxp) and 3Com 3c??? (xl) are known to work. Note that not all these cards have PXE, and different versions of PXE have different sets of bugs, so it's best to get a sample first and try it out before committing to a card. Depedning on usage, it may be OK to get a large number of nodes with 2 interfaces to be edge nodes, and a smaller number with more interfaces to be routers.
Case/Chassis
This will depend on your space requirements. The cheapest option is to buy standard desktop machines, but these are not very space-efficient. 3U or 4U rackmount cases generally have plenty of space for PCI cards and CPU cooling fans, but still may consume too much space for a large-scale testbed. Smaller cases (1U or 2U) have fewer PCI slots (usually 2 for 2U cases, and 1 or 2 in 1U cases), and often require custom motherboards. Heat is an issue in smaller cases, as they do not have room for CPU fans. This limits the processor speed that can be used. For our first round of machines, we bought standard motherboards and 4U cases, and assmbled the machines ourselves. For our second round of PCs, we opted for the Intel ISP1100 server platform, which includes a 1U case, custom motherboard with 2 onboard NICs and serial console redirection, and a power supply. This product has been discontinued, but others like it are available from other vendors.
CPU
Take your pick. Note that small case size (ie. 1U) may limit your options, due to heat issues. Many experiments will not be CPU bound, but some uses (ie. cluster computing experiments) will appreciate fast CPUs.
Memory
Any amount is probably OK. Most of our experimenters don't seem to need much, but at least one has wished we had more than 512MB. We chose to go with ECC, since it is not much more expensive than non-ECC, and with our large scale, ECC will help protect against failure.
Motherboard
Serial console redirection is nice, and the BIOS should have the ability to PXE boot from a card. The ability to set IRQs per slot is desirable (to avoid setups where multiple cards share one IRQ.) Health monitoring hardware (temperature, fans, etc.) is good too, but not required. All of our boards so far have been based on Intel's aging put proven BX chipset. Onboard network interfaces can allow you get get more NICs, something especially valuable for small cases with a limited number of PCI slots.
Hard Drive
Pretty much any one is OK. With a large number of nodes, you are likely to run into failures, so they should be reasonably reliable
Floppy
Handy for BIOS updates, and may be used for 'neutering' PXE BIOSen, but not required otherwise
CDROM
Recommended if you might grow your testbed pretty large (100) some day. For reliable scaling we will probably migrate away from PXE to CD-based booting, as our widearea nodes do.
VGA
Only needed if motherboard does not have serial console redirection. Cheap is fine.

Switches

Control net
Number of ports
You'll need one port for each testbed node, plus ports for control nodes, power controllers, etc.
VLANs
Protects control hw (switches, power, etc.) from nodes and world, and private machines. Without them, control hardware can be attached to an additional NIC on the boss node (requires extra cheap switch.)
Router
Required if VLANs are used. Must have DHCP/bootp forwarding. (Cisco calls this the 'IP Helper') A MSFC card in a Cisco Catalyst supervisor module works well for us, but a PC would probably suffice.
Firewall
Can be the router, without it VLAN security is pretty much useless, meaning that it may be possible for unautorized people to reboot nodes and configure experimental network switches.
Port MAC security
Helps prevent nodes from impersonating each other and control nodes. This is an unlikely attack, since it must be attempted by an experimenter or someone who has compromised an experimental node already.
Multicast
Switch multicast support (IGMP snooping) and router (if present) is used to allow multicast loading of disk images. Otherwise, they must be done unicast and serially, which is an impediment to re-loading disks after experiments (to return nodes to a clean state.)
Experimental net
Vendor
Our software currently supports the Cisco Catalyst 6000 and 4000 series and Intel 510T switches (although the Intels don't support everything you'd want for a large-scale production testbed). Other switches by the same vendors will likely be easy to support - in particular, we expect that Cisco swithces supporting the 'CISCO-VTP-MIB', and the 'CISCO-STACK-MIB' or 'CISCO-VLAN-MEMBERSHIP-MIB' are likely to 'just work'. (You can see a list of which MIBs are supported by which devices at Cisco's MIB page.) If you will be using multiple Ciscos for the experimental net, and trunking between them, your switch should also support the 'CISCO-PAGP-MIB', though this is not required. Switches from other vendors can theoretically be used if they support SNMP for management, but will likely require significant work. We have found that you can often find good deals on new or used equipment on EBay.
Number of ports
You'll need as many ports as you have experimental interfaces on your nodes. In addition, you need 1 port to connect the experimental net switch to the control net (for SNMP configuration.) If you're using multiple switches, you need sufficient ports to 'stack' them together - If your switches are 100Mbit, Gigabit ports are useful for this.
VLAN support
Optimally, configurable via SNMP (we have no tools to configure it otherwise.) If not available, all experimental isolation is lost, delay nodes can't be used well, and method for coming up with globablly unique IP addresses will be required. So, VLANs are basically necessary.
Trunking
If multiple switches, should have a way to trunk them (or experiments are limited to 1 switch.) Ideally, all trunked switches should support VLAN management (like Cisco's VTP) and a VLAN trunking protocol like 802.1q . It's good if the trunk links are at least an order of magnitude faster than node links, and link aggregation (ie. EtherChannel) is desirable.

Servers

Two is pereferable, though 1 could be made to work. The NFS server needs enough space for /users and /proj , as well as a few extra gigs for build trees, logs, and images. If you have more than 128 nodes, and plan to use Cyclades for serial ports, you need 1 tip server per 128 serial lines. >100Mbit line is suggested for disk image distribution machine (usually boss.) Database machine should have reasonably fast CPU and plenty of RAM.

Other Hardware

Network cables
We use Cat5E, chosen becasue they are not much more expensive than Cat5, and can be used in the future for gigabit ethernet. It has been our experience that 'boots' on cables do more harm than good. The main problems are that they make it difficult to disconnect the cables one connected, and that they get in the way on densely-connected switches. Cables with 'molded strain releif' are better than cables with boots, but are often much more extensive. We buy cables in two-foot increments, which keeps slack low without making the order too complicated. Our standard so far has been to make control net cables red, experimental net cables yellow, serial cables white, and cables for control hardware (such as power controllers) green. We've bought all of our cables from dataaccessories.com, and have had excellent luck with them. With prices of $3.00 to $4.25 for premade cables, it's not too expesive to by them rather than make your own, and it's much easier.
Serial cables
We use Cat5E, but with a special pin pattern on the ends to avoid interference between the transmit/receive pairs. We use RJ-45 connectors on both ends, and a custom serial hood to connect to the DB-9 serial ports on the nodes. Contact us to get our custom cable specs.
Power controllers
Without them, nodes can not be reliably rebooted. We started out with 8-port SNMP-controlled power contollers from APC. Our newer nodes use the RPC-27 from BayTech, 20-outlet vertically-mounted, serial-controlled power controllers. The serial controllers are genrally cheaper, and the more ports on each controller, the cheaper.
Serial (console) ports
Custom kernels/OSes (specifically, the OSKit) may not support ssh, etc. Also useful if an experimenter somehow scrogs network. We use the Cyclades Cyclom Ze serial adapters, which allow up to 128 serial ports in a single PC.
Serial port reset controllers
It may be possible to build (or buy) serial-port passthroughs that are wired to reset pins on MB. Some motherboard chipsets (eg. Intel LX+) have this feature built on. NOT TESTED by Utah, and may not be 100% reliable (MB may be able to get into a state where reset pins are not functional.) Theoretically nicer to the hardware than power controllers.