Hardware Recommendations
Contents
Purpose of this document
The purpose of this document is to share our experience in selecting the
hardware in Emulab, to aid others who are considering similar projects. Rather
than a recipe for bulding testbeds, this document gives a set of
recommendations, and outlines the consequences of each choice.
Nodes
- NICs
- At least 2 - one for control net, one for the experimental
network. Our attempts to use nodes with only 1 NIC have not
met with much success. 3 interfaces lets you have delay nodes,
but only linear topolgies (no branching.) 4 lets you have
(really simple) routers. We opted to go with 5. Control net
interface should have PXE capability. All experimental
interfaces should be the same, unless you are purposely going
for a heterogenous environment. Control net interface can be
different. Intel EtherExpress (fxp) and 3Com 3c??? (xl) are
known to work. Note that not all these cards have PXE, and
different versions of PXE have different sets of bugs, so it's
best to get a sample first and try it out before committing to
a card. Depedning on usage, it may be OK to get a large number
of nodes with 2 interfaces to be edge nodes, and a smaller
number with more interfaces to be routers.
- Case/Chassis
- This will depend on your space requirements. The cheapest
option is to buy standard desktop machines, but these are not
very space-efficient. 3U or 4U rackmount cases generally have
plenty of space for PCI cards and CPU cooling fans, but still
may consume too much space for a large-scale testbed. Smaller
cases (1U or 2U) have fewer PCI slots (usually 2 for 2U cases,
and 1 or 2 in 1U cases), and often require custom motherboards.
Heat is an issue in smaller cases, as they do not have room for
CPU fans. This limits the processor speed that can be used.
For our first round of machines, we bought standard
motherboards and 4U cases, and assmbled the machines ourselves.
For our second round of PCs, we opted for the Intel ISP1100 server platform,
which includes a 1U case, custom motherboard with 2 onboard
NICs and serial console redirection, and a power supply. This
product has been discontinued, but others like it are available
from other vendors.
- CPU
- Take your pick. Note that small case size (ie. 1U) may
limit your options, due to heat issues. Many experiments will
not be CPU bound, but some uses (ie. cluster computing
experiments) will appreciate fast CPUs.
- Memory
- Any amount is probably OK. Most of our experimenters don't
seem to need much, but at least one has wished we had more than
512MB. We chose to go with ECC, since it is not much more
expensive than non-ECC, and with our large scale, ECC will help
protect against failure.
- Motherboard
- Serial console redirection is nice, and the BIOS should
have the ability to PXE boot from a card. The ability to set
IRQs per slot is desirable (to avoid setups where multiple
cards share one IRQ.) Health monitoring hardware (temperature,
fans, etc.) is good too, but not required. All of our boards so
far have been based on Intel's aging put proven BX chipset.
Onboard network interfaces can allow you get get more NICs,
something especially valuable for small cases with a limited
number of PCI slots.
- Hard Drive
- Pretty much any one is OK. With a large number of nodes,
you are likely to run into failures, so they should be
reasonably reliable
- Floppy
- Handy for BIOS updates, and may be used for 'neutering' PXE
BIOSen, but not required otherwise
- CDROM
- Recommended if you might grow your testbed pretty large (100)
some day. For reliable scaling we will probably migrate away
from PXE to CD-based booting, as our widearea nodes do.
- VGA
- Only needed if motherboard does not have serial console
redirection. Cheap is fine.
Switches
- Control net
- Number of ports
- You'll need one port for each testbed node, plus
ports for control nodes, power controllers, etc.
- VLANs
- Protects control hw (switches, power, etc.) from
nodes and world, and private machines. Without them,
control hardware can be attached to an additional NIC
on the boss node (requires extra cheap switch.)
- Router
- Required if VLANs are used. Must have DHCP/bootp
forwarding. (Cisco calls this the 'IP Helper') A MSFC
card in a Cisco Catalyst supervisor module works well
for us, but a PC would probably suffice.
- Firewall
- Can be the router, without it VLAN security is
pretty much useless, meaning that it may be possible
for unautorized people to reboot nodes and configure
experimental network switches.
- Port MAC security
- Helps prevent nodes from impersonating each other
and control nodes. This is an unlikely attack, since
it must be attempted by an experimenter or someone who
has compromised an experimental node already.
- Multicast
- Switch multicast support (IGMP snooping) and
router (if present) is used to allow multicast loading
of disk images. Otherwise, they must be done unicast
and serially, which is an impediment to re-loading
disks after experiments (to return nodes to a clean
state.)
- Experimental net
- Vendor
- Our software currently supports
the Cisco Catalyst 6000 and 4000 series and Intel 510T
switches (although the Intels don't support everything you'd
want for a large-scale production testbed). Other
switches by the same vendors will likely be easy to support -
in particular, we expect that Cisco swithces supporting
the 'CISCO-VTP-MIB', and the 'CISCO-STACK-MIB' or
'CISCO-VLAN-MEMBERSHIP-MIB' are likely to 'just work'.
(You can see a list of which MIBs are
supported by which devices at Cisco's
MIB page.) If you will be using multiple Ciscos for the
experimental net, and trunking between them, your switch
should also support the 'CISCO-PAGP-MIB', though this is not
required. Switches from other vendors can theoretically be
used if they support SNMP for management, but will likely
require significant work. We have found that you can often
find good deals on new or used equipment on EBay.
- Number of ports
- You'll need as many ports as you have experimental
interfaces on your nodes. In addition, you need 1 port
to connect the experimental net switch to the control
net (for SNMP configuration.) If you're using multiple
switches, you need sufficient ports to 'stack' them
together - If your switches are 100Mbit, Gigabit ports
are useful for this.
- VLAN support
- Optimally, configurable via SNMP (we have no tools
to configure it otherwise.) If not available, all
experimental isolation is lost, delay nodes can't be
used well, and method for coming up with globablly
unique IP addresses will be required. So, VLANs are
basically necessary.
- Trunking
- If multiple switches, should have a way to trunk
them (or experiments are limited to 1 switch.) Ideally,
all trunked switches should support VLAN management
(like Cisco's VTP) and a VLAN trunking protocol like
802.1q . It's good if the trunk links are at least an
order of magnitude faster than node links, and link
aggregation (ie. EtherChannel) is desirable.
Servers
Two is pereferable, though 1 could be made to work. The NFS server
needs enough space for /users and /proj , as well as a few extra gigs
for build trees, logs, and images. If you have more than 128 nodes, and
plan to use Cyclades for serial ports, you need 1 tip server per 128
serial lines. >100Mbit line is suggested for disk image distribution
machine (usually boss.) Database machine should have reasonably fast
CPU and plenty of RAM.
Other Hardware
- Network cables
- We use Cat5E, chosen becasue they are not much more
expensive than Cat5, and can be used in the future for gigabit
ethernet. It has been our experience that 'boots' on cables do
more harm than good. The main problems are that they make it
difficult to disconnect the cables one connected, and that they
get in the way on densely-connected switches. Cables with
'molded strain releif' are better than cables with boots, but
are often much more extensive. We buy cables in two-foot
increments, which keeps slack low without making the order too
complicated. Our standard so far has been to make control net
cables red, experimental net cables yellow, serial cables
white, and cables for control hardware (such as power
controllers) green. We've bought all of our cables from dataaccessories.com,
and have had excellent luck with them. With prices of $3.00 to
$4.25 for premade cables, it's not too expesive to by them
rather than make your own, and it's much easier.
- Serial cables
- We use Cat5E, but with a special pin pattern on the ends to
avoid interference between the transmit/receive pairs. We use
RJ-45 connectors on both ends, and a custom serial hood to
connect to the DB-9 serial ports on the nodes. Contact us to
get our custom cable specs.
- Power controllers
- Without them, nodes can not be reliably rebooted. We
started out with 8-port SNMP-controlled power contollers from
APC. Our newer nodes use the
RPC-27 from BayTech,
20-outlet vertically-mounted, serial-controlled power
controllers. The serial controllers are genrally cheaper, and
the more ports on each controller, the cheaper.
- Serial (console) ports
- Custom kernels/OSes (specifically, the OSKit) may not
support ssh, etc. Also useful if an experimenter somehow scrogs
network. We use the Cyclades
Cyclom Ze serial adapters, which allow up to 128 serial ports
in a single PC.
- Serial port reset controllers
-
- It may be possible to build (or buy) serial-port
passthroughs that are wired to reset pins on MB. Some
motherboard chipsets (eg. Intel LX+) have this feature built
on. NOT TESTED by Utah, and may not be 100% reliable (MB may be
able to get into a state where reset pins are not functional.)
Theoretically nicer to the hardware than power controllers.
-