Commit 3cb3f3c6 authored by Robert Ricci's avatar Robert Ricci
Browse files

First draft of the 'build-your-own-emulab' page.

parent 41c9582e
<title>Emulab.Net - Hardware Recommendations</title>
<link rel='stylesheet' href='../tbstyle-plain.css' type='text/css'>
<basefont size=4>
<h1>Hardware Recommendations</h1>
<li> <a href="#PURP">Purpose of this document</a>
<li> <a href="#NODES">Nodes</a>
<li> <a href="#SWITCHES">Switches</a>
<li> <a href="#SERVERS">Servers</a>
<a NAME="PURP"></a><h2>Purpose of this document</h2>
The purpose of this document is to share our experience in selecting the
hardware in Emulab, to aid others who are considering similar projects. It
outlines the impact in terms of testbed featres caused by hardware decisions.
<a NAME="NODES"></a><h2>Nodes</h2>
<dd>Take your pick. Note that small case size (ie. 1U) may limit your
options, due to heat issues. Many experiments will not be CPU bound, but
some uses (ie. cluster computing experiments) will appreciate fast
<dd>Any amount is probably OK. Most of our experimenters don't seem to
need much, but at least one has wished we had more than 512MB. We chose
to go with ECC, since it is not much more expensive than non-ECC</dd>
<dd>At least 2. Our attempts to use nodes with only 1 NIC have not met
with much sucess. 3 interfaces lets you have delay nodes, 4 lets you
have (really simple) routers. Control net interface should have PXE
capability. All experimental interfaces should be the same, unless you
are purposely going for a heterogenous environment. Control net
interface can be different. Intel EtherExpress (fxp) and 3Com 3c???
(xl) are known to work. Note that not all these cards have PXE, so make
sure that your control net card is one of the models that does.</dd>
<dd>Serial console redirection is nice, and the BIOS should
the ability to PXE boot from a card. The ability to set IRQs per slot
is desirable (to avoid setups where multiple cards share one IRQ.)
Health monitoring hardware (temperature, fans, etc.) is good too, but
not required. All of our boards so far have been based on Intel's
aging put proven BX chipset.</dd>
<dt><b>Hard Drive</b></dt>
<dd>Pretty much any one is OK. With a large number of nodes, you are
likely to run into failures, so they should be reasonably reliable</dd>
<dd>Handy for BIOS updates, and may be used for 'neutering' PXE
BIOSen, but not required otherwise</dd>
<dd>Not needed</dd>
<dd>Only needed if motherboard does not have serial console
redirection. Cheap is fine.</dd>
<a NAME="SWITCHES"></a><h2>Switches</h2>
<dt><b>Control net</b></dt>
<dd>Protects control hw (switches, power, etc.) from nodes
and world, and private machines. Without them, control
hardware can be attached to an additional NIC on the
boss node (requires extra cheap switch.)</dd>
<dd>Required if VLANs are used. Must have DHCP/bootp forwarding.
(Cisco calls this the 'IP Helper') A MSFC card in a Catalyst
supervisor module works well for us, but a PC would probably
<dd>Can be the router, without it VLAN security is pretty much
useless, meaning that it may be possible for unautorized people to
reboot nodes and configure experimental network switches.</dd>
<dt><b>Port MAC security</b></dt>
<dd>Helps prevent nodes from impersonating each other and control
nodes. This is an unlikely attack, since it must be attempted by
an experimenter or someone who has compromised an experimental
node already.</dd>
<dd>Switch multicast support (IGMP snooping) and router (if
present) is used to allow multicast loading of disk images.
Otherwise, they must be done unicast and serially, which is an
impediment to re-loading disks after experiments (to return nodes
to a clean state.)</dd>
<dt><b>Experimental net</b></dt>
<dt><b>VLAN support</b></dt>
<dd>Optimally, configurable via SNMP (we have no tools to configure
it otherwise.) If not available, all experimental isolation is
lost, delay nodes can't be used well, and method for coming up with
globablly unique IP addresses will be required. So, VLANs are
basically necessary.</dd>
<dd>If multiple switches, should have a way to trunk them (or
experiments are limited to 1 switch.) Ideally, all trunked switches
should support VLAN management (like Cisco's VTP) and a VLAN
trunking protocol like 802.1q . It's good if the trunk links are
at least an order of magnitude faster than node links, and link
aggregation (ie. EtherChannel) is desirable.</dd>
<a NAME="SERVERS"></a><h2>Servers</h2>
Two is pereferable, though 1 could be made to work. The NFS server needs
enough space for /users and /proj , as well as a few extra gigs for build
trees, logs, and images. If you have more than 128 nodes, and plan to use
Cyclades for serial ports, you need 1 tip server per 128 serial lines.
>100Mbit line is suggested for disk image distribution machine (usually
boss.) Database machine should have reasonably fast CPU and plenty of RAM.
<a NAME="OTHER"></a><h2>Other Hardware</h2>
<dt><b>Power controllers</b></dt>
<dd>Without them, nodes can not be reliably rebooted.</dd>
<dt><b>Serial (console) ports</b></dt>
<dd>Custom kernels/OSes (specifically, the OSKit) may not support ssh,
etc. Also useful if an experimenter somehow scrogs network.</dd>
<dt><b>Serial port reset controllers</b><dt>
<dd>It may be possible to build (or buy) serial-port passthroughs that
are wired to reset pins on MB. Some motherboard chipsets (eg. Intel
LX+) have this feature built on. NOT TESTED by Utah, and may not be
100% reliable (MB may be able to get into a state where reset pins are
not functional.) Theoretically nicer to the hardware than power
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment