diff --git a/www/doc/hw-recommend.html b/www/doc/hw-recommend.html new file mode 100644 index 0000000000000000000000000000000000000000..55c7a9d8975aa3ea8a10de8e81dba85c6401eecf --- /dev/null +++ b/www/doc/hw-recommend.html @@ -0,0 +1,162 @@ + + + Emulab.Net - Hardware Recommendations + + + + + +
+

Hardware Recommendations

+
+ +

Contents

+ + +
+ +

Purpose of this document

+The purpose of this document is to share our experience in selecting the +hardware in Emulab, to aid others who are considering similar projects. It +outlines the impact in terms of testbed featres caused by hardware decisions. + +
+ +

Nodes

+
+
CPU
+
Take your pick. Note that small case size (ie. 1U) may limit your + options, due to heat issues. Many experiments will not be CPU bound, but + some uses (ie. cluster computing experiments) will appreciate fast + CPUs.
+ +
Memory
+
Any amount is probably OK. Most of our experimenters don't seem to + need much, but at least one has wished we had more than 512MB. We chose + to go with ECC, since it is not much more expensive than non-ECC
+ +
NICs
+
At least 2. Our attempts to use nodes with only 1 NIC have not met + with much sucess. 3 interfaces lets you have delay nodes, 4 lets you + have (really simple) routers. Control net interface should have PXE + capability. All experimental interfaces should be the same, unless you + are purposely going for a heterogenous environment. Control net + interface can be different. Intel EtherExpress (fxp) and 3Com 3c??? + (xl) are known to work. Note that not all these cards have PXE, so make + sure that your control net card is one of the models that does.
+ +
Motherboard
+
Serial console redirection is nice, and the BIOS should + the ability to PXE boot from a card. The ability to set IRQs per slot + is desirable (to avoid setups where multiple cards share one IRQ.) + Health monitoring hardware (temperature, fans, etc.) is good too, but + not required. All of our boards so far have been based on Intel's + aging put proven BX chipset.
+ +
Hard Drive
+
Pretty much any one is OK. With a large number of nodes, you are + likely to run into failures, so they should be reasonably reliable
+ + +
Floppy
+
Handy for BIOS updates, and may be used for 'neutering' PXE + BIOSen, but not required otherwise
+ +
CDROM
+
Not needed
+ +
VGA
+
Only needed if motherboard does not have serial console + redirection. Cheap is fine.
+
+ +
+ +

Switches

+
+
Control net
+
+
VLANs
+
Protects control hw (switches, power, etc.) from nodes + and world, and private machines. Without them, control + hardware can be attached to an additional NIC on the + boss node (requires extra cheap switch.)
+ +
Router
+
Required if VLANs are used. Must have DHCP/bootp forwarding. + (Cisco calls this the 'IP Helper') A MSFC card in a Catalyst + supervisor module works well for us, but a PC would probably + suffice.
+ +
Firewall
+
Can be the router, without it VLAN security is pretty much + useless, meaning that it may be possible for unautorized people to + reboot nodes and configure experimental network switches.
+ +
Port MAC security
+
Helps prevent nodes from impersonating each other and control + nodes. This is an unlikely attack, since it must be attempted by + an experimenter or someone who has compromised an experimental + node already.
+ +
Multicast
+
Switch multicast support (IGMP snooping) and router (if + present) is used to allow multicast loading of disk images. + Otherwise, they must be done unicast and serially, which is an + impediment to re-loading disks after experiments (to return nodes + to a clean state.)
+
+ +
Experimental net
+
+
VLAN support
+
Optimally, configurable via SNMP (we have no tools to configure + it otherwise.) If not available, all experimental isolation is + lost, delay nodes can't be used well, and method for coming up with + globablly unique IP addresses will be required. So, VLANs are + basically necessary.
+ +
Trunking
+
If multiple switches, should have a way to trunk them (or + experiments are limited to 1 switch.) Ideally, all trunked switches + should support VLAN management (like Cisco's VTP) and a VLAN + trunking protocol like 802.1q . It's good if the trunk links are + at least an order of magnitude faster than node links, and link + aggregation (ie. EtherChannel) is desirable.
+
+
+ +
+ +

Servers

+ Two is pereferable, though 1 could be made to work. The NFS server needs + enough space for /users and /proj , as well as a few extra gigs for build + trees, logs, and images. If you have more than 128 nodes, and plan to use + Cyclades for serial ports, you need 1 tip server per 128 serial lines. + >100Mbit line is suggested for disk image distribution machine (usually + boss.) Database machine should have reasonably fast CPU and plenty of RAM. + +
+ +

Other Hardware

+
+
Power controllers
+
Without them, nodes can not be reliably rebooted.
+ +
Serial (console) ports
+
Custom kernels/OSes (specifically, the OSKit) may not support ssh, + etc. Also useful if an experimenter somehow scrogs network.
+ +
Serial port reset controllers
+
It may be possible to build (or buy) serial-port passthroughs that + are wired to reset pins on MB. Some motherboard chipsets (eg. Intel + LX+) have this feature built on. NOT TESTED by Utah, and may not be + 100% reliable (MB may be able to get into a state where reset pins are + not functional.) Theoretically nicer to the hardware than power + controllers.
+