Bringing up a new Emulab installation requires significant skills, time,
and certain hardware infrastructure. We strongly advise you not to attempt
it without them. If you do, don't expect too much help from those on the
emulab-admins list or
people at Utah.
Appropriate technical expertise: The ideal people to bring up an Emulab are experienced and high quality system and network administrators; good student versions of such people can work. If you don't have such a person or persons, don't proceed.
Design of the overall network architecture of your Emulab, and to some extent, the planned hardware. This aspect is not as simple technically as it appears, and includes financial, security, and support tradeoffs. We advise you to pass your design by the emulab-admins list for feedback.
Serial line consoles and remote power controllers should be connected to at least two experimental nodes.
Supported Hardware and Software Environment
This software makes some assumptions about the environment in which it is run. Some of the most basic are listed below. We don't have the resources to adapt it to every possible environment. So, you will need to either work out a way to match the environment outlined below, or be willing to invest substantial work in adapting the software.
Emulab assumes a minimum of two dedicated server machines, known hereafter as 'boss' and 'ops', for Emulab setup. While the 'ops' server normally hosts the Emulab shared filesystems, we also support the use of a separate, dedicated filesystem server, hereafter referred to as 'fs'. People have set up the Emulab servers as VMs under VMWare and as of the FreeBSD 10.0 based release, these servers can be instantiated as VMs under Xen as well.
You will need at least two network interfaces on each experimental node: one for the control network and one for the experimental network. The experimental network needs to be one on which you can make VLANs with SNMP. Currently, Emulab supports:
Cisco 6500/6500-E and 4000 series switches (though not all switches in these lines have been tested - the 6513, 6509, 6506, 4006, and 4506 are known to work, running either CatOS or IOS). Cisco 3750's probably work but have not been tested. The 2950, 2960 and 2980 switches are known to work, although they are limited to a small number (64) of VLANs. (In general, it's the supervisor module, rather than the chassis, that matters, and Emulab supports all supervisor modules for the 6500s that we know about.)
HP Procurve. The 5400zl line has seen heavy production use on two Emulab sites. Other models may be easy to support, but this has not been tested.
Some Foundry switches.
Some Force10 (Dell) switches.
Some Mellanox switches.
Some Arista switches.
The control network must have full multicast support, including IGMP snooping. Nodes' control network interfaces must support PXE booting.
We highly, highly recommend that boss, ops, fs, and all the nodes be in publicly-routed IP space. If this is not possible, then boss, ops, and fs should be given two interfaces: One in the nodes' control network, and one in public IP space. If you must use private IP space for the nodes' control network, we suggest using the 192.168/16 subnet, which leaves the larger 10/8 subnet available for the experimental network. The defs-example-privatecnet file shows an example configuration like this.
If you have a firewall, you will need to be able to get certain standard ports through to boss and ops, such as the ports for http, https, ssh, named (domain), and smtp. Any other strange network setup (such as NAT) between the boss/ops/fs servers and the outside world will cause really big headaches.
The whole testbed should be in a domain or subdomain for which boss can be the name server.
The testbed nodes must be able to reach boss with DHCP requests on the control network - this means either being in the same broadcast domain (i.e. LAN), or, if there is a router in between, the router must be capable of forwarding DHCP/BOOTP packets. Since the nodes will DHCP from boss, it is important that there not be another DHCP server (i.e. one for another part of your lab) to answer their requests.
The boss node should have its own local disk space, for various reasons:
For logistical reasons, /usr/testbed cannot be shared between boss and ops, or between boss and fs, or between ops and fs. All three install different subsets of the Emulab software.
For security reasons, /usr/testbed/images, which is the home of the "trusted" default disk images, should not be hosted on ops/fs since they are potentially more vulnerable.
Similarly, home directories for "real" (admin) users on boss should not be shared with, or hosted from, ops or fs. See doc/shellonboss.txt for details.
If you have any questions about these requirements, please contact the emulab-admins list before proceeding.