Emulab now supports using IPMI (version 1.5 or 2.0), over either a
dedicated network connection or a shared (with the control network)
connection, for power cycling and the node console. The following describes
in broad terms how to setup such a node.
Note that many vendors implement their own management interface and GUI
with advanced features like virtual CD drives or ssh authentication, but we
only use basic IPMI supported functions. Thus in the text that follows,
when we talk about "enabling the IPMI" you are, for example, really
enabling the Dell iDRAC and then configuring IPMI parameters.
First you need to configure IPMI in the machine BIOS. This is highly vendor
specific, but for a Dell with an iDRAC7 or just BMC, there will be a menu
in which you can configure IPv4 parameters. We statically configure the IPs
though you probably could use DHCP.
If you have a dedicated management interface, then you can put all of these
interfaces in the so-called "hardware control network" switch VLAN where we
place all switches and power controllers. Assign it a unique IP on that
subnet, and set the mask appropriately. A DNS name is not required.
If the management interface is shared (with one of the regular network
interfaces), then make sure it is shared with the Emulab "node control
network" interface. In the shared case, we are assuming a fairly recent
built-in management controller that supports a distinct IP address for
talking to the management controller, ala recent Broadcom and Intel
chips. As with the dedicated case, assign a unique IP from the appropriate
For either case, you will want to also setup authentication. If you are
using a shared interface, it is CRITICAL that you do this, as the
management interface will be accessible to anyone with an account on the
"users" node or with a swapped-in experiment. Emulab supports the use of an
IPMI password to provide authentication. IPMI 2.0 also has support for
packet encryption, which Emulab optionally supports in a very basic way
(i.e., the encryption key is the same as the password). For a shared
interface, you will want encryption to protect the IPMI password when it is
sent to the management controller. If you have a dedicated interface that
is in the hardware control network and thus accessible only from boss, you
could get by without authentication, but it is still a good idea to have at
least a password. Passwords are set per-node in Emulab, you can use the
same password for all nodes.
Note that when using shared interfaces with a node control network that is
directly accessible from the Internet, you will want to firewall-off the
management IPs from the outside world.
Note that many management interfaces can use of an ssh keypair for
authentication which Emulab also supports, but IPMI only supports the
password/encryption-key and that is what is covered here.
Most management controllers have a web interface, so once you get the basic
IP and authentication info setup correctly, you can use the web interface
to configure the rest. If for no other reason, you should try connecting
and logging into the web interface as a test of whether the basic network
parameters are correctly configured.
On the Emulab side, you will need to install the ipmitool port on boss:
boss> sudo portupgrade -N ipmitool
should do the trick. Or, if you don't have portupgrade installed:
boss> cd /usr/ports/sysutils/ipmitoolboss> sudo make install
Configuring IPMI power cycling
We use a common software infrastructure for supporting HP iLo and Dell's
DRAC and DMC controllers. To get a node management interface into Emulab DB
and usable by the "power" command, you use "management_iface" on boss.
For a shared interface:
boss> wap management_iface -a AUTHTYPE -t TYPE NODE_ID MAC IP USER PASSWD
AUTHTYPE is either "pswd" to set just a password, or "key" to set both the
password and encryption key (for IPMI 2.0 controllers).
TYPE is either "ipmi15" or "ipmi20" depending on which version of the IPMI
protocol is supported. Obviously, 2.0 is better than 1.5.
NODE_ID is the Emulab name for the node in the DB (e.g., "pc1").
MAC is the MAC address for the management interface. Even in the shared
interface case, this should be distinct from the control network interface
MAC (even though they physically share the same Ethernet PHY).
IP is the IPv4 address you assigned to the management interface in the BIOS.
USER is the name of the management user configured via the BIOS. By default
it is probably "root" or "admin".
PASSWD is the management password you set in the BIOS. If AUTHTYPE is "key"
then it is both the password and the encryption key.
For a dedicated interface, you will need to specify additional
switch-related information to tell Emulab where the management interface is
boss> wap management_iface -a AUTHTYPE -t TYPE -s SWITCH-INFO NODE_ID MAC IP USER PASSWD
where SWITCH-INFO is of the form "switch,card,port" where switch is the
name of the switch in the DB nodes table, and card and port indicate
where on the switch the interface is connected.
After running management_iface, you should be able to control power on the
node. A non-destructive test is to run:
boss> wap power on pcN
It should say something to the effect that power is now on. If you get a
timeout or some error message, then something is not configured correctly.
A better test is to run:
boss> wap power cycle pcN
and verify that the node actually gets rebooted. Note that this is not a
graceful shutdown, so make sure the node is currently in a state where you
don't care about the state of any filesystems (e.g., the OS FSes are
mounted read-only, or the node is currently running in an MFS, or is in the
Configuring IPMI serial-over-lan (SOL) console
SOL is only standard as of IPMI 2.0. Older management interfaces may have
proprietary implementations of something akin to SOL, sometimes even called
SOL, but we only support the official 2.0 version.
First, you need to configure console redirection in the BIOS. This can be
seriously confusing as you are configuring a "COM" interface that may or
may not really exist and may or may not have an actual connector.
Hopefully, the vendor docs will tell you what to do, but with a Dell
iDRAC7, we configure serial communication to be "on with console
redirection via COM2" and redirection after boot to "always". For a Dell
BMC, the option may be under "Remote Access", which should be enabled with
serial port number as "COM2 as SOL". We always use a baud rate of 115200.
There may be additional options which route one of the two "standard" COM
ports to the (usually) single physical connector on the back of the
chassis, but you should not need to mess with those.
For Emulab, you will need to set a couple of node or node_type attributes
to ensure that the MFSes and OSes are properly configured to use the
correct console device. We generally configure the attributes at the
node_type level, but if you for example only have a single machine of a
particular type that you want SOL on, you can set it as a node
attribute. The attribute names and values are the same either way. To set
it for all nodes of a particular type, use the Emulab web interface to edit
the appropriate node type.
You will need to create (or change the value of, if it already exists) the
attribute "pxe_boot_path" (type: string) and set the value to
"/tftpboot/pxeboot.emu-sio2" (or "-sio1", "-sio3", "-sio4" depending on
what serial port is used by the management interface).
Then create (or modify) "console_type", another string attribute with the
value "sio2" (or "sio1" ...).
After changing these, you will need to run:
boss> wap dhcpd_makeconf -ir
to ensure that the pxe_boot_path takes effect. Then you will need to reload
any default OS on the nodes in question, e.g.: