Emulab now supports using IPMI (version 1.5 or 2.0), over either a dedicated network connection or a shared (with the control network) connection, for power cycling and the node console. The following describes in broad terms how to setup such a node.
Note that many vendors implement their own management interface and GUI with advanced features like virtual CD drives or ssh authentication, but we only use basic IPMI supported functions. Thus in the text that follows, when we talk about "enabling the IPMI" you are, for example, really enabling the Dell iDRAC and then configuring IPMI parameters.
First you need to configure IPMI in the machine BIOS. This is highly vendor specific, but for a Dell with an iDRAC7 or just BMC, there will be a menu in which you can configure IPv4 parameters. We statically configure the IPs though you probably could use DHCP.
If you have a dedicated management interface, then you can put all of these interfaces in the so-called "hardware control network" switch VLAN where we place all switches and power controllers. Assign it a unique IP on that subnet, and set the mask appropriately. A DNS name is not required.
If the management interface is shared (with one of the regular network interfaces), then make sure it is shared with the Emulab "node control network" interface. In the shared case, we are assuming a fairly recent built-in management controller that supports a distinct IP address for talking to the management controller, ala recent Broadcom and Intel chips. As with the dedicated case, assign a unique IP from the appropriate subnet.
For either case, you will want to also setup authentication. If you are using a shared interface, it is CRITICAL that you do this, as the management interface will be accessible to anyone with an account on the "users" node or with a swapped-in experiment. Emulab supports the use of an IPMI password to provide authentication. IPMI 2.0 also has support for packet encryption, which Emulab optionally supports in a very basic way (i.e., the encryption key is the same as the password). For a shared interface, you will want encryption to protect the IPMI password when it is sent to the management controller. If you have a dedicated interface that is in the hardware control network and thus accessible only from boss, you could get by without authentication, but it is still a good idea to have at least a password. Passwords are set per-node in Emulab, you can use the same password for all nodes.
Note that when using shared interfaces with a node control network that is directly accessible from the Internet, you will want to firewall-off the management IPs from the outside world.
Note that many management interfaces can use of an ssh keypair for authentication which Emulab also supports, but IPMI only supports the password/encryption-key and that is what is covered here.
Most management controllers have a web interface, so once you get the basic IP and authentication info setup correctly, you can use the web interface to configure the rest. If for no other reason, you should try connecting and logging into the web interface as a test of whether the basic network parameters are correctly configured.
On the Emulab side, you will need to install the ipmitool port on boss:
boss> sudo portupgrade -N ipmitool
should do the trick. Or, if you don't have portupgrade installed:
boss> cd /usr/ports/sysutils/ipmitool boss> sudo make install
Configuring IPMI power cycling
We use a common software infrastructure for supporting HP iLo and Dell's DRAC and DMC controllers. To get a node management interface into Emulab DB and usable by the "power" command, you use "management_iface" on boss.
For a shared interface:
boss> wap management_iface -a AUTHTYPE -t TYPE NODE_ID MAC IP USER PASSWD
- AUTHTYPE is either "pswd" to set just a password, or "key" to set both the password and encryption key (for IPMI 2.0 controllers).
- TYPE is either "ipmi15" or "ipmi20" depending on which version of the IPMI protocol is supported. Obviously, 2.0 is better than 1.5.
- NODE_ID is the Emulab name for the node in the DB (e.g., "pc1").
- MAC is the MAC address for the management interface. Even in the shared interface case, this should be distinct from the control network interface MAC (even though they physically share the same Ethernet PHY).
- IP is the IPv4 address you assigned to the management interface in the BIOS.
- USER is the name of the management user configured via the BIOS. By default it is probably "root" or "admin".
- PASSWD is the management password you set in the BIOS. If AUTHTYPE is "key" then it is both the password and the encryption key.
For a dedicated interface, you will need to specify additional switch-related information to tell Emulab where the management interface is connected:
boss> wap management_iface -a AUTHTYPE -t TYPE -s SWITCH-INFO NODE_ID MAC IP USER PASSWD
where SWITCH-INFO is of the form "switch,card,port" where switch is the name of the switch in the DB nodes table, and card and port indicate where on the switch the interface is connected.
After running management_iface, you should be able to control power on the node. A non-destructive test is to run:
boss> wap power on pcN
It should say something to the effect that power is now on. If you get a timeout or some error message, then something is not configured correctly.
A better test is to run:
boss> wap power cycle pcN
and verify that the node actually gets rebooted. Note that this is not a graceful shutdown, so make sure the node is currently in a state where you don't care about the state of any filesystems (e.g., the OS FSes are mounted read-only, or the node is currently running in an MFS, or is in the BIOS).
Configuring IPMI serial-over-lan (SOL) console
SOL is only standard as of IPMI 2.0. Older management interfaces may have proprietary implementations of something akin to SOL, sometimes even called SOL, but we only support the official 2.0 version.
First, you need to configure console redirection in the BIOS. This can be seriously confusing as you are configuring a "COM" interface that may or may not really exist and may or may not have an actual connector. Hopefully, the vendor docs will tell you what to do, but with a Dell iDRAC7, we configure serial communication to be "on with console redirection via COM2" and redirection after boot to "always". For a Dell BMC, the option may be under "Remote Access", which should be enabled with serial port number as "COM2 as SOL". We always use a baud rate of 115200.
There may be additional options which route one of the two "standard" COM ports to the (usually) single physical connector on the back of the chassis, but you should not need to mess with those.
For Emulab, you will need to set a couple of node or node_type attributes to ensure that the MFSes and OSes are properly configured to use the correct console device. We generally configure the attributes at the node_type level, but if you for example only have a single machine of a particular type that you want SOL on, you can set it as a node attribute. The attribute names and values are the same either way. To set it for all nodes of a particular type, use the Emulab web interface to edit the appropriate node type.
You will need to create (or change the value of, if it already exists) the attribute "pxe_boot_path" (type: string) and set the value to "/tftpboot/pxeboot.emu-sio2" (or "-sio1", "-sio3", "-sio4" depending on what serial port is used by the management interface).
Then create (or modify) "console_type", another string attribute with the value "sio2" (or "sio1" ...).
After changing these, you will need to run:
boss> wap dhcpd_makeconf -ir
to ensure that the pxe_boot_path takes effect. Then you will need to reload any default OS on the nodes in question, e.g.:
boss> wap sched_reload -t _node_type_