|
|
Emulab now supports using IPMI (version 1.5 or 2.0), over either a
|
|
|
dedicated network connection or a shared (with the control network)
|
|
|
connection, for power cycling and the node console. The following describes
|
|
|
in broad terms how to setup such a node.
|
|
|
|
|
|
Note that many vendors implement their own management interface and GUI
|
|
|
with advanced features like virtual CD drives or ssh authentication, but we
|
|
|
only use basic IPMI supported functions. Thus in the text that follows,
|
|
|
when we talk about "enabling the IPMI" you are, for example, really
|
|
|
enabling the Dell iDRAC and then configuring IPMI parameters.
|
|
|
|
|
|
### Enable IPMI
|
|
|
|
|
|
First you need to configure IPMI in the machine BIOS. This is highly vendor
|
|
|
specific, but for a Dell with an iDRAC7 or just BMC, there will be a menu
|
|
|
in which you can configure IPv4 parameters. We statically configure the IPs
|
|
|
though you probably could use DHCP.
|
|
|
|
|
|
If you have a dedicated management interface, then you can put all of these
|
|
|
interfaces in the so-called "hardware control network" switch VLAN where we
|
|
|
place all switches and power controllers. Assign it a unique IP on that
|
|
|
subnet, and set the mask appropriately. A DNS name is not required.
|
|
|
|
|
|
If the management interface is shared (with one of the regular network
|
|
|
interfaces), then make sure it is shared with the Emulab "node control
|
|
|
network" interface. In the shared case, we are assuming a fairly recent
|
|
|
built-in management controller that supports a distinct IP address for
|
|
|
talking to the management controller, ala recent Broadcom and Intel
|
|
|
chips. As with the dedicated case, assign a unique IP from the appropriate
|
|
|
subnet.
|
|
|
|
|
|
For either case, you will want to also setup authentication. If you are
|
|
|
using a shared interface, it is **CRITICAL** that you do this, as the
|
|
|
management interface will be accessible to anyone with an account on the
|
|
|
"users" node or with a swapped-in experiment. Emulab supports the use of an
|
|
|
IPMI password to provide authentication. IPMI 2.0 also has support for
|
|
|
packet encryption, which Emulab optionally supports in a very basic way
|
|
|
(i.e., the encryption key is the same as the password). For a shared
|
|
|
interface, you will want encryption to protect the IPMI password when it is
|
|
|
sent to the management controller. If you have a dedicated interface that
|
|
|
is in the hardware control network and thus accessible only from boss, you
|
|
|
could get by without authentication, but it is still a good idea to have at
|
|
|
least a password. Passwords are set per-node in Emulab, you can use the
|
|
|
same password for all nodes.
|
|
|
|
|
|
Note that when using shared interfaces with a node control network that is
|
|
|
directly accessible from the Internet, you will want to firewall-off the
|
|
|
management IPs from the outside world.
|
|
|
|
|
|
Note that many management interfaces can use of an ssh keypair for
|
|
|
authentication which Emulab also supports, but IPMI only supports the
|
|
|
password/encryption-key and that is what is covered here.
|
|
|
|
|
|
Most management controllers have a web interface, so once you get the basic
|
|
|
IP and authentication info setup correctly, you can use the web interface
|
|
|
to configure the rest. If for no other reason, you should try connecting
|
|
|
and logging into the web interface as a test of whether the basic network
|
|
|
parameters are correctly configured.
|
|
|
|
|
|
On the Emulab side, you will need to install the ipmitool port on boss:
|
|
|
```
|
|
|
boss> sudo portupgrade -N ipmitool
|
|
|
```
|
|
|
should do the trick. Or, if you don't have portupgrade installed:
|
|
|
```
|
|
|
boss> cd /usr/ports/sysutils/ipmitool
|
|
|
boss> sudo make install
|
|
|
```
|
|
|
|
|
|
### Configuring IPMI power cycling
|
|
|
|
|
|
We use a common software infrastructure for supporting HP iLo and Dell's
|
|
|
DRAC and DMC controllers. To get a node management interface into Emulab DB
|
|
|
and usable by the "power" command, you use "management_iface" on boss.
|
|
|
|
|
|
For a shared interface:
|
|
|
```
|
|
|
boss> wap management_iface -a AUTHTYPE -t TYPE NODE_ID MAC IP USER PASSWD
|
|
|
```
|
|
|
|
|
|
* AUTHTYPE is either "pswd" to set just a password, or "key" to set both the
|
|
|
password and encryption key (for IPMI 2.0 controllers).
|
|
|
* TYPE is either "ipmi15" or "ipmi20" depending on which version of the IPMI
|
|
|
protocol is supported. Obviously, 2.0 is better than 1.5.
|
|
|
* NODE_ID is the Emulab name for the node in the DB (e.g., "pc1").
|
|
|
* MAC is the MAC address for the management interface. Even in the shared
|
|
|
interface case, this should be distinct from the control network interface
|
|
|
MAC (even though they physically share the same Ethernet PHY).
|
|
|
* IP is the IPv4 address you assigned to the management interface in the BIOS.
|
|
|
* USER is the name of the management user configured via the BIOS. By default
|
|
|
it is probably "root" or "admin".
|
|
|
* PASSWD is the management password you set in the BIOS. If AUTHTYPE is "key"
|
|
|
then it is both the password and the encryption key.
|
|
|
|
|
|
For a dedicated interface, you will need to specify additional
|
|
|
switch-related information to tell Emulab where the management interface is
|
|
|
connected:
|
|
|
```
|
|
|
boss> wap management_iface -a AUTHTYPE -t TYPE -s SWITCH-INFO NODE_ID MAC IP USER PASSWD
|
|
|
```
|
|
|
|
|
|
where SWITCH-INFO is of the form "switch,card,port" where _switch_ is the
|
|
|
name of the switch in the DB nodes table, and _card_ and _port_ indicate
|
|
|
where on the switch the interface is connected.
|
|
|
|
|
|
After running management_iface, you should be able to control power on the
|
|
|
node. A non-destructive test is to run:
|
|
|
```
|
|
|
boss> wap power on pcN
|
|
|
```
|
|
|
|
|
|
It should say something to the effect that power is now on. If you get a
|
|
|
timeout or some error message, then something is not configured correctly.
|
|
|
|
|
|
A better test is to run:
|
|
|
```
|
|
|
boss> wap power cycle pcN
|
|
|
```
|
|
|
|
|
|
and verify that the node actually gets rebooted. Note that this is *not* a
|
|
|
graceful shutdown, so make sure the node is currently in a state where you
|
|
|
don't care about the state of any filesystems (e.g., the OS FSes are
|
|
|
mounted read-only, or the node is currently running in an MFS, or is in the
|
|
|
BIOS).
|
|
|
|
|
|
### Configuring IPMI serial-over-lan (SOL) console
|
|
|
|
|
|
SOL is only standard as of IPMI 2.0. Older management interfaces may have
|
|
|
proprietary implementations of something akin to SOL, sometimes even called
|
|
|
SOL, but we only support the official 2.0 version.
|
|
|
|
|
|
First, you need to configure console redirection in the BIOS. This can be
|
|
|
seriously confusing as you are configuring a "COM" interface that may or
|
|
|
may not really exist and may or may not have an actual connector.
|
|
|
Hopefully, the vendor docs will tell you what to do, but with a Dell
|
|
|
iDRAC7, we configure serial communication to be "on with console
|
|
|
redirection via COM2" and redirection after boot to "always". For a Dell
|
|
|
BMC, the option may be under "Remote Access", which should be enabled with
|
|
|
serial port number as "COM2 as SOL". We always use a baud rate of 115200.
|
|
|
|
|
|
There may be additional options which route one of the two "standard" COM
|
|
|
ports to the (usually) single physical connector on the back of the
|
|
|
chassis, but you should not need to mess with those.
|
|
|
|
|
|
For Emulab, you will need to set a couple of node or node_type attributes
|
|
|
to ensure that the MFSes and OSes are properly configured to use the
|
|
|
correct console device. We generally configure the attributes at the
|
|
|
node_type level, but if you for example only have a single machine of a
|
|
|
particular type that you want SOL on, you can set it as a node
|
|
|
attribute. The attribute names and values are the same either way. To set
|
|
|
it for all nodes of a particular type, use the Emulab web interface to edit
|
|
|
the appropriate node type.
|
|
|
|
|
|
You will need to create (or change the value of, if it already exists) the
|
|
|
attribute "pxe_boot_path" (type: string) and set the value to
|
|
|
"/tftpboot/pxeboot.emu-sio2" (or "-sio1", "-sio3", "-sio4" depending on
|
|
|
what serial port is used by the management interface).
|
|
|
|
|
|
Then create (or modify) "console_type", another string attribute with the
|
|
|
value "sio2" (or "sio1" ...).
|
|
|
|
|
|
After changing these, you will need to run:
|
|
|
```
|
|
|
boss> wap dhcpd_makeconf -ir
|
|
|
```
|
|
|
|
|
|
to ensure that the pxe_boot_path takes effect. Then you will need to reload
|
|
|
any default OS on the nodes in question, e.g.:
|
|
|
```
|
|
|
boss> wap sched_reload -t _node_type_
|
|
|
```
|
|
|
|