|
|
|
This page explains how to get nodes set up and added to the testbed.
|
|
|
|
|
|
|
|
### Step 1 - BIOS setup on the nodes
|
|
|
|
|
|
|
|
First, we need to get some things set up in the nodes' BIOS. For now, just
|
|
|
|
do this on one of the nodes, you'll do the rest later. The following items
|
|
|
|
are necessary, although you may wish to configure other options such as a
|
|
|
|
BIOS password, etc.
|
|
|
|
|
|
|
|
1. **Booting from PXE -** The node must be configured to boot from PXE on
|
|
|
|
its control network interface. This is how the testbed exercises
|
|
|
|
control over what the node will boot. In most BIOSes, this should be
|
|
|
|
as simple as finding the boot order options, and putting PXE on the
|
|
|
|
top. Things can get a bit confusing if you have more than one
|
|
|
|
PXE-capable interface, because often times the BIOS provides no way of
|
|
|
|
distinguishing between them. In this case you'll have to do some
|
|
|
|
trial-and-error to figure out which is which.
|
|
|
|
|
|
|
|
3. **Disable PXE on experimental interfaces -** Nodes will boot much
|
|
|
|
quicker if you disable PXE booting on each other interface, though
|
|
|
|
whichever means provided by the card. On some Intel cards, for
|
|
|
|
example, you may press CTRL-S during the POST to enter a card's
|
|
|
|
configuration screen when prompted.
|
|
|
|
|
|
|
|
3. **Set power-loss behavior -** Many BIOSes have an option about what to
|
|
|
|
do after a power failure (which is what it looks like to the node when
|
|
|
|
it gets power cycled by a power controller). They're usually 'always
|
|
|
|
off', 'always on', and 'last state'. Always on is the best - last
|
|
|
|
state is OK, but if someone does a 'shutdown -h' on the node, you
|
|
|
|
can't bring it back up with power cycling - you have to go punch the
|
|
|
|
power button. Just make sure they're not set to always off...
|
|
|
|
|
|
|
|
### Step 2 - Type information for the nodes
|
|
|
|
|
|
|
|
Unless you're adding some more nodes, identical to the ones you already
|
|
|
|
have, you will need to put some type information about them into the
|
|
|
|
database. You can do this through the web interface: log in and go into
|
|
|
|
'red dot' mode. Now, click on the 'Node Status' link on the menu, and use
|
|
|
|
the 'Create a new type' link. The important items on this page to fill out
|
|
|
|
are: (you can leave defaults for the rest)
|
|
|
|
|
|
|
|
| Item Name | Description |
|
|
|
|
| ---- | ----------- |
|
|
|
|
| Type | We typically name types 'pcXXX', where XXX is a short (a few characters) string describing something unique about the nodes, such as processor speed, chipset, etc. (e.g. 'pc3000' for 3GHz nodes) |
|
|
|
|
| Class | Leave this as 'pc' |
|
|
|
|
| AdminMFS OSID | Select the FREEBSD-MFS image |
|
|
|
|
| BIOS Waittime | Delay (in seconds) for the BIOS to initialize the system |
|
|
|
|
| Bootdisk Unit | If the node's disk is 'hdd' or 'ad4', then you would enter '4' |
|
|
|
|
| Control Network Interface | Name of the control network interface under Linux. Usually, just a concatenation of 'eth' and the Control Network number from 'Control Network' below (e.g. 'eth0') |
|
|
|
|
| Control Network | Interface number (described above) of the control network interface |
|
|
|
|
| Default ImageID | Select the image(s) you want to be loaded by default |
|
|
|
|
| Default OSID | Select the operating system from one of the images you selected above for the default ImageID |
|
|
|
|
| Delay Capacity | How many delay nodes this node can be. For example, nodes with 2 experimental interfaces can be 1 delay node, nodes with 4 experimental interfaces can be 2 delay nodes, etc. If your nodes are not especially "beefy" then you might not want to set this number to the maximum physical limit |
|
|
|
|
| Diskload MFS OSID | Select the FRISBEE-MFS image |
|
|
|
|
| Disk Size | Hard disk size in GB (e.g. '500') |
|
|
|
|
| Disk Type | FreeBSD-style disk name for the primary hard drive. Choices are 'ad' (IDE), 'sd' (SCSI), or 'ar' (IDE RAID) |
|
|
|
|
| Frequency | CPU frequency in MHz (e.g. '2400') |
|
|
|
|
| Max Interfaces | Maximum number of NIC ports (dual-port cards count as 2). Include the Control network interface in this count |
|
|
|
|
| Memory | Amount of RAM in MB (e.g. '4096') |
|
|
|
|
| Power Delay | 60 |
|
|
|
|
| Processor | Class of processor, such as 'Core 2' |
|
|
|
|
|
|
|
|
|
|
|
|
### Step 3 - Bringing up the first node
|
|
|
|
|
|
|
|
We will start by bringing up the first node in the testbed, to make sure
|
|
|
|
that things are working, and so that you can set some initial values. You
|
|
|
|
should already have the *newnode* and other MFSes installed, as described
|
|
|
|
previously.
|
|
|
|
|
|
|
|
What is going to happen as we bring nodes up is that they should PXE boot
|
|
|
|
into the 'newnode' MFS, which is a stripped-down version of FreeBSD that
|
|
|
|
runs out of a memory filesystem. This MFS reports to boss, informing it of
|
|
|
|
its existence and key things such as its MAC addresses. Ensure that you
|
|
|
|
have configured the BIOS as previously described, and fire it up. By the
|
|
|
|
time it's got a FreeBSD login prompt on the console, it should have
|
|
|
|
reported in. This will send mail to the local testbed-ops list.
|
|
|
|
|
|
|
|
Now, let's take a look at the web page where nodes that have checked in and
|
|
|
|
are awaiting creation as 'real' nodes, show up. Log into the web interface
|
|
|
|
and go into 'red dot' mode. Now, go to the 'Add Testbed Nodes'
|
|
|
|
link. Clicking on the numeric ID next to a node will bring up a page with
|
|
|
|
more information about the node, which you can edit. You can select nodes
|
|
|
|
with the checkboxes along the left side - actions taken by the buttons
|
|
|
|
below operate on the selected nodes. Note that nothing on this page asks
|
|
|
|
for confirmation, so be careful where you click.
|
|
|
|
|
|
|
|
On this page, you should now see the first node booted up, which should
|
|
|
|
have received the name 'pc1'. Click on the ID number (which is probably
|
|
|
|
'1') to see more detail. Make sure that the number of interfaces reported
|
|
|
|
is correct. Note that the 'Temporary IP' shown on this page is the dynamic
|
|
|
|
address assigned to the node by DHCP on boss, from the dynamic range you
|
|
|
|
set up back in the definitions file. If you need to SSH into it to check
|
|
|
|
things out before it's really been added to the testbed, use this IP
|
|
|
|
address.
|
|
|
|
|
|
|
|
Next, make sure that the 'Type' column is filled in with the one you
|
|
|
|
entered into the types table earlier. If it isn't, fix that now using the
|
|
|
|
'Set Type' box.
|
|
|
|
|
|
|
|
Next, set the node name to your preferred naming scheme. We strongly
|
|
|
|
suggest leaving it as-is (i.e. using pcXXX to name the nodes), but if you
|
|
|
|
must change it, do so now - nodes you add later will get a name based on
|
|
|
|
this one, with the number and the end incremented. The code that does this
|
|
|
|
guessing supports node naming schemes that end in numbers, or end with '-a'.
|
|
|
|
|
|
|
|
Now, you'll need to set the IP address for this node. Subsequent nodes will
|
|
|
|
have their IP addresses computed from this one. (I.e. if you set pc1 to an
|
|
|
|
address that ends in '.1', pc3 will get address '.3').
|
|
|
|
|
|
|
|
The other thing to check here is to see in what order the interface got
|
|
|
|
detected. Unfortunately, FreeBSD and Linux sometimes detect them in
|
|
|
|
different orders. If you will usually be running Linux on the nodes, you
|
|
|
|
probably want to re-order them to the Linux order so that the database
|
|
|
|
state will make more sense to you. At this point, figure out the mapping
|
|
|
|
from the FreeBSD order to the Linux one, and write that down. (If
|
|
|
|
necessary, you could boot the node up from Knoppix, or some distribution's
|
|
|
|
install CD to determine the Linux order - use MAC addresses to map this to
|
|
|
|
the FreeBSD one).
|
|
|
|
|
|
|
|
If you somehow make a mistake and need to delete a node that has been created:
|
|
|
|
```
|
|
|
|
boss> wap deletenode pc1
|
|
|
|
```
|
|
|
|
|
|
|
|
### Step 4 - Bringing up the rest of the nodes
|
|
|
|
|
|
|
|
Okay, now that you've got the first node up, the rest should be easy. Bring
|
|
|
|
the second node (pc2) up, just like you did the first one. Check to make
|
|
|
|
sure that it got an appropriate name and IP address, extrapolated from the
|
|
|
|
first one. If that works, start bringing the others up in order. It's
|
|
|
|
important to do them in order, because identifying which is which if you do
|
|
|
|
them out of order can be very painful! If there are some nodes you simply
|
|
|
|
cannot bring up, because of bad hardware, etc., write these down, and we
|
|
|
|
will fix things up later.
|
|
|
|
|
|
|
|
**Note:** You defined the size of the dynamic range of IP addresses in the
|
|
|
|
definitions file. If you find that you do not have a large enough range to
|
|
|
|
bring up all of your nodes, continue on with the nodes you have up, and
|
|
|
|
repeat these steps later for the remainder.
|
|
|
|
|
|
|
|
Okay, got all the nodes up? Good. At this point, you can fix things up for
|
|
|
|
any you had to skip, using the 'Add to Node ID suffix' box. If, for
|
|
|
|
example, you could not boot pc10, select all the nodes detected as pc10 and
|
|
|
|
higher, and add 1 to their node numbers. You will then want to use the
|
|
|
|
'Recalculate IPs' button on these nodes to get their IP addresses set
|
|
|
|
appropriately.
|
|
|
|
|
|
|
|
The type for each node is supposed to get detected automatically, but this
|
|
|
|
can be a bit imprecise (i.e., processor speeds are never exactly as
|
|
|
|
advertised - a 3 GHz processor may really be 2.98 GHz.) So, if the nodes
|
|
|
|
did not get their types detected correctly, just select them all, and use
|
|
|
|
the 'set type' button.
|
|
|
|
|
|
|
|
### Step 5 - Figuring out interfaces
|
|
|
|
|
|
|
|
If you found earlier that the FreeBSD and Linux ordering for interfaces was
|
|
|
|
different, we'll fix that up now. Use the boxes right above the 'Re-number
|
|
|
|
interfaces' button to do this. Just leave blank any interface numbers your
|
|
|
|
nodes don't have. For example, if you have two interfaces, and what FreeBSD
|
|
|
|
detects as eth0 is eth1 under Linux, and vice versa for eth1, you'd enter
|
|
|
|
'1' and '0' in the first two boxes. Select all nodes, and hit the
|
|
|
|
're-number' button. Once you have got this sorted out, the 'Control MAC'
|
|
|
|
column should be correct.
|
|
|
|
|
|
|
|
Now, we are going to figure out where the interfaces are plugged in on your
|
|
|
|
switches. You should have already entered your switch information into the
|
|
|
|
database from previous instructions. If they are not already up, enable all
|
|
|
|
of the ports on your experimental network that have experimental interfaces
|
|
|
|
connected to them. Also, in order to work around some strange behavior
|
|
|
|
(possible a bug?), you'll need to place these ports into some VLAN other
|
|
|
|
than VLAN 1. If you just now enabled these interfaces, wait a few minutes
|
|
|
|
to give the switches time to learn the nodes' MAC addresses. Now, select
|
|
|
|
all the nodes, and click the 'search switch ports' button. This will grab
|
|
|
|
the MAC tables from all switches you put into the database, which we will
|
|
|
|
match up with the MACs that the nodes themselves reported. This will take a
|
|
|
|
little while, and it will report any interfaces it failed to find.
|
|
|
|
|
|
|
|
If you did not enter your control network switch into the database, this is
|
|
|
|
okay, but this step will not find any control network interfaces. That's
|
|
|
|
acceptable, but make sure it doesn't complain about any experimental
|
|
|
|
network interfaces.
|
|
|
|
|
|
|
|
### Step 6 - Creating the nodes
|
|
|
|
|
|
|
|
You are finally ready to take these nodes and actually create them! (By
|
|
|
|
this point you should already have at least one disk image from
|
|
|
|
Utah). Select all the nodes, hit 'Create', and wait a while. This will
|
|
|
|
enter all of the nodes into their permanent locations in the database, and
|
|
|
|
will reboot them into a 'full' FreeBSD MFS. It also puts them into the
|
|
|
|
`emulab-ops/hwdown` experiment, to make sure that no experimenters get them
|
|
|
|
in case something went wrong. Inspect a few to make sure they booted
|
|
|
|
correctly. If so, free them from the hwdown experiment with:
|
|
|
|
|
|
|
|
```
|
|
|
|
boss> wap nfree emulab-ops hwdown pc1 pc2 pc3 ...
|
|
|
|
```
|
|
|
|
|
|
|
|
At this point, they should get a disk image loaded and end up in the free pool.
|
|
|
|
|
|
|
|
* [Prev](instal/Memory Filesystems)
|
|
|
|
* [Next](install/Importing OS Images from Utah)
|
|
|
|
* [Home](install/Installing Emulab) |
