setup-serial-and-power.txt 7.37 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209
The following is a guide to setting up power control and serial lines.

Overview:

0) Setup capture logging
1) Enter all the power controller's info into the emulab DB 
   and "capture" startup script (if applicable).
2) Enter all the nodes' outlet info into the DB
3) Enter all the nodes' serial line info into the DB and "capture" startup
4) Enter all tipserver machines into the database.
5) Collect interface mapping from nodes

Details:

Steps two and three can be easily scripted (automated) if you have an
online mapping of node to outlet and node to serial line.

Note that your boss node should *NOT* also be a tipserver node! You can
host tiplines on your users node if you like, or on other dedicated
tipserver nodes. Hosting tiplines on boss (and thus exporting them via
capture) is generally a bad idea because of security issues, not to mention
that it just will plain not work.

0) Setup capture logging

Before you can start capture for any power controllers or nodes, you should
setup the logging files.  Capture is hardwired to log into /var/log/tiplogs.
We usually create this as a symlink to /usr/testbed/log/tiplogs (yes, yes,
we should change the path in the source...).  So for each serial line server
you should:

	mkdir -p /usr/testbed/log/tiplogs
	ln -s /usr/testbed/log/tiplogs /var/log/tiplogs
	cp /dev/null /usr/testbed/log/tiplogs/capture.log

To setup logging for the capture processes themselves, add the following
to each /etc/syslog.conf:

   # testbed stuff
   !capture
   *.*              /usr/testbed/log/tiplogs/capture.log

and then HUP syslog.  To ensure the logfile gets "rolled" periodically,
put this in /etc/newsyslog.conf do:

   # testbed stuff
   /usr/testbed/log/tiplogs/capture.log    664  7     1000 *     Z

*** export log filesystem to ops node.  So log dir should really be a
    distinct filesystem, else you export more than you intend!

# rc.conf: added by Emulab setup: allow export of log filesystem
rpcbind_enable="YES"
nfs_server_enable="YES"
nfs_reserved_port_only="YES"
mountd_enable="YES"
mountd_flags="-r -p 900"

# exports: added by Emulab setup: allow export of log filesystem
/var/log/tiplogs ops -ro -alldirs

*** install boss rootkey in tipserv authorized_keys

*** install script(s) to keep groups file in sync

*** add NFS mount to ops fstab

    tipserv:/var/log/tiplogs /var/log/tiplogs nfs ro,-b,-i,nodev,nosuid 0 0

    note that ops-install script assumes that ops is a tipserv machine
    if mounting remote tipserv directory on ops:/var/log/tiplogs, be sure
    to disable newsyslog.conf entry.

    # exports: added by Emulab setup: allow export of log filesystem
/var/log/tiplogs ops -ro -alldirs

*** install script(s) to keep groups file in sync

*** note that ops-install script assumes that ops is a tipserv machine
    if mounting remote tipserv directory on ops:/var/log/tiplogs, be sure
    to disable newsyslog.conf entry.

1) Enter all the power controller's serial line info into the emulab DB
   and add a line to the "capture" startup script.

For each power controller, do the following:

* Create an entry in the "nodes" table.

insert into nodes (node_id, type, phys_nodeid, role)
	values ("power<X>", "<type>", "power<X>", "powerctrl");

Where "<X>" is the power controller designation you pick.  I suggest just 
using a simple number.  You can, of course, change this name if you wish, 
just use the same designation below.  <type> is the type of power 
controller; valid values are "RPC27" and "APC".

If this power controller of type "APC", make sure it has an entry in your 
/ets/hosts file on your boss node, and skip to step 2.

* Create an entry in the "tiplines" table.

insert into tiplines (tipname, node_id, server)
	values ("power<X>", "power<X>", "<serial_server>");

Where "<X>" is the same value as above and <serial_server> is the hostname 
of the machine that has the serial connection to the power controller.

* Create a capture startup line in /usr/local/etc/rc.d/capture.sh on the 
  machine that has the serial connection to "power<X>".

If the file doesn't exist yet, create it, and put

#!/bin/sh

at the top, then change the file's mode to 0755.

Add a line like this to the above file:

/usr/testbed/sbin/capture -T 15 -r -s 9600 power<X> <ttyport> >/dev/null 2>&1 &

Where <X> is as above, and <ttyport> is the tty device file associated 
with this power controller.  To make sure the console log for the power
controller gets "rolled", add the following to /etc/newsyslog.conf:

   /usr/testbed/log/tiplogs/power<X>.log      640  7     1000 *     Z /usr/testbed/log/tiplogs/power<X>.pid

* Disable the status detail and confirmation prompt on the power 
  controller.

Manually execute the command line you added above, then run "console 
power<X>".  You will now have access to this device's serial line.  
Navigate the menus to turn off status reporting and confirmation.

2) Enter all the nodes' outlet info into the DB

For each node, do the following:

* insert row into outlets table

insert into outlets values ("pc<XXX>", "power<X>", <outlet>, NULL);

Where <XXX> is the testbed node unique suffix (e.g., number), <X> is the 
power controller that its connected to, and <outlet> is the outlet number 
on the power controller.

* Try power cycling the node (perhaps wait to do this until you get to 
  the end of step 3)

on your boss node, execute

sudo /usr/testbed/bin/power cycle pc<XXX>

You shouldn't get any errors, and the machine should go through a cold 
reboot cycle.

3) Enter all the nodes' serial line info into the DB and "capture" startup

This process is nearly identical to step one.

For each node, do the following:

* Insert tipline row into DB

insert into tiplines (tipname, node_id, server)
	values ("pc<XXX>", "pc<XXX>", "<serial_server>");

* Create a capture startup line in /usr/local/etc/rc.d/capture.sh on the 
  machine with the serial conection to "pc<XXX>".

If the file doesn't exist, follow the script creation step outlined above 
in step 1.

Open the above file, and add a line like this:

/usr/testbed/sbin/capture -T 15 -r -s 115200 pc<XXX> <ttyport> >/dev/null 2>&1 &

Where <XXX> is as above, and <ttyport> is the tty device file associated
with this pc.  To make sure the console log for the node gets "rolled",
add the following to /etc/newsyslog.conf:

   /usr/testbed/log/tiplogs/pc<XXX>.log      640  7     1000 *     Z /usr/testbed/log/tiplogs/pc<XXX>.pid

* Test the console line

Run "console pc<XXX>" on serial 2.  Hit enter a couple of times, or reboot
(using "power"!) the machine and wait to see if the PXE and MFS kernel 
console messages appear.

4) Enter all tipserver machines into the database.

For each independent tipserver machine you have, do the following in mysql:

insert info tipservers values ("<serial_server>");

5) Collect interface mapping from nodes
   XXXX this should move elsewhere XXXX

Once you have the consoles up and running, it would be a good time to
collect interface mapping information from the nodes.  Unfortunately,
FreeBSD and Linux might number interfaces in different orders and they
do give them different names.  Thus, Emulab requires a mapping from
FreeBSD interface number to Linux interface number.

You'll need to boot one of each different node type into both freebsd and
linux, recording the interface number to MAC address mapping.  Use these
two mappings to correlate the FreeBSD and Linux interface numbers.  You'll 
use this information in conjunction with the instructions for adding nodes 
in setup-nodes.txt