setup-db.txt 14.9 KB
Newer Older
1 2 3 4 5 6
#
# EMULAB-COPYRIGHT
# Copyright (c) 2002-2005 University of Utah and the Flux Group.
# All rights reserved.
#

7
#####
8
##### Setting up tbdb for a new boss node
9 10 11 12
#####

Note: we are working on better automating many of the procedures in this
document - for now, though, a few of them are still manual.
13

14 15 16
Note: steps labeled "Local Only" are only required when setting up a testbed
with local nodes - they can be skipped in a widearea-only testbed.

17 18 19 20
##### Step 1 - Setup users, projects, and experiments

In order to proceed, you should have the following working (from the boss and
ops setup documentation):
21 22 23

   NFS mounts between boss and ops
   Root ssh keys (so that root on boss can ssh to ops without a password)
24
   The web interface
25

Leigh Stoller's avatar
Leigh Stoller committed
26 27 28 29 30 31 32 33 34
Make sure you can log into the web interface using the 'elabman' account.
The password for the elabman account is the same as the root password on
your boss node (see, we told you to remember it!).

This account is created as a testbed administrator, but there is one thing
you will need to do in order to use your admin powers. For the same reason
that you use 'su' and/or 'sudo' on your UNIX boxes instead of logging in as
root, you must explicitly enable admin privileges after you log in. When
logged in as a user who is allowed to become an admin, you will see a green
35
dot on the left side of the header above the main page content. The green
Leigh Stoller's avatar
Leigh Stoller committed
36 37
dot means that although you are allowed admin powers, they are currently
turned off, and you see the same web pages that a regular user sees, and
38
can use the same actions. If you click on the dot, it will turn red, and you
Leigh Stoller's avatar
Leigh Stoller committed
39
will have full administrator privileges; we call this 'going red dot'.
40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65
If you click on the dot again, it will go back to green, thus you can
easily flip back and forth between normal privs and admin privs. Note
that most of the procedures in this file require you to be in red dot mode.

Now, we will use the elabman user to bootstrap your first real account and
project. Note that while you will use the elabman account to do this, the
elabman account should not be considered a real account; it is intended to
help bootstrap only and as such, does not have the power to perform many
actions that are required later (such as adding nodes to the testbed).

Login as user 'elabman' if you have not already done so. Go into 'red dot'
mode by clicking on the green dot. You should see the "Start New Project"
page.  Fill in your own information in the 'Project Head Information'
section.  It is important that you provide a working email address! Select
your initial Project Name in the 'Project Information' section (we call
ours 'testbed', but you can call yours whatever you call your project or
research group).  Also specify a *working* URL (it is required) for the
project. Submit the form using the Submit button at the bottom of the page.

The web interface will grind along for a minute or so. DO NOT CLICK THE
STOP BUTTON! When it is done, you will see a message that invites you to
login as the user you just created. Do this now so that you can continue
with setting up your testbed. Note that the elabman account has been
deactivated during this process to avoid problems later on (and potential
security breaches). 

66
Before we continue, lets explain a few more important items:
67 68 69 70 71 72 73 74 75 76

* Project Membership: In addition to the project you just created, you have
  automatically been added to the "emulab-ops" project with trust value
  "group_root". This allows you to approve new members to that project as
  well as your own project.

* Admin Mode: Your new account has been given "administrator" mode, as
  described above. To change that value for other users after their
  accounts are created, you can do this on boss:

Leigh Stoller's avatar
Leigh Stoller committed
77
	echo 'update users set admin=1 where uid="<username>"' | mysql tbdb
78

79 80 81 82 83 84
* Shell on Boss: Give yourself the special ability to login to boss;
  in contrast, most (normal) users have a restricted shell on boss,
  and are not allowed to log in using a password. Login to boss as root,
  and edit the password file using the 'vipw' command (FreeBSD requires
  some special processing on the password file after editing, which vipw
  does.). Give yourself a real shell (say, /bin/csh) and then exit the
85 86 87 88 89
  editor. Then give yourself a password (in general, it is safer to have a
  different password on boss then on ops!). Use this command:

	passwd <your username>

90
  NOTE: See doc/shelloboss.txt for important security issues w.r.t. giving
91 92 93 94 95
  real shells on boss. Before you give a real shell to someone, it is a
  good idea for them to read this file!

* Now logout and back in as yourself. In general, it is safer and better to
  not do things as root. In fact, many testbed programs will complain if
96
  you invoke them as root because it makes accounting and auditing more
97 98 99 100
  difficult.

* Unix Group Membership: The Emulab account system manages both the
  password file and the group file (/etc/group) on both boss and ops. If
101
  you edit them directly, those changes will likely be lost. If you want to be
102 103 104 105 106 107 108 109 110 111 112
  a member of any UNIX groups on boss, use our 'unixgroups' command. For
  example, to add yourself to the "operator" group, do this on boss (as
  yourself, not root):

	withadminprivs unixgroups -a <username> operator

  NOTE: Your initial account created above was already placed in the wheel
  and tbadmin groups.

  NOTE: Just as you need to go 'red dot' to use admin privileges on the web
  interface, you must also explicitly enable them on the command line. To
113 114
  do this, prefix the command you want to run with 'withadminprivs',
  which can be abbreviated as 'wap'.
115

116
* Set your path: withadminprivs and many other admin-type commands live in
117 118
  /usr/testbed/sbin - you'll want to put this and /usr/testbed/bin in your
  $PATH.
119 120 121 122

Others at your site can now apply to join your project, or start their own.

##### Step 2 - Setup web sql editor
123 124 125 126 127 128 129 130 131 132 133 134 135 136

Several of the steps below require you to add data to the database by hand. If
you're comfortable with SQL, you can do this directly with SQL queries, or you
can use the generic web-based SQL table editor provided with the testbed
software. If you plan to use the former method, you can skip this step.

********************************** WARNING **********************************
* Many tables depend on data in other tables, or depend on programs running *
* to effect a change. Thus, you should not edit tables other than the ones  *
* described in this document.                                               *
* You have been warned......                                                *
********************************** WARNING **********************************

First, you'll want to protect the webdb script from outside browsers. Because
137
of its flexibility, it would be quite dangerous if it were broken into. So, we
138
add an additional layer of protection by limiting the IP addresses it may be
Leigh Stoller's avatar
Leigh Stoller committed
139 140 141 142 143 144 145 146 147 148
used from. Open your httpd.conf file (located in /usr/local/etc/apache)
and find the 'Directory' directives. Add a section such as this:

	<Directory /usr/testbed/www/webdb>
		AllowOverride None
		order deny,allow
		deny from all
		allow from 155.99.212.
	</Directory>
	
149
If you installed the testbed tree somewhere other than /usr/testbed, fix the
150
directory. Change the 'allow from' line to match your IP subnet (note the '.'
151
on the end of the address, to match the entire subnet). You can have as
Leigh Stoller's avatar
Leigh Stoller committed
152 153 154 155
many 'allow' lines as you want. Restart apache:

	sudo /usr/local/etc/rc.d/apache.sh stop
	sudo /usr/local/etc/rc.d/apache.sh start
156 157

Next, you'll need to specify which users have the ability to edit the database.
158 159
This is done with the 'dbedit' column in the users table. You can turn on
a user's dbedit bit like so:
160

Leigh Stoller's avatar
Leigh Stoller committed
161
	echo 'update users set dbedit=1 where uid="<username>"' | mysql tbdb
162

163
##### Step 3 - Setup switches
164
##### Local Only
165 166 167

1) Create node types for switches

Leigh Stoller's avatar
Leigh Stoller committed
168 169 170 171 172 173
Add entries to the node_types table for each type of switch you'll be
using. You can do this by talking to mysql directly, but is more easily
accomplished using the web interface. In 'red dot' mode, go to:

	https://<yourbossnode>/editnodetype.php3?new_type=1

174 175 176 177 178 179 180 181 182 183 184 185 186
These are the switch types currently supported:
    cisco{65xx,40xx,45xx,29xx,55xx}
        For example, if you had a 6509, you'd enter 'cisco6509'
        If your switch runs IOS (instead of CatOS), append '-ios' to the
        type.
    intel
        The supported type is the 510T (but just put 'intel' in the type field)
    nortel1100, nortel5510
    foundry1500, foundry9604
(Note: Case sensitive!)

Set the "class" to 'switch' and set "processor" to whatever you used for the
"type" field.
Leigh Stoller's avatar
Leigh Stoller committed
187 188 189 190 191

Most of the other columns are not important for switches (so you can set
them to 0), but putting in "Max Interfaces" (if the switch is expandable)
can be useful for your own information.

192

193
2a) Create interface types for switch interconnects (if any)
194

195 196 197
If you'll be connecting the experimental switches together, you'll add
interface types for the ports you'll be using to link them together.  These go
into the interface_types table. Make up something descriptive for the 'type'
Leigh Stoller's avatar
Leigh Stoller committed
198 199 200 201 202 203 204 205
column (no spaces). Make sure to set the max_speed (in Kbps) and full_duplex
(1 for full duplex, 0 for half duplex) columns correctly. As an example:

	insert into interface_types set
	    type='cisco_gigE',max_speed=1000000,full_duplex=1,
	    manufacturuer='Cisco',model='WS-X6316-GE-TX',ports=16,
	    connector='RJ45';

206

207 208 209 210 211 212 213
2b) Create interface capabilities entries for switch interconnects (if any)
    
If you added any entries to interface_types in the previous step, you also
need to add entries to the interface_capabilities table. Using the same
type name you used above, add a capkey for "protocols", usually set to
"ethernet" and another capkey for "ethernet_defspeed" to the same value you
used in the interface_types table. Do this for each type you entered in
Leigh Stoller's avatar
Leigh Stoller committed
214 215
step 2a above. For example:

216 217
	insert into interface_capabilities set
	    type='cisco_gigE',capkey='protocols',capval='ethernet';
Leigh Stoller's avatar
Leigh Stoller committed
218 219
	insert into interface_capabilities set
	    type='cisco_gigE',capkey='ethernet_defspeed',capval='1000000';
220

221
3) Create switches in nodes table
222 223

Next, add the switches to the nodes table. The only necessary fields here are
224 225
node_id (choose one appropriate to the switch), type (use one of the ones you
created earlier,) and role. Role, for switches, should be either 'testswitch'
226
or 'controlswitch', depending on whether you're using it for the experimental
Leigh Stoller's avatar
Leigh Stoller committed
227 228 229 230 231
or control network. If it's used for both, call it a 'testswitch'. Example:

	insert into nodes set
	    node_id='cisco1',phys_nodeid='cisco1',
	    type='cisco6509',role='testswitch';
232

233 234
The node_id you select must resolve in DNS and/or boss's /etc/host file - we'll
use this name to contact the switch for SNMP. Ie., you must be able to ping the
Leigh Stoller's avatar
Leigh Stoller committed
235
name you select as the switch's node_id. 
236

237
4) Add state for switch interconnects
238

239 240 241 242 243
If you'll be connecting the experimental switches together, add interfaces to
each of them.  Use the interface types you created above for them, and be sure
to get the correct card and port numbers. In modular switches, such as the
Cisco Cat6500 series, use the switch's 'native' module and port numbers.  For
switches that do not have expansion modules, the card number is assumed to be
244 245
0. Make sure to get the current_speed (this time, in Mbps, and note, this is
an enumerated type, not an integer, so you need to put quotes around the
246 247
speed) and duplex correct. You'll also need to set 'iface' to some string
representing the port - I tend to use 'module/port' (ie. '1/1').
248 249 250 251

Now, go into the wires table and add wires connecting the switch interfaces you
just created. Make sure to set the type to 'Trunk'. In this case, it doesn't
matter which switch you use as node 1 (the node_id1, card1, port1 columns), and
252 253
which you use as node 2. If you have a non-modular switch, all ports are
considered to be on card 0.
254 255 256 257 258 259

7) Create switch stacks

Into the switch_stack_types table, create entries for each stack in your
network. A stack is set of switches that share common VLANs. Usually, your
experimental switches will be one stack, and the control network another. You
260 261 262 263 264 265
should create a stack even if there is only one switch in it. The control
stack should be named 'Control', and the experimental stack 'Experiment' (make
sure to get the capitalization right). The 'leader' field should be set to the
node_id of the master switch in the stack.  The stack_type column is used by
snmpit to determine which module to use to talk to the stack. The currently
supported values are 'cisco' and 'intel'. Making a control network stack is
266
optional. There are a few columns in this table that you will need to set:
Leigh Stoller's avatar
Leigh Stoller committed
267

268 269 270 271 272 273 274 275 276 277 278
supports_private: (Cisco only) This switch can make private VLANs - probably
    best to leave it as 0
single_domain: (Cisco only) Means that all switches in the stack use VTP to
    maintain a common set of VLANs. This means that we will only create VLANs
    on the stack leader - if set to 0, we create VLANs on each switch
    individually
snmp_community: The SNMP community string we use for read-write access to the
    switch. Most switches use 'private' by default.
min_vlan: The smallest VLAN number that the testbed software will use. If left
    NULL, defaults to 2.
max_vlan: Ditto, but the biggest number. Defaults to 1000.
279 280 281

Finally, add switches to these stacks by putting entries in the switch_stacks
table, listing the node_id of each switch and the stack_id of the stack it
282 283
belongs to. You can leave the 'is_primary' column of these rows with its default
value (1).
284

285 286 287 288 289 290
After getting this set up, run 'snmpit -l', and make sure it doesn't give you
any error messages. If it tells you that your type of device isn't supported,
ask Utah - we have a list of supported devices in snmpit, but they are just
the ones we've tested. Hopefully, yours will work too, we just haven't tried
it.

291
##### Step 4 - Setup control hardware
292
##### Local Only
293

294
1) Power controllers
295 296 297 298 299

The only two supported power controllers at this time are the APC AP9210 and
the BayTech RPC27. Theoretically, other SNMP and serial-controlled power
controllers should not be hard to support.

Leigh Stoller's avatar
Leigh Stoller committed
300
If your power controllers are controlled via IP and SNMP, add a node type andor
301 302 303 304 305 306 307 308
interface type for them. Then, give them an entries in the nodes, interfaces,
and wires tables. Make sure to get the IP address correct in the interfaces
table. The role listed in the nodes table should be 'powerctrl'.

If you have serial power controllers, just create entries node_types and nodes
- no interfaces or wires. Again, the role in the nodes table should be
'powerctrl'.

309
##### Step 5 - Images and OSids
310 311 312 313

These will depend on the image(s) and any OSKit kernels you've recieved, or
built yourself. Since you're probably using disk images from Utah, the best
thing to do is ask us for the appropriate lines for the os_info and images
314 315 316 317 318 319 320 321
tables. Make sure to get the PXEBOOT, PXEFRISBEE, and PXEFBSD OSIDs, which are
required to load disk images.  For a widearea-only testbed, no images entries
in the images table a required, but OSIDs are still useful, to determine what a
node's capabilities are. 

##### Step 6 - Setup nodes

To add nodes to the testbed, see setup-nodes.txt .