adding_nodes.txt 6.85 KB
Newer Older
1 2 3 4 5 6
This file documents the process of adding a new node to the testbed.

A. Information about the node
-----------------------------

1. MAC address. For each port in the new node you need to find out the
Mike Hibler's avatar
Mike Hibler committed
7 8
   MAC address, and which port (eth0/fxp0) it is in software.  You need
   to know both the Linux (eth) and BSD (fxp,xl,dc,etc.) names.
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

2. Wiring. We need to know which physical port on the back of the
   machine maps to eth0, eth1, etc., and where each port is connected
   to the cisco (get module/port, ie 3/21).

3. Power. Plug it into a power controller, and make note of which one
   it is (name or IP) and which port you plug it into (1-8).

4. Serial line(s). When you plug in the serial lines, make sure which
   ports on the serial expander they are plugged into.

B. If the node is of a new type
-------------------------------

1. You'll need specs on the nodes for the node type table. You'll
   need a name for the type, processor class (ie PIII), speed (MHz),
   RAM size (in MB), Hard disk size (in GB), max # of physical cards
   it holds (including the motherboard as a card if it has blt-in
   ethernet), and the maximum number of ports it has on a card (ie 1,
   2 for dual card, 4 for quad card). You'll also need to give it a
   default OS id (the default OS to boot), which port is the control
   net (ie 4), how long you should make people wait before power cycling
   again, the default image id, delay capacity, control iface (ie
32 33
   eth4), the type of the primary disk ('ad'==IDE,'da'==SCSI') and the
   OS id to boot when it is acting as a delay node.
Mike Hibler's avatar
Mike Hibler committed
34 35 36
   Example:

   insert into node_types
37
   (class,type,proc,speed,RAM,HD,max_interfaces,osid,control_net,imageid,imageable,delay_capacity,virtnode_capacity,control_iface,disktype,bootdisk_unit,delay_osid,jail_osid)
Mike Hibler's avatar
Mike Hibler committed
38
   values
39
   ("pc","pc2000","P4",2000,512,20.0,5,"RHL-STD",0,"emulab-ops-FBSD47+RHL73-STD",1,2,10,"eth0","ad",0,"FBSD-STD","FBSD-STD");
40

Mike Hibler's avatar
Mike Hibler committed
41 42 43
2. There are several scripts that limit searches to certain classes.
   If the new type you have added does not have class "pc", you may need
   to include this new class as appropriate.
44
   Some of the scripts that might need to be updated are:
Mike Hibler's avatar
Mike Hibler committed
45
   /db/avail.in
46
   /db/nfree.in
Mike Hibler's avatar
Mike Hibler committed
47
   /tbsetup/assign_wrapper.in
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62
   /tbsetup/batchexp.in
   /tbsetup/reload_daemon.in
   /tbsetup/exports_setup.in
   /tbsetup/snmpit_lib.pm
   /www/nodecontrol_list.php3
   /www/reserved.php3
   /www/showexp_list.php3
   /www/tutorial/nscommands.html
   /www/updown.php3
   /sql/database-create.sql

C. What to do on boss:
----------------------

1. Insert entries into interfaces table using info from A(1). Try:
Mike Hibler's avatar
Mike Hibler committed
63 64 65
   insert into interfaces (node_id,card,port,MAC,IP,interface_type,iface)
   values
   ("pcN",0,1,"00b0d0f01020",NULL,"BSDTYPE","eth0")  
66 67 68 69 70

2. Insert entries into wires table using info from A(2). Try:
   insert into wires (node_id1,card1,port1,node_id2,card2,port2) values
   ("pcN",0,1,"ciscoX",5,1)

71 72 73 74 75 76 77
   For the control interface do:
   insert into wires (type,node_id1,card1,port1,node_id2,card2,port2)
   values ("Control","pcN",0,1,"ciscoX",5,1)

   Check to make sure your cards and ports match up with what you 
   entered in the interfaces table.

78 79 80 81 82
3. Insert entry into outlets table, using info from A(3). Try:
   insert into outlets (node_id, power_id, outlet) values
   ("pcN","powerX",Y)

4. Add entries to the nodes table for each node. Try:
83
   insert into nodes (node_id,type,phys_nodeid,role,def_boot_osid,priority,op_mode)
Mike Hibler's avatar
Mike Hibler committed
84
   values
85
   ("pcN","pc1u","pcN","testnode","FBSD45-STD",P,'NORMAL')
86

87
   P (priority) is the where it gets printed out. These need to be
88 89 90 91 92
   ascending numbers, and in the right region. See the table.

4a. Add entries into the tiplines table. The "server" field is where
    the actual capture process runs:

93 94
    INSERT INTO tiplines VALUES ('pc1','pc1','users.emulab.net',0,0,'');
    INSERT INTO tiplines VALUES ('pc111','pc111','tipserv1.emulab.net',0,0,'');
95 96 97 98 99 100 101 102 103 104

    You need to add the usual lines in /etc/remote on the machine
    where the capture process runs. In addition, add a line on users
    to that users can use tip to connect to a console on a remote tip
    server. So, on users:

	pc111|tbpc111:dv=/dev/tip/pc111:br#115200:nt:pa=none:

    The device field is ignored, but something must be there. 

105
5. Until you are ready to put it in service, reserve it to an expt,
106 107 108
   either with nalloc or by adding an entry to the reserved table
   directly. You'll probably also want to put its ports in a vlan to
   enable them.
109 110 111

6. Add the node to the system files:
   - DNS: on boss, cd /etc/namedb
Mike Hibler's avatar
Mike Hibler committed
112
	  co -l emulab.net.db.head
113 114
	  add these lines with all the others:
	  pcN    IN      A       155.101.132.N
115 116
                 IN      MX 10   ops
                 IN      MX 20   fast.cs.utah.edu.
117
	  ci -u emulab.net.db.head
118
          cd reverse/
119
	  co -l 155.101.132.db
120 121 122 123
	  in 155.101.132.db, make these changes:
	  update serial number on line 10
	  add entry for node, like this:
	  N      IN      PTR     pcN.emulab.net.
124
	  ci -u 155.101.132.db
125
	  run /usr/testbed/sbin/named_setup to update.
Mike Hibler's avatar
Mike Hibler committed
126 127 128 129 130 131 132 133 134 135 136
   - DHCP: on boss, cd /usr/local/etc/
	   if you added a new node type, then you need to add a line
	   of the form:
	      %%nodetype=<type>
	   (where <type> is the new type is called) to dhcpd.conf.template.
	   Then as root run:
	      dhcpd_makeconf dhcpd.conf.template > Ndhcpd.conf
	   you can diff dhcpd.conf with the new file to verify nothing
           catostrophic happened.  Finally:
	      sudo cp Ndhcpd.conf dhcpd.conf
	      sudo /usr/local/etc/rc.d/2.dhcpd.sh restart 
137
   - tip: on ops or tipserv1, edit /etc/remote
138 139 140 141 142 143
	  add a line like this:
	  pcN:dv=/dev/tip/pcN:br#115200:nt:pa=none:
	  pcN-tty:dv=/dev/cua<port #>:br#115200:nt:pa=none:
	  then do these:
	  sudo touch /var/log/tiplogs/pcN.log
	  sudo touch /var/log/tiplogs/pcN.run
144
   - capture: on ops or tipserv1, edit /usr/site/etc/capture.rc:
145 146 147
	  add a line like this:
	  /usr/site/bin/capture -r -s 115200 pcN tty<port#> >/dev/null 2>&1 &

148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177

D. How to get the first image on it:
------------------------------------

1. If everything is set up right, you can use the magic PXE Flash Floppy 
   to put the right thing on the PXE card. Edit the BIOS to put the
   boot order to Floppy, PXE, Hard Drive, then reboot it.

2. If everything goes right, you should see it PXE boot and find its
   DHCP info, then contact the ProxyDHCP server to get its bootinfo
   data, then it should decide according to that what to boot.

3. If process 3 went okay to that point, do an os_load to try to
   install the standard testbed images for the node.

4. If it doesn't seem to be working just like the others, talk to
   Leigh and Mike.

E. What next
------------

1. Test it out and see if it works well enough to put into service. If
   its ready, release it into the wild with nfree or by deleting its
   entry in the reserved table.

2. Do some more tests to find any obvious problems. Fix them, if any.

3. Sit back and relax for a few minutes until the bug reports start
   flowing in.