UPDATING 19.5 KB
Newer Older
1
2
3
4
5
6
7
8
Information necessary for keeping boss and ops node up-to-date. This
file contains updates that must be made outside of the database and
the intall tree (ie. /usr/testbed).

This file is in the same format at the FreeBSD UPDATING file, whis is
to say, in reverse chronological order, with the date of the change
in YYYYMMDD format.

9
10
11
12
13
14
15
16
17
18
19
20
After each date, you will find one of the following, which tells you
when the action needs to be performed.

Before build - before doing a 'gmake' in your testbed object tree on
	boss and/or ops	
After install - after doing a 'gmake boss-install' on boss or
    'gmake ops-install' on ops

Note that some instructions may have steps that need to occur at a few
different points in the install process - these are marked with the
earliest time one of the steps needs to occur.

21
22
20060606:

Leigh B. Stoller's avatar
Leigh B. Stoller committed
23
24
25
26
27
28
      Be sure to firewall the mysql port (3306) at your router so that
      outside networks cannot access that port on your ops node (we now
      run mysqld on ops).

20060525:  Do this before installing tbsetup/libtblog.pm.in rev 2.14
      It would be best if your Emulab was down during this change.
29

30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
      No schema changes, but due to a bug in libtblog.pm.in the uid &
      exptidx columns got swapped for new values inserted into the
      session_info table after the 4.56 schema change.  This is fixed
      in revision number 2.14 of libtblog.pm.in.

      The values of session_info are currently not used by any testbed
      code.  However, it is still a good idea to fix the existing data
      in the database.  To do so determine the "session" where the
      values got swapped.  For emulab.net that was 9359824:

      +---------+-------+---------+
      | session | uid   | exptidx |
      +---------+-------+---------+
      | 9359747 | 31359 | 24893   |
      | 9359824 | 24893 | 31359   |
      +---------+-------+---------+

      Once this is determined, lock the session_info table, swap
      the columns, install the fix, than unlock the table:

        lock tables session_info write;
      
        create temporary table session_info_wk as select * from session_info
          where session >=SESSION;
        replace into session_info select session,exptidx,uid 
          from session_info_wk;
        drop table session_info_wk;

        [Now install the new code]
  
        unlock tables;

      Where SESSION is the session number where the values where first 
      swapped, 9359824, for emulab.net.


66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
20060322: Before installing tbsetup/ns2ir/sim.tcl.in rev 1.109 (3/22/06).

	This revision introduces a new features/desire pair called
	"firewallable".  All nodes in a firewalled experiment require
	this feature from a node.  So you will need to do:

        insert into node_type_features values ('TYPE','firewallable',0);

	for every node type 'TYPE' which can participate in a firewalled
	experiment.  The current requirement for a 'firewallable' node
	is that it can be powered off (not just power cycled).  This is
	required for proper post-swapout cleaning of nodes.  Another
	attribute to consider is whether you can prevent a node from
	booting anywhere other than the network.  If you have nodes that
	can timeout from PXE boot and that will fall back to booting from
	the hard drive, you may want to NOT make those nodes 'firewallable'.

83
84
85
86
87
88
89
90
91
92
93
94
95
96
20051212: After DB schema is updated, and updated software installed.

         These changes are for shifting from SSH protocol 1 keys to
         protocol 2 keys.

         For users:

	        wap /usr/testbed/sbin/genpubkeys

         this will generate new Protocol 2 (rsa) keys for all users,
         and regen their authorized_keys file. Note that existing sites
         will need to retain protocol 1 keys for users because of old
         images.

Kirk Webb's avatar
   
Kirk Webb committed
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
         For boss and ops:

         You must setup a new passphraseless keypair for root on boss.
         To do this, simply run the following command on boss:

                sudo /usr/bin/ssh-keygen -t rsa -N ""

         Next, append the contents of /root/.ssh/id_rsa.pub on boss to
         the /root/.ssh/authorized_keys file on both your ops and boss
         nodes.  Remove any ssh version 1 keys that appear in either
         of these authorized_keys files.  Also, remove the
         authorized_keys2 file (if it exists) from the /root/.ssh
         directory on boss and ops.  Next, change the
         /etc/ssh/sshd_config (sshd config) file on both boss and ops
         to try ssh protocol 2 first: Edit the line that reads
Kirk Webb's avatar
   
Kirk Webb committed
112
113
114
         "Protocol 1,2" to be "Procotol 2,1".  Finally, be sure to
         remove any "Protocol" directives from /root/.ssh/config on
         boss and ops.
Kirk Webb's avatar
   
Kirk Webb committed
115

Kirk Webb's avatar
   
Kirk Webb committed
116
117
118
119
120
121
122
123
         In your database:

         You'll need to update the key stored in the node/ssh_pubkey
         variable to contain the public half of the new v2 rsa key you
         created for root on boss.  This key will be automatically
         distributed to the nodes as they boot up when using images
         dating from late 2004 onward.

Kirk Webb's avatar
   
Kirk Webb committed
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
         For your default disk images:

         Add the contents of /root/.ssh/id_rsa.pub on your boss node
         to the /root/.ssh/authorized_keys file in each of your
         default disk images.  Remove any version 1 ssh keys that
         remain in authorized_keys.  Also, remove any authorized_keys2
         file that might exist in the /root/.ssh directory.  Next,
         edit the /etc/ssh/sshd_config file, changing the "Procotol
         1,2" line to "Protocol 2,1".  Re-create the modified images
         (of course).

         Note on user disk images:

         User images may still require access via ssh version 1, so
         don't delete the ssh version 1 key on your boss node
         (/root/.ssh/identity[.pub]).  The ssh wrapper command (sshtb)
         has been modified to try authenticating with both ssh protocols
         to preserve backward compatibility with existing user images.

143
144
145
146
147
148
149

20051212: After DB schema is updated, and updated software installed.

          Fix an error in the statistics gathering code, that was
          causing duplicate swapin records in some case. From the source
          directory:

Leigh B. Stoller's avatar
Leigh B. Stoller committed
150
		perl sql/fixstats-swapin.pl | mysql tbdb
151
152


153
154
155
156
157
158
159
160
20050818: Anytime after the DB schema is updated

	Load the initial contents of the new knowledge_base_entries table
        from sql/knowlbase-create.sql

		cat sql/knowlbase-create.sql | mysql tbdb


161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
20050707: Before build

	Decide if you want CVS support turned on. This is some simple
        support for per-project CVS trees, stored as /proj/$pid/CVS.
        The main thing is that you can use the cvsweb interface. If you
        want it, in your defs file:

		CVSSUPPORT=1

	After your build and install:

		sudo /usr/testbed/sbin/cvsinit

        which creates the initial CVS trees in all of the existing project
        trees.

Leigh B. Stoller's avatar
Leigh B. Stoller committed
177
178
179
180
181
20050205: Before build/config

	* Install the rsync port from /usr/ports/net/rsync


182
183
184
185
186
187
20041108: Anytime

	* Update sql/database-fill-suplement.sql stuff.

		cat sql/database-fill-suplement.sql | mysql tbdb

188
189
190
191
192
193
194
195
196
197
198
20041108: Before build
	We've fixed a bug in the elvin libraries that underlie our
	event system. You'll want to install the new one from the
	port. You need to do this on both boss and ops. Copy the
	contents of install/ports/libelvin to /usr/ports/misc/libelvin.
	Then, change to that directory and run 'make deinstall reinstall'
	to get the new version of the port installed. Now, you'll need to
	do a clean rebuild of everything in the testbed source tree, by
	doing a 'gmake clean' before the usual 'gmake ops-install' and
	'gmake boss-install'.

199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
20041104: Before build
	A new directory, pxe/freebsd, contains the diffs necessary
	to build your own pxeboot program more easily.  The entry
	for 20040112 below instructs you to "Rebuild pxeboot kernel
	from source".  Now you can actually do that!  See the README
	file in pxe/freebsd.  You should rebuild your pxeboot program
	to take advantage of a couple of new features: the ability
	to specify an explicit machine from which to TFTP load a file,
	and the ability to specify command lines to Linux (actually,
	to LILO).

	You will need to update your frisbee/freebsd MFSes to take
	full advantage of recent changes.  A new file, doc/update-mfs.txt,
	describes the process.  Read it. Believe it. Live by it.

214
20041029: Before build
215
216
217
218
219
220
221
222
223
224
225
	The ulsshxmlrpcpp package was updated to version 0.1.1:

	     http://boss.emulab.net/downloads/ulsshxmlrpcpp-0.1.1.tar.gz

             cd /someplace
	     tar zxf ulsshxmlrpcpp-0.1.1.tar.gz
	     cd ulsshxmlrpcpp-0.1.1
	     configure
	     gmake
	     sudo gmake install

226
227
228
229
230
231
232
233
234
235
236
237
238
20041027: After install (be sure to update sitevars):

	A new sitevariable was added to hold root's ssh public key.
	When nodes boot using the latest image, the public key is
	handed across via tmcd and statshed in /root/.ssh. This
	localizes the image (wrt the root pub key) to the local
	testbed.

	Using the web interface in red dot mode, go to the Edit Site
	Variables link and store boss:/root/.ssh/identity.pub into the
	"node/ssh_pubkey" site variable. Make sure there are no
	newlines when you paste it in.

239
20041025: After install
240
241
242

	Run sql/fixstats_swapmod.pl to fix up stats records.

243
244
		sql/fixstats_swapmod.pl | mysql tbdb

245
246
247
248
249
	Run "/usr/testbed/etc/rc.d/and start" as root on ops to start
	up the auto nice daemon.  This daemon is not required,
	however, it can save you from greedy users that consume too
	much CPU and cause experiment swapin to fail.

250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
20040930: Before build

	You need to update your defs-* file to include some new configure
	time variables.  These are:

	SSLCERT_COUNTRY
	SSLCERT_STATE
	SSLCERT_LOCALITY
	SSLCERT_ORGNAME
	TESTBED_NETWORK
	TESTBED_NETMASK
	BOSSNODE_IP
	USERNODE_IP
	CONTROL_ROUTER_IP
	CONTROL_NETWORK
	CONTROL_NETMASK
	PRIVATE_NETWORK
	PRIVATE_ROUTER
	PRIVATE_NETMASK
	PUBLIC_NETWORK
	PUBLIC_ROUTER
	PUBLIC_NETMASK
	DHCPD_DYNRANGE

	See defs-example for details.
	YOU MUST RERUN configure AFTER SETTING THESE!

277
20040901: Before build
278
279
	New SSL version of the XMLRPC server.

280
281
282
283
284
	* Make sure python is installed on both boss and ops. If not,
          the port is located in:

		/usr/ports/lang/python		

285
286
	* Install this port on both boss and ops:

287
288
289
290
291
		/usr/ports/security/py-m2crypto

          then patch a silly bug on both boss and ops:

		patch -f -p0 -l -i patches/m2crypto.patch
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329

	* Reconfig, rebuild, and reinstall the testbed software on your
	  boss node. Be sure to pick up changes in sql/database-migrate.

	* Install some ssl files on boss:

		cd /usr/object/tree/ssl
		gmake usercert-install

        * Install new version of testbed startup script on boss
	  
		cd /usr/object/tree/rc.d
		sudo gmake install

        * Generate SSL certificates for existing emulab users

		cd /usr/src/tree/sql
		./initcerts.pl

	* Add the following lines to /etc/syslog.conf on boss

	  !sslxmlrpc
	  *.*                            /usr/testbed/log/sslxmlrpc.log

	* Add the following line to /etc/newsyslog.conf on boss

	  /usr/testbed/log/sslxmlrpc.log          640  7     1000 *     Z

        * Create the log file and restart syslog on boss
	
		sudo touch /usr/testbed/log/sslxmlrpc.log
		sudo kill -HUP `cat /var/run/syslog.pid`

        * Fire up the SSL XMLRPC server

		sudo /usr/testbed/sbin/sslxmlrpc_server.py


330
20040816: Before build
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
	The event system has moved to users.emulab.net.

	* Install the elvin libraries and daemon on users.emulab.net.

	* Pick up this package:

	     http://boss.emulab.net/downloads/ulsshxmlrpcpp-0.1.0.tar.gz

             cd /someplace
	     tar zxf ulsshxmlrpcpp-0.1.0.tar.gz
	     cd ulsshxmlrpcpp-0.1.0
	     configure
	     gmake
	     sudo gmake install

346
347
348
349
	* Install this port (might already be install).

		/usr/ports/databases/py-MySQLdb

350
351
352
353
354
355
356
357
	* Reconfig, rebuild, and reinstall the testbed software on your
	  boss node.

	* Reconfig, rebuild, and reinstall the testbed software on your
          users node. 

		ssh users
		cd /your/object/tree/obj/
358
		gmake ops-install
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396

        * Install the new elvind config file on boss and restart elvind
          Note that this is not done as part of toplevel install.

		cd /your/object/tree/obj/event/etc
		sudo gmake install
		sudo /usr/local/etc/rc.d/2.elvind.sh stop
		sudo /usr/local/etc/rc.d/2.elvind.sh start

	* Restart tmcd and stated on boss (note that if you plan to just
	  reboot boss, a lot of these steps can be skipped).

		sudo /usr/testbed/sbin/tmcd.restart
		sudo kill -USR1 `cat /var/run/stated.pid`

        * Install the elvind config file on users.
          Note that this is not done as part of toplevel install.

		ssh users
		cd /your/object/tree/obj/event/etc
		sudo gmake control-install

	* Install new rc.d file on boss.
          Note that this is not done as part of toplevel install.

		cd /your/object/tree/obj/rc.d
		sudo gmake install

        * Install the rc.d file that starts elvind on users.
          Note that this is not done as part of toplevel install.

		ssh users
		cd /your/object/tree/obj/rc.d
		sudo gmake control-install

        * Start the elvin server on users.emulab.net:

		ssh users
397
		sudo /usr/local/etc/rc.d/2.elvind.sh start
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412

	* Create the ops certificate (on boss) and copy it to your users node.

		cd /your/object/tree/obj/ssl
		gmake ctrlnode.pem
		sudo scp ctrlnode.pem users:/etc/emulab/client.pem
		sudo ssh users chmod 600 /etc/emulab/client.pem
		sudo scp emulab.pem users:/etc/emulab

        * Install the tmcd client stuff on users.emulab.net.

		ssh users.emulab.net
		cd /your/object/tree/obj/tmcd/freebsd
		sudo gmake control-install

413
414
415
416
	* Update sql/database-fill-suplement.sql stuff.

		cat sql/database-fill-suplement.sql | mysql tbdb

417
        * Create the emulab-ops/opsnode experiment. Do this the same way
Leigh B. Stoller's avatar
Leigh B. Stoller committed
418
419
420
421
422
423
424
          that hwdown is created in setup-db.txt. 

          Then go to the experiment information page (in red dot mode)
          for the emulab-ops/opsnode experiment, and then click on the
          "Edit Experiment MetaData". In the "Swapping" Section, make
          sure Idle Ignore is checked, and the other three entries
          (Swappable, Idle-Swap, Max Duration) are unchecked.
425

426
427
428
429
430
431
432
433
	* Make sure your source tree on ops is updated. Then, from that source
	  tree, run 'install/newclient -o'. (Note: You will probably have to
	  turn on the 'executable bit' for this script before running it, with
	  'chmod a+x install/newclient'.) This will allow ops to check into the
	  testbed.

	* On boss, run: 'wap /usr/testbed/libexec/newnode ops' to actually add
	  ops to the database.
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459

	* Place the ops node into the above experiment:

		wap nalloc emulab-ops opsnode ops

	* Make sure that netcat is installed.

		cd /usr/ports/net/netcat
		sudo make
		sudo make install
		
	* Add entry to /etc/services for elvin:

		elvin		2917/tcp

	* Change the CNAME event-server to your zone file from boss to ops.

		event-server	IN	CNAME	ops

	* Start the alternate inetd service so that old images trying to
	  connect to the event server are redirected to users.emulab.net.
	  Be sure to replace boss.foo.bar with the right name.

		sudo inetd -a boss.foo.bar -p /var/run/testbed-inetd.pid \
			/usr/testbed/etc/inetd.conf

460
461
462
463
	* Stick boss's host keys onto ops so that users do not get
	  those silly questions about adding know hosts.

		sudo cat /etc/ssh/*.pub |
464
			ssh ops '(cat > /etc/ssh/ssh_known_hosts)'
465

466
20040815: After install
467
468
469
470
471
472
	After some changes to ptopgen, you'll need to run the following SQL
	command to keep vnodes working properly:

	update os_info set nextosid='FBSD-STD' \
	    where osid='emulab-ops-FBSD-JAIL';

473
20040809: After install
474
475
476
477
478
479
480
481
482
483
484
485
486
	New syslog file for logging requests to the XMLRPC server.

		sudo touch /usr/testbed/log/xmlrpcbag.log

	Add these two lines to /etc/syslog.conf and then HUP syslogd:

		!xmlrpcbag
		*.*                      /usr/testbed/log/xmlrpcbag.log

        Add this line to /etc/newsyslog.conf:

		/usr/testbed/log/xmlrpcbag.log      640  7     1000 *     Z

487
20040805: After install
488
489
490
491
	Add a new CNAME for boss to /etc/namedb/<OURDMAIN>.db.head -
	event-server . Re-run named_setup on boss to make this change
	take place. In the future, we may move this CNAME to ops.

492
20040713: After install
Leigh B. Stoller's avatar
Leigh B. Stoller committed
493
494
495
496
497
	For new distributed routing support, must first run this script as
        root to generate a data file for each existing experiment.

		sudo sql/gentopmaps.pl

498
20040625: After install
Leigh B. Stoller's avatar
Leigh B. Stoller committed
499
	After updating to revision 1.257 of sql/database-migrate.txt,
500
501
502
503
504
505
        run sql/devlnodes.pl to clear out all of the non reserved
        pcvm nodes. Be sure to do a current install of the software, and
        then:

		update node_types set isdynamic=1 where type='pcvm';

506
20040615: Before build
507
508
509
510
511
512
513
	If you update the ports on your boss node, you'll need to take
	into accout that FreeBSD is in the process of changing how its
	rc.d scripts work - add the following two lines to the bottom
	of your /etc/rc.conf on boss:
	  dhcpd_enable="YES"
	  apache_enable="YES"

514
20040601: Before build
515
516
	The p5-Time-HiRes port is now required on both boss and ops -
	install it from /usr/ports/devel/p5-Time-HiRes .
517

518
20040420: Before build
519
520
521
	Switched web search button backend from webglimpse to swish-e.
        Just need to go to /usr/ports/www/swish-e and install the port.

522
523
524
525
526
527
20040315: Before build

	XMLRPC Server. Install this port:

		/usr/ports/databases/py-MySQLdb

528
20040303: Before build
529
530
531
532
533
534
535
536
537
538
539
	Fixed the way we handle the certificate for capture with tiptunnel.
	We no longer hard-code the certificate digest in nodetipacl.php3 .
	However, as a result, we must place this fingerprint in a publically-
	readable file on boss. So, if you have serial lines that you're
	running with capture:
	1) Copy your /usr/testbed/etc/capture.pem file from your tipserver to
	boss, if it isn't there already.
	2) In /usr/testbed/etc/ on boss, run: 'openssl x509 -sha -noout
	-fingerprint -in capture.pem > capture.fingerprint', and make this
	file world-readable.

540
20040302: Before build
541
542
543
544
545
	Changed the length of the node_id columns from 10 to 32. Make sure
	you re-compile and restart all daemons written in C (such as
	tmcd, tevd, and the event schedulers) before putting any nodes in
	the database with names longer than 10 characters.

546
20040226: After install
547
548
549
550
551
552
553
554
	Re-worked the way named_setup handles virtual nodes and unroutable
	IP addresses - no point in handling the two differently. Thus, the
	'vnodes.<OURDOMAIN>.db' DNS zone file is no longer generated.
	Instead, if you have a mix of routable and unroutable IP addresses,
	you'll need to switch to using the '<OURDOMAIN>.internal.db' zone
	file - see Step 6 of doc/setup.txt for instructions on setting up
	this file.

555
20040121: After install
556
557
558
559
	Changed the 3.testbed.sh rc.d script to handle 'start' and 'stop'
	arguments properly. Install a fresh copy of rc.d/3.testbed.sh into
	/usr/local/etc/rc.d .

560
20040120: After install
561
562
563
564
565
566
567
	Bumped up the sizes on some buffers to help network performance.
	Add the following two lines to /etc/sysctl.conf:

	net.local.dgram.maxdgram=65536
	net.local.dgram.recvspace=65536


568
20040112: After install
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
       Big change to bootinfo sequence.

       * Update DB with new contents of sql/database-fill.sql *and
         sql/database-fill-suplement.sql.
       * Rebuild pxeboot kernel from source (src/sys/boot/i386/emuboot).
         Install emuboot to /tftpboot/pxeboot.emu
       * Install new version of dhcpd.conf file that has all the PXE
         crap removed and includes the single line:

		filename "/tftpboot/pxeboot.emu";

       * Restart dhcpd. 
       * Kill proxydhcp. Be sure to install new 3.testbed.sh from the
         rc.d directory to /usr/local/etc/rc.d, or just edit in place
       * Restart bootinfo.
       * Restart stated.
       * Reboot all non allocated nodes so that they go into pxewait mode.

587
20031215: Before build
588
589
590
591
	The mod_php4 port on boss should be recompiled with the
	'xmlrpc' option, in addition to the 'mhash' option already
	used. Apache on boss will need to be restarted after the new
	port is installed.
592