Commit 30e72494 authored by Grant Ayers's avatar Grant Ayers

Moved old setup documentation to setup-archive.

parent a563f099
# Copyright (c) 2002-2008 University of Utah and the Flux Group.
# All rights reserved.
The Emulab installation documentation has been moved to the main Emulab
All information contained in this directory should be considered deprecated
and for reference only.
Information necessary for keeping boss and ops node up-to-date. This
file contains updates that must be made outside of the database and
the install tree (ie. /usr/testbed).
This file is in the same format at the FreeBSD UPDATING file, whis is
to say, in reverse chronological order, with the date of the change
in YYYYMMDD format.
After each date, you will find one of the following, which tells you
when the action needs to be performed.
Before build - before doing a 'gmake' in your testbed object tree on
boss and/or ops
After install - after doing a 'gmake boss-install' on boss or
'gmake ops-install' on ops
Note that some instructions may have steps that need to occur at a few
different points in the install process - these are marked with the
earliest time one of the steps needs to occur.
20080506: After install.
A bug introduced on 2007/10/19 caused the "ezid" flag to not
be properly set when creating an image using the "easy" form.
The net effect of this is that OSID where not being properly
deleted when the image id was. The following two queries
will clean up the mess the bug introduced.
update images set ezid = 1 where not ezid and
exists (select 1 from os_info where osid=imageid and ezid);
delete from os_info where ezid and osid
not in (select osid from osidtoimageid);
Both queries are safe to run, even if you where not affected
by the bug.
20080501: Before install.
Create the per project admin mailing lists if you enabled
mailman support (MAILMANSUPPORT set to 1 in defs file).
This should be done before the final boss-install.
Unfortunately, it also depends on some modification
associated with this commit. Thus do the following
1) Go to build tree
2) Reconfigure and build but don't install
3) Do a "gmake install" in "collab/mailman/".
4) Go to source tree and run the script
5) Go back to build tree and do a "gmake boss-install"
The script will take a while to run so be patent (It took
around an hour for Utah's emulab). It will also create a
lot of output so you might want to redirect stdout.
20080219: After install:
The newnode MFS ssh keys have been updated from protocol 1
to protocol 2 (rsa key). The new keys were installed in
/usr/testbed/etc by the top level install target. If you
plan to add new nodes to your testbed you will want to put the
new key into the newnode MFS. Do the following as root:
mdconfig -a -t vnode -u 1 -f /tftpboot/freebsd.newnode/boot/mfsroot
mount /dev/md1 /mnt
cat install/newnode_sshkeys/identity >>/mnt/root/.ssh/authorized_keys
umount /dev/mnt
mdconfig -d -u 1
cd /tftpboot/freebsd.newnode/boot; ./prepare
20070511: Before build.
Install the new publish/subscribe system which is replacing Elvin.
On both boss and ops, do the following:
* Grab and install the package:
cd /var/tmp
sudo pkg_add /var/tmp/pubsub-elvincompat-0.8.tbz
* Add this line to your defs file:
and reconfig/rebuild/reinstall on bother boss and ops.
* Add these lines to /etc/syslog.conf on boss and ops:
*.* /usr/testbed/log/pubsubd.log
and also this line to /etc/syslog.conf on ops:
*.* /usr/testbed/log/elvin_gateway.log
* Create logfile and restart syslog:
sudo cp /dev/null /usr/testbed/log/pubsubd.log
sudo kill -HUP `cat /var/run/`
* Start pubsubd:
sudo /usr/local/etc/rc.d/ start
* ON OPS ONLY: start the elvin gateway:
sudo /usr/local/etc/rc.d/ start
* ON BOSS ONLY: disable elvind:
sudo rm -f /usr/local/etc/rc.d/
20070320: Time to upgrade your tftpd. This has only been tested on
FreeBSD 6.x, so if you are still running FreeBSD 4.10, then
you can skip this. The only significant difference (other
than going from version 0.33 to 0.48, which is ancillary) is
a change to make tftpd work better under load when it hits its
"maxchildren" limit.
* See if your tftpd is already up to date on your boss:
pkg_info -E 'emulab-tftp-hpa-*'
If it reports anything other than "emulab-tftp-hpa-0.48",
you should continue.
* Grab the new package:
cd /var/tmp
* Shutdown and deinstall the old server:
sudo /usr/local/etc/rc.d/ stop
sudo pkg_delete -f emulab-tftp-hpa-0.33
* Install and start the new server:
sudo pkg_add /var/tmp/emulab-tftp-hpa-0.48.tbz
sudo /usr/local/etc/rc.d/ start
If you do want to update in a FreeBSD 4.10 environment, you will
have to build from it from the port. Make sure you have the
recommended port tree installed as /usr/ports and then:
cd <emulab-srcdir>/install/ports; sudo ./ports-install
cd /usr/ports/misc/emulab-tftp-hpa
sudo make
sudo /usr/local/etc/rc.d/ stop
sudo pkg_delete -f emulab-tftp-hpa-0.33
sudo make install
sudo /usr/local/etc/rc.d/ start
20070317: Before build.
Due to a change in gcc's handling of c++ bools in functions with
variable argument lists, you will need to reinstall ulsshxmlrpcpp and
rebuild/reinstall in your testbed source tree so that the event
system is recompiled against the new version.
* Grab this file from our website and unpack it somewhere convenient
on your boss node:
* Unpack, configure, build, install:
cd someplace
tar xzf ulsshxmlrpcpp-0.1.2.tar.gz
mkdir ulbuild
cd ulbuild
gmake && gmake install
* Rebuild your testbed source tree:
cd testbed-build-dir
gmake && gmake install
20061201: Time to upgrade your mysql ... If you are still running
FreeBSD 4.10 on your boss node, then you are almost certainly
running an ancient version of mysql (3.23). In lieu of upgrading
to FreeBSD 6.1 (the "right" solution, albiet very time
consuming), you can upgrade your mysql in place. Here is how:
* Grab this file from our website and place it someplace on your
boss node with at least 250MB of disk space:
* Unpack it:
cd someplace
tar zxf mysql-5.0.27.tar.gz
* Configure:
cd mysql-5.0.27
./configure --localstatedir=/var/db/mysql \
--without-debug \
--with-readline \
--without-libedit \
--without-bench \
--without-extra-tools \
--with-libwrap \
--with-mysqlfs \
--with-low-memory \
--with-comment='FreeBSD build for Emulab' \
* Build and install:
sudo make install
* You want to grab a new version of this file from emulab:
and place it in your testbed source directory, in the rc.d
subdir. Then go to your object tree, cd into the rc.d
directory and type gmake. You probably just want to copy the
new file out to /usr/local/etc/rc.d/ instead of doing an
* Reboot your boss node.
20060831: needs to be patched on boss
This needs to be done before installing rev 1.254 of db/
Locate which should be somewhere in
find /usr/local/lib/perl5/site_perl -name
Than go into the "patches" directory apply the patch "":
patch /usr/local/lib/perl5/site_perl/.../ <
Replacing "/usr/local/lib/perl5/site_perl/.../" with the real
location of on your system.
Watch out for revision 4.74 in sql/database-migrate.sql ... You want
to do that revision by hand so that you do not lose important data.
(Some sites script the contents of sql/database-migrate.sql)
Be sure to firewall the mysql port (3306) at your router so that
outside networks cannot access that port on your ops node (we now
run mysqld on ops).
20060525: Do this before installing tbsetup/ rev 2.14
It would be best if your Emulab was down during this change.
No schema changes, but due to a bug in the uid &
exptidx columns got swapped for new values inserted into the
session_info table after the 4.56 schema change. This is fixed
in revision number 2.14 of
The values of session_info are currently not used by any testbed
code. However, it is still a good idea to fix the existing data
in the database. To do so determine the "session" where the
values got swapped. For that was 9359824:
| session | uid | exptidx |
| 9359747 | 31359 | 24893 |
| 9359824 | 24893 | 31359 |
Once this is determined, lock the session_info table, swap
the columns, install the fix, than unlock the table:
lock tables session_info write;
create temporary table session_info_wk as select * from session_info
where session >=SESSION;
replace into session_info select session,exptidx,uid
from session_info_wk;
drop table session_info_wk;
[Now install the new code]
unlock tables;
Where SESSION is the session number where the values where first
swapped, 9359824, for
20060322: Before installing tbsetup/ns2ir/ rev 1.109 (3/22/06).
This revision introduces a new features/desire pair called
"firewallable". All nodes in a firewalled experiment require
this feature from a node. So you will need to do:
insert into node_type_features values ('TYPE','firewallable',0);
for every node type 'TYPE' which can participate in a firewalled
experiment. The current requirement for a 'firewallable' node
is that it can be powered off (not just power cycled). This is
required for proper post-swapout cleaning of nodes. Another
attribute to consider is whether you can prevent a node from
booting anywhere other than the network. If you have nodes that
can timeout from PXE boot and that will fall back to booting from
the hard drive, you may want to NOT make those nodes 'firewallable'.
20051212: After DB schema is updated, and updated software installed.
These changes are for shifting from SSH protocol 1 keys to
protocol 2 keys.
For users:
wap /usr/testbed/sbin/genpubkeys
this will generate new Protocol 2 (rsa) keys for all users,
and regen their authorized_keys file. Note that existing sites
will need to retain protocol 1 keys for users because of old
For boss and ops:
You must setup a new passphraseless keypair for root on boss.
To do this, simply run the following command on boss:
sudo /usr/bin/ssh-keygen -t rsa -N ""
Next, append the contents of /root/.ssh/ on boss to
the /root/.ssh/authorized_keys file on both your ops and boss
nodes. Remove any ssh version 1 keys that appear in either
of these authorized_keys files. Also, remove the
authorized_keys2 file (if it exists) from the /root/.ssh
directory on boss and ops. Next, change the
/etc/ssh/sshd_config (sshd config) file on both boss and ops
to try ssh protocol 2 first: Edit the line that reads
"Protocol 1,2" to be "Procotol 2,1". Finally, be sure to
remove any "Protocol" directives from /root/.ssh/config on
boss and ops.
In your database:
You'll need to update the key stored in the node/ssh_pubkey
variable to contain the public half of the new v2 rsa key you
created for root on boss. This key will be automatically
distributed to the nodes as they boot up when using images
dating from late 2004 onward.
For your default disk images:
Add the contents of /root/.ssh/ on your boss node
to the /root/.ssh/authorized_keys file in each of your
default disk images. Remove any version 1 ssh keys that
remain in authorized_keys. Also, remove any authorized_keys2
file that might exist in the /root/.ssh directory. Next,
edit the /etc/ssh/sshd_config file, changing the "Procotol
1,2" line to "Protocol 2,1". Re-create the modified images
(of course).
Note on user disk images:
User images may still require access via ssh version 1, so
don't delete the ssh version 1 key on your boss node
(/root/.ssh/identity[.pub]). The ssh wrapper command (sshtb)
has been modified to try authenticating with both ssh protocols
to preserve backward compatibility with existing user images.
20051212: After DB schema is updated, and updated software installed.
Fix an error in the statistics gathering code, that was
causing duplicate swapin records in some case. From the source
perl sql/ | mysql tbdb
20050818: Anytime after the DB schema is updated
Load the initial contents of the new knowledge_base_entries table
from sql/knowlbase-create.sql
cat sql/knowlbase-create.sql | mysql tbdb
20050707: Before build
Decide if you want CVS support turned on. This is some simple
support for per-project CVS trees, stored as /proj/$pid/CVS.
The main thing is that you can use the cvsweb interface. If you
want it, in your defs file:
After your build and install:
sudo /usr/testbed/sbin/cvsinit
which creates the initial CVS trees in all of the existing project
20050205: Before build/config
* Install the rsync port from /usr/ports/net/rsync
20041108: Anytime
* Update sql/database-fill-suplement.sql stuff.
cat sql/database-fill-suplement.sql | mysql tbdb
20041108: Before build
We've fixed a bug in the elvin libraries that underlie our
event system. You'll want to install the new one from the
port. You need to do this on both boss and ops. Copy the
contents of install/ports/libelvin to /usr/ports/misc/libelvin.
Then, change to that directory and run 'make deinstall reinstall'
to get the new version of the port installed. Now, you'll need to
do a clean rebuild of everything in the testbed source tree, by
doing a 'gmake clean' before the usual 'gmake ops-install' and
'gmake boss-install'.
20041104: Before build
A new directory, pxe/freebsd, contains the diffs necessary
to build your own pxeboot program more easily. The entry
for 20040112 below instructs you to "Rebuild pxeboot kernel
from source". Now you can actually do that! See the README
file in pxe/freebsd. You should rebuild your pxeboot program
to take advantage of a couple of new features: the ability
to specify an explicit machine from which to TFTP load a file,
and the ability to specify command lines to Linux (actually,
to LILO).
You will need to update your frisbee/freebsd MFSes to take
full advantage of recent changes. A new file, doc/update-mfs.txt,
describes the process. Read it. Believe it. Live by it.
20041029: Before build
The ulsshxmlrpcpp package was updated to version 0.1.1:
cd /someplace
tar zxf ulsshxmlrpcpp-0.1.1.tar.gz
cd ulsshxmlrpcpp-0.1.1
sudo gmake install
20041027: After install (be sure to update sitevars):
A new sitevariable was added to hold root's ssh public key.
When nodes boot using the latest image, the public key is
handed across via tmcd and statshed in /root/.ssh. This
localizes the image (wrt the root pub key) to the local
Using the web interface in red dot mode, go to the Edit Site
Variables link and store boss:/root/.ssh/ into the
"node/ssh_pubkey" site variable. Make sure there are no
newlines when you paste it in.
20041025: After install
Run sql/ to fix up stats records.
sql/ | mysql tbdb
Run "/usr/testbed/etc/rc.d/and start" as root on ops to start
up the auto nice daemon. This daemon is not required,
however, it can save you from greedy users that consume too
much CPU and cause experiment swapin to fail.
20040930: Before build
You need to update your defs-* file to include some new configure
time variables. These are:
See defs-example for details.
20040901: Before build
New SSL version of the XMLRPC server.
* Make sure python is installed on both boss and ops. If not,
the port is located in:
* Install this port on both boss and ops:
then patch a silly bug on both boss and ops:
patch -f -p0 -l -i patches/m2crypto.patch
* Reconfig, rebuild, and reinstall the testbed software on your
boss node. Be sure to pick up changes in sql/database-migrate.
* Install some ssl files on boss:
cd /usr/object/tree/ssl
gmake usercert-install
* Install new version of testbed startup script on boss
cd /usr/object/tree/rc.d
sudo gmake install
* Generate SSL certificates for existing emulab users
cd /usr/src/tree/sql
* Add the following lines to /etc/syslog.conf on boss
*.* /usr/testbed/log/sslxmlrpc.log
* Add the following line to /etc/newsyslog.conf on boss
/usr/testbed/log/sslxmlrpc.log 640 7 1000 * Z
* Create the log file and restart syslog on boss
sudo touch /usr/testbed/log/sslxmlrpc.log
sudo kill -HUP `cat /var/run/`
* Fire up the SSL XMLRPC server
sudo /usr/testbed/sbin/
20040816: Before build
The event system has moved to
* Install the elvin libraries and daemon on
* Pick up this package:
cd /someplace
tar zxf ulsshxmlrpcpp-0.1.0.tar.gz
cd ulsshxmlrpcpp-0.1.0
sudo gmake install
* Install this port (might already be install).
* Reconfig, rebuild, and reinstall the testbed software on your
boss node.
* Reconfig, rebuild, and reinstall the testbed software on your
users node.
ssh users
cd /your/object/tree/obj/
gmake ops-install
* Install the new elvind config file on boss and restart elvind
Note that this is not done as part of toplevel install.
cd /your/object/tree/obj/event/etc
sudo gmake install
sudo /usr/local/etc/rc.d/ stop
sudo /usr/local/etc/rc.d/ start
* Restart tmcd and stated on boss (note that if you plan to just
reboot boss, a lot of these steps can be skipped).
sudo /usr/testbed/sbin/tmcd.restart
sudo kill -USR1 `cat /var/run/`
* Install the elvind config file on users.
Note that this is not done as part of toplevel install.
ssh users
cd /your/object/tree/obj/event/etc
sudo gmake control-install
* Install new rc.d file on boss.
Note that this is not done as part of toplevel install.
cd /your/object/tree/obj/rc.d
sudo gmake install
* Install the rc.d file that starts elvind on users.
Note that this is not done as part of toplevel install.
ssh users
cd /your/object/tree/obj/rc.d
sudo gmake control-install
* Start the elvin server on
ssh users
sudo /usr/local/etc/rc.d/ start
* Create the ops certificate (on boss) and copy it to your users node.
cd /your/object/tree/obj/ssl
gmake ctrlnode.pem
sudo scp ctrlnode.pem users:/etc/emulab/client.pem
sudo ssh users chmod 600 /etc/emulab/client.pem
sudo scp emulab.pem users:/etc/emulab
* Install the tmcd client stuff on
cd /your/object/tree/obj/tmcd/freebsd
sudo gmake control-install
* Update sql/database-fill-suplement.sql stuff.
cat sql/database-fill-suplement.sql | mysql tbdb
* Create the emulab-ops/opsnode experiment. Do this the same way
that hwdown is created in setup-db.txt.
Then go to the experiment information page (in red dot mode)
for the emulab-ops/opsnode experiment, and then click on the
"Edit Experiment MetaData". In the "Swapping" Section, make
sure Idle Ignore is checked, and the other three entries
(Swappable, Idle-Swap, Max Duration) are unchecked.
* Make sure your source tree on ops is updated. Then, from that source
tree, run 'install/newclient -o'. (Note: You will probably have to
turn on the 'executable bit' for this script before running it, with