From 30e724944d63966d7d87e64673306ab88dd5b822 Mon Sep 17 00:00:00 2001 From: Grant Ayers Date: Wed, 25 Jun 2008 23:48:54 +0000 Subject: [PATCH] Moved old setup documentation to setup-archive. --- doc/setup-archive/README | 14 + doc/setup-archive/UPDATING | 832 +++++++++++++++++++ doc/setup-archive/adding_nodes.txt | 211 +++++ doc/setup-archive/setup-cisco.txt | 220 +++++ doc/setup-archive/setup-db.txt | 386 +++++++++ doc/setup-archive/setup-fs.txt | 247 ++++++ doc/setup-archive/setup-network.txt | 148 ++++ doc/setup-archive/setup-nodes.txt | 275 ++++++ doc/setup-archive/setup-nortel.txt | 89 ++ doc/setup-archive/setup-ops.txt | 217 +++++ doc/setup-archive/setup-power.txt | 120 +++ doc/setup-archive/setup-serial-and-power.txt | 209 +++++ doc/setup-archive/setup.txt | 355 ++++++++ doc/setup-archive/tipserver.txt | 51 ++ doc/setup-archive/tipwithoutops.txt | 55 ++ doc/setup-archive/update-testbed.txt | 77 ++ 16 files changed, 3506 insertions(+) create mode 100644 doc/setup-archive/README create mode 100644 doc/setup-archive/UPDATING create mode 100644 doc/setup-archive/adding_nodes.txt create mode 100644 doc/setup-archive/setup-cisco.txt create mode 100644 doc/setup-archive/setup-db.txt create mode 100644 doc/setup-archive/setup-fs.txt create mode 100644 doc/setup-archive/setup-network.txt create mode 100644 doc/setup-archive/setup-nodes.txt create mode 100644 doc/setup-archive/setup-nortel.txt create mode 100644 doc/setup-archive/setup-ops.txt create mode 100644 doc/setup-archive/setup-power.txt create mode 100644 doc/setup-archive/setup-serial-and-power.txt create mode 100644 doc/setup-archive/setup.txt create mode 100644 doc/setup-archive/tipserver.txt create mode 100644 doc/setup-archive/tipwithoutops.txt create mode 100644 doc/setup-archive/update-testbed.txt diff --git a/doc/setup-archive/README b/doc/setup-archive/README new file mode 100644 index 000000000..daa04a2b6 --- /dev/null +++ b/doc/setup-archive/README @@ -0,0 +1,14 @@ +# +# EMULAB-COPYRIGHT +# Copyright (c) 2002-2008 University of Utah and the Flux Group. +# All rights reserved. +# + +The Emulab installation documentation has been moved to the main Emulab +wiki: + +https://users.emulab.net/trac/emulab/wiki + + +All information contained in this directory should be considered deprecated +and for reference only. diff --git a/doc/setup-archive/UPDATING b/doc/setup-archive/UPDATING new file mode 100644 index 000000000..e6b0bad87 --- /dev/null +++ b/doc/setup-archive/UPDATING @@ -0,0 +1,832 @@ +Information necessary for keeping boss and ops node up-to-date. This +file contains updates that must be made outside of the database and +the install tree (ie. /usr/testbed). + +This file is in the same format at the FreeBSD UPDATING file, whis is +to say, in reverse chronological order, with the date of the change +in YYYYMMDD format. + +After each date, you will find one of the following, which tells you +when the action needs to be performed. + +Before build - before doing a 'gmake' in your testbed object tree on + boss and/or ops +After install - after doing a 'gmake boss-install' on boss or + 'gmake ops-install' on ops + +Note that some instructions may have steps that need to occur at a few +different points in the install process - these are marked with the +earliest time one of the steps needs to occur. + +20080506: After install. + + A bug introduced on 2007/10/19 caused the "ezid" flag to not + be properly set when creating an image using the "easy" form. + The net effect of this is that OSID where not being properly + deleted when the image id was. The following two queries + will clean up the mess the bug introduced. + + update images set ezid = 1 where not ezid and + exists (select 1 from os_info where osid=imageid and ezid); + delete from os_info where ezid and osid + not in (select osid from osidtoimageid); + + Both queries are safe to run, even if you where not affected + by the bug. + +20080501: Before install. + + Create the per project admin mailing lists if you enabled + mailman support (MAILMANSUPPORT set to 1 in defs file). + + This should be done before the final boss-install. + Unfortunately, it also depends on some modification + associated with this commit. Thus do the following + + 1) Go to build tree + 2) Reconfigure and build but don't install + 3) Do a "gmake install" in "collab/mailman/". + 4) Go to source tree and run the script + sql/create-projadminlists.pl + 5) Go back to build tree and do a "gmake boss-install" + + The script will take a while to run so be patent (It took + around an hour for Utah's emulab). It will also create a + lot of output so you might want to redirect stdout. + +20080219: After install: + + The newnode MFS ssh keys have been updated from protocol 1 + to protocol 2 (rsa key). The new keys were installed in + /usr/testbed/etc by the top level install target. If you + plan to add new nodes to your testbed you will want to put the + new key into the newnode MFS. Do the following as root: + + mdconfig -a -t vnode -u 1 -f /tftpboot/freebsd.newnode/boot/mfsroot + mount /dev/md1 /mnt + cat install/newnode_sshkeys/identity >>/mnt/root/.ssh/authorized_keys + umount /dev/mnt + mdconfig -d -u 1 + cd /tftpboot/freebsd.newnode/boot; ./prepare + +20070511: Before build. + + Install the new publish/subscribe system which is replacing Elvin. + DO THIS ONLY IF YOU ARE AN EXISTING SITE WITH ELVIN INSTALLED! + On both boss and ops, do the following: + + * Grab and install the package: + + cd /var/tmp + fetch http://www.emulab.net/downloads/pubsub-elvincompat-0.8.tbz + sudo pkg_add /var/tmp/pubsub-elvincompat-0.8.tbz + + * Add this line to your defs file: + + ELVIN_COMPAT=1 + + and reconfig/rebuild/reinstall on bother boss and ops. + + * Add these lines to /etc/syslog.conf on boss and ops: + + !pubsubd + *.* /usr/testbed/log/pubsubd.log + + and also this line to /etc/syslog.conf on ops: + + !elvin_gateway + *.* /usr/testbed/log/elvin_gateway.log + + * Create logfile and restart syslog: + + sudo cp /dev/null /usr/testbed/log/pubsubd.log + sudo kill -HUP `cat /var/run/syslog.pid` + + * Start pubsubd: + + sudo /usr/local/etc/rc.d/2.pubsubd.sh start + + * ON OPS ONLY: start the elvin gateway: + + sudo /usr/local/etc/rc.d/3.elvin_gateway.sh start + + * ON BOSS ONLY: disable elvind: + + sudo rm -f /usr/local/etc/rc.d/2.elvind.sh + +20070320: Time to upgrade your tftpd. This has only been tested on + FreeBSD 6.x, so if you are still running FreeBSD 4.10, then + you can skip this. The only significant difference (other + than going from version 0.33 to 0.48, which is ancillary) is + a change to make tftpd work better under load when it hits its + "maxchildren" limit. + + * See if your tftpd is already up to date on your boss: + + pkg_info -E 'emulab-tftp-hpa-*' + + If it reports anything other than "emulab-tftp-hpa-0.48", + you should continue. + + * Grab the new package: + + cd /var/tmp + fetch http://www.emulab.net/downloads/emulab-tftp-hpa-0.48.tbz + + * Shutdown and deinstall the old server: + + sudo /usr/local/etc/rc.d/tftpd-hpa.sh stop + sudo pkg_delete -f emulab-tftp-hpa-0.33 + + * Install and start the new server: + + sudo pkg_add /var/tmp/emulab-tftp-hpa-0.48.tbz + sudo /usr/local/etc/rc.d/tftpd-hpa.sh start + + If you do want to update in a FreeBSD 4.10 environment, you will + have to build from it from the port. Make sure you have the + recommended port tree installed as /usr/ports and then: + + cd /install/ports; sudo ./ports-install + cd /usr/ports/misc/emulab-tftp-hpa + sudo make + sudo /usr/local/etc/rc.d/tftpd-hpa.sh stop + sudo pkg_delete -f emulab-tftp-hpa-0.33 + sudo make install + sudo /usr/local/etc/rc.d/tftpd-hpa.sh start + +20070317: Before build. + + Due to a change in gcc's handling of c++ bools in functions with + variable argument lists, you will need to reinstall ulsshxmlrpcpp and + rebuild/reinstall in your testbed source tree so that the event + system is recompiled against the new version. + + * Grab this file from our website and unpack it somewhere convenient + on your boss node: + + http://www.emulab.net/downloads/ulsshxmlrpcpp-0.1.2.tar.gz + + * Unpack, configure, build, install: + + cd someplace + tar xzf ulsshxmlrpcpp-0.1.2.tar.gz + mkdir ulbuild + cd ulbuild + ../ulsshxmlrpcpp-0.1.2/configure + gmake && gmake install + + * Rebuild your testbed source tree: + + cd testbed-build-dir + gmake && gmake install + +20061201: Time to upgrade your mysql ... If you are still running + FreeBSD 4.10 on your boss node, then you are almost certainly + running an ancient version of mysql (3.23). In lieu of upgrading + to FreeBSD 6.1 (the "right" solution, albiet very time + consuming), you can upgrade your mysql in place. Here is how: + + * Grab this file from our website and place it someplace on your + boss node with at least 250MB of disk space: + + http://boss.emulab.net/downloads/mysql-5.0.27.tar.gz + + * Unpack it: + + cd someplace + tar zxf mysql-5.0.27.tar.gz + + * Configure: + + cd mysql-5.0.27 + ./configure --localstatedir=/var/db/mysql \ + --without-debug \ + --with-readline \ + --without-libedit \ + --without-bench \ + --without-extra-tools \ + --with-libwrap \ + --with-mysqlfs \ + --with-low-memory \ + --with-comment='FreeBSD build for Emulab' \ + --enable-thread-safe-client + + * Build and install: + + make + sudo make install + + * You want to grab a new version of this file from emulab: + + http://boss.emulab.net/downloads/2.mysql-server.sh.in + + and place it in your testbed source directory, in the rc.d + subdir. Then go to your object tree, cd into the rc.d + directory and type gmake. You probably just want to copy the + new file out to /usr/local/etc/rc.d/ instead of doing an + install. + + * Reboot your boss node. + + +20060831: Mysql.pm needs to be patched on boss + + This needs to be done before installing rev 1.254 of db/libdb.pm.in + + Locate Mysql.pm which should be somewhere in + /usr/local/lib/perl5/site_perl: + + find /usr/local/lib/perl5/site_perl -name Mysql.pm + + Than go into the "patches" directory apply the patch "Mysql.pm.patch": + + patch /usr/local/lib/perl5/site_perl/.../Mysql.pm < Mysql.pm.patch + + Replacing "/usr/local/lib/perl5/site_perl/.../Mysql.pm" with the real + location of Mysql.pm on your system. + +20060717: + Watch out for revision 4.74 in sql/database-migrate.sql ... You want + to do that revision by hand so that you do not lose important data. + + (Some sites script the contents of sql/database-migrate.sql) + +20060606: + + Be sure to firewall the mysql port (3306) at your router so that + outside networks cannot access that port on your ops node (we now + run mysqld on ops). + +20060525: Do this before installing tbsetup/libtblog.pm.in rev 2.14 + It would be best if your Emulab was down during this change. + + No schema changes, but due to a bug in libtblog.pm.in the uid & + exptidx columns got swapped for new values inserted into the + session_info table after the 4.56 schema change. This is fixed + in revision number 2.14 of libtblog.pm.in. + + The values of session_info are currently not used by any testbed + code. However, it is still a good idea to fix the existing data + in the database. To do so determine the "session" where the + values got swapped. For emulab.net that was 9359824: + + +---------+-------+---------+ + | session | uid | exptidx | + +---------+-------+---------+ + | 9359747 | 31359 | 24893 | + | 9359824 | 24893 | 31359 | + +---------+-------+---------+ + + Once this is determined, lock the session_info table, swap + the columns, install the fix, than unlock the table: + + lock tables session_info write; + + create temporary table session_info_wk as select * from session_info + where session >=SESSION; + replace into session_info select session,exptidx,uid + from session_info_wk; + drop table session_info_wk; + + [Now install the new code] + + unlock tables; + + Where SESSION is the session number where the values where first + swapped, 9359824, for emulab.net. + +20060322: Before installing tbsetup/ns2ir/sim.tcl.in rev 1.109 (3/22/06). + + This revision introduces a new features/desire pair called + "firewallable". All nodes in a firewalled experiment require + this feature from a node. So you will need to do: + + insert into node_type_features values ('TYPE','firewallable',0); + + for every node type 'TYPE' which can participate in a firewalled + experiment. The current requirement for a 'firewallable' node + is that it can be powered off (not just power cycled). This is + required for proper post-swapout cleaning of nodes. Another + attribute to consider is whether you can prevent a node from + booting anywhere other than the network. If you have nodes that + can timeout from PXE boot and that will fall back to booting from + the hard drive, you may want to NOT make those nodes 'firewallable'. + +20051212: After DB schema is updated, and updated software installed. + + These changes are for shifting from SSH protocol 1 keys to + protocol 2 keys. + + For users: + + wap /usr/testbed/sbin/genpubkeys + + this will generate new Protocol 2 (rsa) keys for all users, + and regen their authorized_keys file. Note that existing sites + will need to retain protocol 1 keys for users because of old + images. + + For boss and ops: + + You must setup a new passphraseless keypair for root on boss. + To do this, simply run the following command on boss: + + sudo /usr/bin/ssh-keygen -t rsa -N "" + + Next, append the contents of /root/.ssh/id_rsa.pub on boss to + the /root/.ssh/authorized_keys file on both your ops and boss + nodes. Remove any ssh version 1 keys that appear in either + of these authorized_keys files. Also, remove the + authorized_keys2 file (if it exists) from the /root/.ssh + directory on boss and ops. Next, change the + /etc/ssh/sshd_config (sshd config) file on both boss and ops + to try ssh protocol 2 first: Edit the line that reads + "Protocol 1,2" to be "Procotol 2,1". Finally, be sure to + remove any "Protocol" directives from /root/.ssh/config on + boss and ops. + + In your database: + + You'll need to update the key stored in the node/ssh_pubkey + variable to contain the public half of the new v2 rsa key you + created for root on boss. This key will be automatically + distributed to the nodes as they boot up when using images + dating from late 2004 onward. + + For your default disk images: + + Add the contents of /root/.ssh/id_rsa.pub on your boss node + to the /root/.ssh/authorized_keys file in each of your + default disk images. Remove any version 1 ssh keys that + remain in authorized_keys. Also, remove any authorized_keys2 + file that might exist in the /root/.ssh directory. Next, + edit the /etc/ssh/sshd_config file, changing the "Procotol + 1,2" line to "Protocol 2,1". Re-create the modified images + (of course). + + Note on user disk images: + + User images may still require access via ssh version 1, so + don't delete the ssh version 1 key on your boss node + (/root/.ssh/identity[.pub]). The ssh wrapper command (sshtb) + has been modified to try authenticating with both ssh protocols + to preserve backward compatibility with existing user images. + + +20051212: After DB schema is updated, and updated software installed. + + Fix an error in the statistics gathering code, that was + causing duplicate swapin records in some case. From the source + directory: + + perl sql/fixstats-swapin.pl | mysql tbdb + + +20050818: Anytime after the DB schema is updated + + Load the initial contents of the new knowledge_base_entries table + from sql/knowlbase-create.sql + + cat sql/knowlbase-create.sql | mysql tbdb + + +20050707: Before build + + Decide if you want CVS support turned on. This is some simple + support for per-project CVS trees, stored as /proj/$pid/CVS. + The main thing is that you can use the cvsweb interface. If you + want it, in your defs file: + + CVSSUPPORT=1 + + After your build and install: + + sudo /usr/testbed/sbin/cvsinit + + which creates the initial CVS trees in all of the existing project + trees. + +20050205: Before build/config + + * Install the rsync port from /usr/ports/net/rsync + + +20041108: Anytime + + * Update sql/database-fill-suplement.sql stuff. + + cat sql/database-fill-suplement.sql | mysql tbdb + +20041108: Before build + We've fixed a bug in the elvin libraries that underlie our + event system. You'll want to install the new one from the + port. You need to do this on both boss and ops. Copy the + contents of install/ports/libelvin to /usr/ports/misc/libelvin. + Then, change to that directory and run 'make deinstall reinstall' + to get the new version of the port installed. Now, you'll need to + do a clean rebuild of everything in the testbed source tree, by + doing a 'gmake clean' before the usual 'gmake ops-install' and + 'gmake boss-install'. + +20041104: Before build + A new directory, pxe/freebsd, contains the diffs necessary + to build your own pxeboot program more easily. The entry + for 20040112 below instructs you to "Rebuild pxeboot kernel + from source". Now you can actually do that! See the README + file in pxe/freebsd. You should rebuild your pxeboot program + to take advantage of a couple of new features: the ability + to specify an explicit machine from which to TFTP load a file, + and the ability to specify command lines to Linux (actually, + to LILO). + + You will need to update your frisbee/freebsd MFSes to take + full advantage of recent changes. A new file, doc/update-mfs.txt, + describes the process. Read it. Believe it. Live by it. + +20041029: Before build + The ulsshxmlrpcpp package was updated to version 0.1.1: + + http://boss.emulab.net/downloads/ulsshxmlrpcpp-0.1.1.tar.gz + + cd /someplace + tar zxf ulsshxmlrpcpp-0.1.1.tar.gz + cd ulsshxmlrpcpp-0.1.1 + configure + gmake + sudo gmake install + +20041027: After install (be sure to update sitevars): + + A new sitevariable was added to hold root's ssh public key. + When nodes boot using the latest image, the public key is + handed across via tmcd and statshed in /root/.ssh. This + localizes the image (wrt the root pub key) to the local + testbed. + + Using the web interface in red dot mode, go to the Edit Site + Variables link and store boss:/root/.ssh/identity.pub into the + "node/ssh_pubkey" site variable. Make sure there are no + newlines when you paste it in. + +20041025: After install + + Run sql/fixstats_swapmod.pl to fix up stats records. + + sql/fixstats_swapmod.pl | mysql tbdb + + Run "/usr/testbed/etc/rc.d/and start" as root on ops to start + up the auto nice daemon. This daemon is not required, + however, it can save you from greedy users that consume too + much CPU and cause experiment swapin to fail. + +20040930: Before build + + You need to update your defs-* file to include some new configure + time variables. These are: + + SSLCERT_COUNTRY + SSLCERT_STATE + SSLCERT_LOCALITY + SSLCERT_ORGNAME + TESTBED_NETWORK + TESTBED_NETMASK + BOSSNODE_IP + USERNODE_IP + CONTROL_ROUTER_IP + CONTROL_NETWORK + CONTROL_NETMASK + PRIVATE_NETWORK + PRIVATE_ROUTER + PRIVATE_NETMASK + PUBLIC_NETWORK + PUBLIC_ROUTER + PUBLIC_NETMASK + DHCPD_DYNRANGE + + See defs-example for details. + YOU MUST RERUN configure AFTER SETTING THESE! + +20040901: Before build + New SSL version of the XMLRPC server. + + * Make sure python is installed on both boss and ops. If not, + the port is located in: + + /usr/ports/lang/python + + * Install this port on both boss and ops: + + /usr/ports/security/py-m2crypto + + then patch a silly bug on both boss and ops: + + patch -f -p0 -l -i patches/m2crypto.patch + + * Reconfig, rebuild, and reinstall the testbed software on your + boss node. Be sure to pick up changes in sql/database-migrate. + + * Install some ssl files on boss: + + cd /usr/object/tree/ssl + gmake usercert-install + + * Install new version of testbed startup script on boss + + cd /usr/object/tree/rc.d + sudo gmake install + + * Generate SSL certificates for existing emulab users + + cd /usr/src/tree/sql + ./initcerts.pl + + * Add the following lines to /etc/syslog.conf on boss + + !sslxmlrpc + *.* /usr/testbed/log/sslxmlrpc.log + + * Add the following line to /etc/newsyslog.conf on boss + + /usr/testbed/log/sslxmlrpc.log 640 7 1000 * Z + + * Create the log file and restart syslog on boss + + sudo touch /usr/testbed/log/sslxmlrpc.log + sudo kill -HUP `cat /var/run/syslog.pid` + + * Fire up the SSL XMLRPC server + + sudo /usr/testbed/sbin/sslxmlrpc_server.py + + +20040816: Before build + The event system has moved to users.emulab.net. + + * Install the elvin libraries and daemon on users.emulab.net. + + * Pick up this package: + + http://boss.emulab.net/downloads/ulsshxmlrpcpp-0.1.0.tar.gz + + cd /someplace + tar zxf ulsshxmlrpcpp-0.1.0.tar.gz + cd ulsshxmlrpcpp-0.1.0 + configure + gmake + sudo gmake install + + * Install this port (might already be install). + + /usr/ports/databases/py-MySQLdb + + * Reconfig, rebuild, and reinstall the testbed software on your + boss node. + + * Reconfig, rebuild, and reinstall the testbed software on your + users node. + + ssh users + cd /your/object/tree/obj/ + gmake ops-install + + * Install the new elvind config file on boss and restart elvind + Note that this is not done as part of toplevel install. + + cd /your/object/tree/obj/event/etc + sudo gmake install + sudo /usr/local/etc/rc.d/2.elvind.sh stop + sudo /usr/local/etc/rc.d/2.elvind.sh start + + * Restart tmcd and stated on boss (note that if you plan to just + reboot boss, a lot of these steps can be skipped). + + sudo /usr/testbed/sbin/tmcd.restart + sudo kill -USR1 `cat /var/run/stated.pid` + + * Install the elvind config file on users. + Note that this is not done as part of toplevel install. + + ssh users + cd /your/object/tree/obj/event/etc + sudo gmake control-install + + * Install new rc.d file on boss. + Note that this is not done as part of toplevel install. + + cd /your/object/tree/obj/rc.d + sudo gmake install + + * Install the rc.d file that starts elvind on users. + Note that this is not done as part of toplevel install. + + ssh users + cd /your/object/tree/obj/rc.d + sudo gmake control-install + + * Start the elvin server on users.emulab.net: + + ssh users + sudo /usr/local/etc/rc.d/2.elvind.sh start + + * Create the ops certificate (on boss) and copy it to your users node. + + cd /your/object/tree/obj/ssl + gmake ctrlnode.pem + sudo scp ctrlnode.pem users:/etc/emulab/client.pem + sudo ssh users chmod 600 /etc/emulab/client.pem + sudo scp emulab.pem users:/etc/emulab + + * Install the tmcd client stuff on users.emulab.net. + + ssh users.emulab.net + cd /your/object/tree/obj/tmcd/freebsd + sudo gmake control-install + + * Update sql/database-fill-suplement.sql stuff. + + cat sql/database-fill-suplement.sql | mysql tbdb + + * Create the emulab-ops/opsnode experiment. Do this the same way + that hwdown is created in setup-db.txt. + + Then go to the experiment information page (in red dot mode) + for the emulab-ops/opsnode experiment, and then click on the + "Edit Experiment MetaData". In the "Swapping" Section, make + sure Idle Ignore is checked, and the other three entries + (Swappable, Idle-Swap, Max Duration) are unchecked. + + * Make sure your source tree on ops is updated. Then, from that source + tree, run 'install/newclient -o'. (Note: You will probably have to + turn on the 'executable bit' for this script before running it, with + 'chmod a+x install/newclient'.) This will allow ops to check into the + testbed. + + * On boss, run: 'wap /usr/testbed/libexec/newnode ops' to actually add + ops to the database. + + * Place the ops node into the above experiment: + + wap nalloc emulab-ops opsnode ops + + * Make sure that netcat is installed. + + cd /usr/ports/net/netcat + sudo make + sudo make install + + * Add entry to /etc/services for elvin: + + elvin 2917/tcp + + * Change the CNAME event-server to your zone file from boss to ops. + + event-server IN CNAME ops + + * Start the alternate inetd service so that old images trying to + connect to the event server are redirected to users.emulab.net. + Be sure to replace boss.foo.bar with the right name. + + sudo inetd -a boss.foo.bar -p /var/run/testbed-inetd.pid \ + /usr/testbed/etc/inetd.conf + + * Stick boss's host keys onto ops so that users do not get + those silly questions about adding know hosts. + + sudo cat /etc/ssh/*.pub | + ssh ops '(cat > /etc/ssh/ssh_known_hosts)' + +20040815: After install + After some changes to ptopgen, you'll need to run the following SQL + command to keep vnodes working properly: + + update os_info set nextosid='FBSD-STD' \ + where osid='emulab-ops-FBSD-JAIL'; + +20040809: After install + New syslog file for logging requests to the XMLRPC server. + + sudo touch /usr/testbed/log/xmlrpcbag.log + + Add these two lines to /etc/syslog.conf and then HUP syslogd: + + !xmlrpcbag + *.* /usr/testbed/log/xmlrpcbag.log + + Add this line to /etc/newsyslog.conf: + + /usr/testbed/log/xmlrpcbag.log 640 7 1000 * Z + +20040805: After install + Add a new CNAME for boss to /etc/namedb/.db.head - + event-server . Re-run named_setup on boss to make this change + take place. In the future, we may move this CNAME to ops. + +20040713: After install + For new distributed routing support, must first run this script as + root to generate a data file for each existing experiment. + + sudo sql/gentopmaps.pl + +20040625: After install + After updating to revision 1.257 of sql/database-migrate.txt, + run sql/devlnodes.pl to clear out all of the non reserved + pcvm nodes. Be sure to do a current install of the software, and + then: + + update node_types set isdynamic=1 where type='pcvm'; + +20040615: Before build + If you update the ports on your boss node, you'll need to take + into accout that FreeBSD is in the process of changing how its + rc.d scripts work - add the following two lines to the bottom + of your /etc/rc.conf on boss: + dhcpd_enable="YES" + apache_enable="YES" + +20040601: Before build + The p5-Time-HiRes port is now required on both boss and ops - + install it from /usr/ports/devel/p5-Time-HiRes . + +20040420: Before build + Switched web search button backend from webglimpse to swish-e. + Just need to go to /usr/ports/www/swish-e and install the port. + +20040315: Before build + + XMLRPC Server. Install this port: + + /usr/ports/databases/py-MySQLdb + +20040303: Before build + Fixed the way we handle the certificate for capture with tiptunnel. + We no longer hard-code the certificate digest in nodetipacl.php3 . + However, as a result, we must place this fingerprint in a publically- + readable file on boss. So, if you have serial lines that you're + running with capture: + 1) Copy your /usr/testbed/etc/capture.pem file from your tipserver to + boss, if it isn't there already. + 2) In /usr/testbed/etc/ on boss, run: 'openssl x509 -sha -noout + -fingerprint -in capture.pem > capture.fingerprint', and make this + file world-readable. + +20040302: Before build + Changed the length of the node_id columns from 10 to 32. Make sure + you re-compile and restart all daemons written in C (such as + tmcd, tevd, and the event schedulers) before putting any nodes in + the database with names longer than 10 characters. + +20040226: After install + Re-worked the way named_setup handles virtual nodes and unroutable + IP addresses - no point in handling the two differently. Thus, the + 'vnodes..db' DNS zone file is no longer generated. + Instead, if you have a mix of routable and unroutable IP addresses, + you'll need to switch to using the '.internal.db' zone + file - see Step 6 of doc/setup.txt for instructions on setting up + this file. + +20040202: Before install: + + batchexp -q -i -w -f -n -S 'System Experiment' \ + -L 'System Experiment' \ + -E 'Nodes in limbo during nfree - DO NOT DELETE' \ + -p emulab-ops -e nfree-locked + +20040121: After install + Changed the 3.testbed.sh rc.d script to handle 'start' and 'stop' + arguments properly. Install a fresh copy of rc.d/3.testbed.sh into + /usr/local/etc/rc.d . + +20040120: After install + Bumped up the sizes on some buffers to help network performance. + Add the following two lines to /etc/sysctl.conf: + + net.local.dgram.maxdgram=65536 + net.local.dgram.recvspace=65536 + + +20040112: After install + Big change to bootinfo sequence. + + * Update DB with new contents of sql/database-fill.sql *and + sql/database-fill-suplement.sql. + * Rebuild pxeboot kernel from source (src/sys/boot/i386/emuboot). + Install emuboot to /tftpboot/pxeboot.emu + * Install new version of dhcpd.conf file that has all the PXE + crap removed and includes the single line: + + filename "/tftpboot/pxeboot.emu"; + + * Restart dhcpd. + * Kill proxydhcp. Be sure to install new 3.testbed.sh from the + rc.d directory to /usr/local/etc/rc.d, or just edit in place + * Restart bootinfo. + * Restart stated. + * Reboot all non allocated nodes so that they go into pxewait mode. + +20031215: Before build + The mod_php4 port on boss should be recompiled with the + 'xmlrpc' option, in addition to the 'mhash' option already + used. Apache on boss will need to be restarted after the new + port is installed. + diff --git a/doc/setup-archive/adding_nodes.txt b/doc/setup-archive/adding_nodes.txt new file mode 100644 index 000000000..b41c5039b --- /dev/null +++ b/doc/setup-archive/adding_nodes.txt @@ -0,0 +1,211 @@ +This file documents the process of adding a new node to the testbed. + +A. Information about the node +----------------------------- + +1. MAC address. For each port in the new node you need to find out the + MAC address, and which port (eth0/fxp0) it is in software. You need + to know both the Linux (eth) and BSD (fxp,xl,dc,etc.) names. + +2. Wiring. We need to know which physical port on the back of the + machine maps to eth0, eth1, etc., and where each port is connected + to the cisco (get module/port, ie 3/21). + +3. Power. Plug it into a power controller, and make note of which one + it is (name or IP) and which port you plug it into (1-8). + +4. Serial line(s). When you plug in the serial lines, make sure which + ports on the serial expander they are plugged into. + +B. If the node is of a new type +------------------------------- + +1. You'll need specs on the nodes for the node_types and node_type_attributes + tables. For node_types, you'll just need a name for the type: + + insert into `node_types` (class,type) values ("pc", "pc2800d"); + + For node_type_attributes you will create a row for each of several + attributes. For this, you will need a name for the processor class + (e.g. Core Duo), speed (MHz), RAM size (in MB), boot hard disk type and unit + ('ad' and '0' for IDE, 'da' and '0' for SCSI, 'ad' and '4' for SATA), + boot disk size (in GB), max # of physical cards it holds (including the + motherboard as a card if it has built-in ethernet), and the approximate + amount of time it takes the machine to "power cycle" (in seconds). + + You'll also need to give it a default OS id (the default OS to boot) + and image ID (disk image the default OS comes from), which port is the + control net (e.g. 4) and what its Linux name is (e.g. "eth4"), how many + links this node can delay (usually: num_experimental_links / 2), and + how many virtual nodes ("jails") the machine can support. + + Example: + + insert into `node_type_attributes` values + ('pc2800d','processor','Pentium D','string'), + ('pc2800d','frequency','2800','integer'), + ('pc2800d','memory','2048','integer'), + ('pc2800d','disktype','ad','string'), + ('pc2800d','bootdisk_unit','4','integer'), + ('pc2800d','disksize','160.00','float'), + ('pc2800d','max_interfaces','4','integer'), + ('pc2800d','power_delay','60','integer'), + ('pc2800d','default_imageid','emulab-ops-FBSD54+FC4-STD','string'), + ('pc2800d','default_osid','emulab-ops-FBSD54','string'), + ('pc2800d','control_network','0','integer'), + ('pc2800d','control_interface','eth0','string'), + ('pc2800d','delay_capacity','2','integer'), + ('pc2800d','virtnode_capacity','50','integer'); + + There are also assorted other attributes you need not change, just + use these: + + insert into `node_type_attributes` values + ('pc2800d','delay_osid','FBSD-STD','string'), + ('pc2800d','jail_osid','FBSD-STD','string'), + ('pc2800d','adminmfs_osid','FREEBSD-MFS','string'), + ('pc2800d','diskloadmfs_osid','FRISBEE-MFS','string'), + ('pc2800d','pxe_boot_path','/tftpboot/pxeboot.emu','string'), + ('pc2800d','imageable','1','boolean'), + ('pc2800d','rebootable','1','boolean'), + ('pc2800d','simnode_capacity','0','integer'), + ('pc2800d','trivlink_maxspeed','0','integer'); + + +2. There are several scripts that limit searches to certain classes. + If the new type you have added does not have class "pc", you may need + to include this new class as appropriate. + Some of the scripts that might need to be updated are: + /db/avail.in + /db/nfree.in + /tbsetup/assign_wrapper.in + /tbsetup/batchexp.in + /tbsetup/reload_daemon.in + /tbsetup/exports_setup.in + /tbsetup/snmpit_lib.pm + /www/nodecontrol_list.php3 + /www/reserved.php3 + /www/showexp_list.php3 + /www/tutorial/nscommands.html + /www/updown.php3 + /sql/database-create.sql + +C. What to do on boss: +---------------------- + +1. Insert entries into interfaces table using info from A(1). Try: + insert into interfaces (node_id,card,port,MAC,IP,interface_type,iface) + values + ("pcN",0,1,"00b0d0f01020",NULL,"BSDTYPE","eth0") + +2. Insert entries into wires table using info from A(2). Try: + insert into wires (node_id1,card1,port1,node_id2,card2,port2) values + ("pcN",0,1,"ciscoX",5,1) + + For the control interface do: + insert into wires (type,node_id1,card1,port1,node_id2,card2,port2) + values ("Control","pcN",0,1,"ciscoX",5,1) + + Check to make sure your cards and ports match up with what you + entered in the interfaces table. + +3. Insert entry into outlets table, using info from A(3). Try: + insert into outlets (node_id, power_id, outlet) values + ("pcN","powerX",Y) + +4. Add entries to the nodes table for each node. Try: + insert into nodes (node_id,type,phys_nodeid,role,def_boot_osid,priority,op_mode) + values + ("pcN","pc1u","pcN","testnode","FBSD45-STD",P,'NORMAL') + + P (priority) is the where it gets printed out. These need to be + ascending numbers, and in the right region. See the table. + +4a. Add entries into the tiplines table. The "server" field is where + the actual capture process runs: + + INSERT INTO tiplines VALUES ('pc1','pc1','users.emulab.net',0,0,''); + INSERT INTO tiplines VALUES ('pc111','pc111','tipserv1.emulab.net',0,0,''); + + You need to add the usual lines in /etc/remote on the machine + where the capture process runs. In addition, add a line on users + to that users can use tip to connect to a console on a remote tip + server. So, on users: + + pc111|tbpc111:dv=/dev/tip/pc111:br#115200:nt:pa=none: + + The device field is ignored, but something must be there. + +5. Until you are ready to put it in service, reserve it to an expt, + either with nalloc or by adding an entry to the reserved table + directly. You'll probably also want to put its ports in a vlan to + enable them. + +6. Add the node to the system files: + - DNS: on boss, cd /etc/namedb + co -l emulab.net.db.head + add these lines with all the others: + pcN IN A 155.101.132.N + IN MX 10 ops + IN MX 20 fast.cs.utah.edu. + ci -u emulab.net.db.head + cd reverse/ + co -l 155.101.132.db + in 155.101.132.db, make these changes: + update serial number on line 10 + add entry for node, like this: + N IN PTR pcN.emulab.net. + ci -u 155.101.132.db + run /usr/testbed/sbin/named_setup to update. + - DHCP: on boss, cd /usr/local/etc/ + if you added a new node type, then you need to add a line + of the form: + %%nodetype= + (where is the new type is called) to dhcpd.conf.template. + Then as root run: + dhcpd_makeconf dhcpd.conf.template > Ndhcpd.conf + you can diff dhcpd.conf with the new file to verify nothing + catostrophic happened. Finally: + sudo cp Ndhcpd.conf dhcpd.conf + sudo /usr/local/etc/rc.d/2.dhcpd.sh restart + - tip: on ops or tipserv1, edit /etc/remote + add a line like this: + pcN:dv=/dev/tip/pcN:br#115200:nt:pa=none: + pcN-tty:dv=/dev/cua:br#115200:nt:pa=none: + then do these: + sudo touch /var/log/tiplogs/pcN.log + sudo touch /var/log/tiplogs/pcN.run + - capture: on ops or tipserv1, edit /usr/site/etc/capture.rc: + add a line like this: + /usr/site/bin/capture -r -s 115200 pcN tty >/dev/null 2>&1 & + + +D. How to get the first image on it: +------------------------------------ + +1. If everything is set up right, you can use the magic PXE Flash Floppy + to put the right thing on the PXE card. Edit the BIOS to put the + boot order to Floppy, PXE, Hard Drive, then reboot it. + +2. If everything goes right, you should see it PXE boot and find its + DHCP info, then contact the ProxyDHCP server to get its bootinfo + data, then it should decide according to that what to boot. + +3. If process 3 went okay to that point, do an os_load to try to + install the standard testbed images for the node. + +4. If it doesn't seem to be working just like the others, talk to + Leigh and Mike. + +E. What next +------------ + +1. Test it out and see if it works well enough to put into service. If + its ready, release it into the wild with nfree or by deleting its + entry in the reserved table. + +2. Do some more tests to find any obvious problems. Fix them, if any. + +3. Sit back and relax for a few minutes until the bug reports start + flowing in. + diff --git a/doc/setup-archive/setup-cisco.txt b/doc/setup-archive/setup-cisco.txt new file mode 100644 index 000000000..6b5f7fd4c --- /dev/null +++ b/doc/setup-archive/setup-cisco.txt @@ -0,0 +1,220 @@ +# +# EMULAB-COPYRIGHT +# Copyright (c) 2002-2006 University of Utah and the Flux Group. +# All rights reserved. +# + +##### +##### Configuration suggestions for Cisco switches +##### + +This file contains some configuration guidelines that we (Utah) have found +useful to improve the performance of our Cisco switches. + +All commands given are to be typed at the (enable) prompt on your cisco +switches. They are for CatOS - switches that run IOS may not have these +commands. + + means a list of ports, which on the CatOS command line, can include +lists and rages, such as "3/1,3/2" or "3/1-48" or "3/1-48,4/1-48,5/1-48" + +##### Allowing ports to come up quicker + +This one is useful on both the experimental and control nets: + set spantree portfast enable +Use this on all ports that are directly connected to nodes, servers, power +controllers - anything that is not another switch. Normally, the switch waits +a while (several seconds) when a port first comes up before forwarding traffic +from this port - it does so to prevent loops in the switch topology. The main +place you will see the benefit of this is on the control net - with portfast +disabled, the first few DHCP packets sent by booting nodes will get dropped, +causing the DHCP to take much longer than necessary. + +##### Reducing stray traffic + +Disable spanning tree (STP.) If on, STP sends out packets approximately every +two seconds on every port. You can disable it on all VLANs with the command + set spantree disable all +There are two major consequences (for our purposes) of disabling STP: + 1) You cannot have _any_ loops in your switch topology, or bad things will + happen. + 2) VLAN pruning on trunks won't work, causing broadcast traffic to be + forwarded across trunks that it does not need to cross. We've added + features to snmpit to manually do STP's job in this case, so this + problem is taken care of. +You must have STP disabled on _all_ switches that are trunked together! If it +is enabled on even one, STP traffic will be seen on all of them. + +The switch doesn't trust you to use portfast responsibly. So, it has a +'bpdu-guard' feature that helps guard against loops. Turn off this feature +with the command: + set spantree portfast bpdu-guard disable + +Cisco uses a protocol called 'CDP' to discover other Cisco devices. This sends +out small packets every two minutes. You can disable it with: + set cdp disable +Ideally, you should only disable CDP on ports that don't have other Cisco +devices attached, but in practice, running with CDP disabled on all ports is +fine. + +Switch ports will, by default, try to negotiate trunking and channeling. +Cisco provides a handy macro: + set port host +to disable both of these. Also enables portfast on the ports. + +##### Setting MAC address aging time + +We have found that some experimenters use applications, kernels, etc. that only +receive traffic, not send it. This presents a problem, because it prevents the +switch from learning which port the node is on, and thus broadcasting traffic +for it to every port in the VLAN. This can be solved by 'priming' - ie. having +the receive-only node send some traffic (like an ARP response) at the beginning +of the experiment. However, the default aging time of 300 seconds makes this +impractical. So, we have disabled this aging, making learned MACs permanent +(until the VLAN is torn down.) + +You must do this for each VLAN, with the command: + set cam agingtime 0 +For convenience, we've supplied a file (in this directory) called +'no-cam-aging.cfg' that disables aging on VLANs 2-999 (the ones potentially +used by our software.) Transfer this file to the switch using the: + copy tftp config +command. + +We also suggest that you do this on your control network as well - part of the +booting process leaves the nodes sitting dormant at a boot loader for extended +periods of time, so the switch will tend to forget their MACs. Turning off +aging is not critical, but we suggest it, because it will reduce stray traffic +while the switch re-learns MAC addresses. + +##### Setting up multicast between multiple switches + +If you have more than one switch on the experimental or control networks, you +may need to do a little setup to get multicast between them. The symptom of +this problem is that multicast doesn't work between two nodes on different +switches, and if you run 'show multicast groups' on each switch, some will show +the group as existing, and others will not. + +Run the following command for both sides (ie. on both switches) of every trunk +link: +set multicast router 1/1 +(assuming that port 1/1 is your trunk link). If you are using EtherChannel to +bond together multiple links to form a single trunk, you only need to run this +command for the first port in the channel. + +We had some problems running this command on the trunk on one of our switches: +it failed with the error: +Failed to add port 2/1 to multicast router port list. +What I finally did to resolve this was to tear down the trunk link and +EtherChannel that port was a part of, run the command on it (which succeeded +this time), and then build the EtherChannel and trunk back up. + +##### Setting the clock + +Since bos is an NTP server, you should set your switches to sync time with it. +On CatOS, this is accomplised with: +set ntp server 10.11.12.1 +set ntp timezone MST -7 +set ntp summertime MDT +set ntp summertime enable +set ntp summertime recurring +set ntp client enable +show time + +Of course, you'll need to replace 10.11.12.1 with the IP address your boss node +uses to talk to the switches (usually its control-hardware interface), and +'MST', -7, and 'MDT' with the names of your timezone and its offset from GMT. +If you don't use daylight-savings time, leave out the 'summertime' steps, and +instead do: +set ntp summertime enable + +Watch the output of 'show time' for a while to make sure the clock syncs up. +It may take a few minutes. + +On IOS, these commands are: +configure terminal +ntp server 10.11.12.1 +clock timezone MST -7 +clock summer-time MDT recurring +exit + +... and to see the current time, run 'show clock' + +##### IOS commands + +The above commands are given under the assummption that your switches are +running CatOS. If you are running IOS, here are a few notes that may help you +'translate' the above commands. + +Interfaces in CatOS are named as module/port, while interfaces in IOS are named +as TypeModule/Port - For example, if module 1 has gigabit interfaces, what you +call 1/1 in CatOS is Gi1/1 in IOS. 100Mbit Ethernet is 'Fa'. (Really, these are +'GigabitEthernet' and 'FastEthernet' respectively, but you can abbreviate them.) + +In order to operate on many interfaces at once, you can issue configuration +commands like this: +range gi1/1 - 48, gi2/1 - 48, gi3/1 - 48 +... which would configure all 48 Gigabit interfaces on modules 1, 2, and 3. + +The equivalent of 'set port host' (which sets portfast, disabled BPDU guard, +etc.) is: +switchport host +... applied to an interface or a range of interfaces. As in: +interface range gi1/1 - 48, gi2/1 - 48, gi3/1 - 48 +switchport host +exit + +In order to disable spanning tree, you would use: +no spanning-tree vlan 1-1005 + +In order to create a VLAN and set its name: +vlan 10 +name control-hardware +exit + +In order to set the IP address of the interface in VLAN 10: +interface vlan 10 +ip address 10.11.13.183 255.255.255.0 +exit + +In order to enable an interface: +interface vlan10 +no shutdown +exit + +In order to remove a VLAN: +no vlan 1000 + +To put an interface into a VLAN: +interface gi0/1 +switchport access vlan 10 +exit + +In order to turn on trunking for an interface: +interface gi0/1 +switchport mode trunk +exit + +In order to turn off trunking for an interface: +interface gi0/1 +switchport mode access +exit + +In order to put interfaces into an EtherChannel: +interface range gi1/41 - 48 +channel-group 1 mode on +exit + +(Notes: If you want to make more than one channel, give each set of ports +a different channel number. And, now, you will configure the whole channel +as 'interface port-channel 1') + +To set the native VLAN on a trunk: +interface gi0/1 +switchport trunk native vlan 1 + +To set the read-write SNMP community string to 'public': +snmp-server community public rw + +To globally disable the Cisco Router Discovery (cdp) protocol: +no cdp run diff --git a/doc/setup-archive/setup-db.txt b/doc/setup-archive/setup-db.txt new file mode 100644 index 000000000..b661f1f00 --- /dev/null +++ b/doc/setup-archive/setup-db.txt @@ -0,0 +1,386 @@ +# +# EMULAB-COPYRIGHT +# Copyright (c) 2002-2007 University of Utah and the Flux Group. +# All rights reserved. +# + +##### +##### Setting up tbdb for a new boss node +##### + +Note: we are working on better automating many of the procedures in this +document - for now, though, a few of them are still manual. + +Note: steps labeled "Local Only" are only required when setting up a testbed +with local nodes - they can be skipped in a widearea-only testbed. + +##### Step 1 - Setup users, projects, and experiments + +In order to proceed, you should have the following working (from the boss and +ops setup documentation): + + NFS mounts between boss and ops + Root ssh keys (so that root on boss can ssh to ops without a password) + The web interface + +Make sure you can log into the web interface using the 'elabman' account. +The password for the elabman account is the same as the root password on +your boss node (see, we told you to remember it!). + +This account is created as a testbed administrator, but there is one thing +you will need to do in order to use your admin powers. For the same reason +that you use 'su' and/or 'sudo' on your UNIX boxes instead of logging in as +root, you must explicitly enable admin privileges after you log in. When +logged in as a user who is allowed to become an admin, you will see a green +dot on the left side of the header above the main page content. The green +dot means that although you are allowed admin powers, they are currently +turned off, and you see the same web pages that a regular user sees, and +can use the same actions. If you click on the dot, it will turn red, and you +will have full administrator privileges; we call this 'going red dot'. +If you click on the dot again, it will go back to green, thus you can +easily flip back and forth between normal privs and admin privs. Note +that most of the procedures in this file require you to be in red dot mode. + +Now, we will use the elabman user to bootstrap your first real account and +project. Note that while you will use the elabman account to do this, the +elabman account should not be considered a real account; it is intended to +help bootstrap only and as such, does not have the power to perform many +actions that are required later (such as adding nodes to the testbed). + +Login as user 'elabman' if you have not already done so. Go into 'red dot' +mode by clicking on the green dot. You should see the "Start a New Testbed +Project" page, with a "Create First Project" link on the left menu under +"Experimentation". + +Fill in your own information in the 'Project Head Information' +section. It is important that you provide a working email address! Select +your initial Project Name in the 'Project Information' section (we call +ours 'testbed', but you can call yours whatever you call your project or +research group). Also specify a *working* URL (it is required) for the +project. Submit the form using the Submit button at the bottom of the page. + +The web interface will grind along for a minute or so. DO NOT CLICK THE +STOP BUTTON! When it is done, you will see a message that invites you to +login as the user you just created. Do this now so that you can continue +with setting up your testbed. Note that the elabman account has been +deactivated during this process to avoid problems later on (and potential +security breaches). + +Before we continue, lets explain a few more important items: + +* Project Membership: In addition to the project you just created, you have + automatically been added to the "emulab-ops" project with trust value + "group_root". This allows you to approve new members to that project as + well as your own project. + +* Admin Mode: Your new account has been given "administrator" mode, as + described above. To change that value for other users after their + accounts are created, you can do this on boss: + + echo 'update users set admin=1 where uid=""' | mysql tbdb + +* Shell on Boss: Give yourself the special ability to login to boss; + in contrast, most (normal) users have a restricted shell on boss, + and are not allowed to log in using a password. Login to boss as root, + and edit the password file using the 'vipw' command (FreeBSD requires + some special processing on the password file after editing, which vipw + does.). Give yourself a real shell (say, /bin/csh) and then exit the + editor. Then give yourself a password (in general, it is safer to have a + different password on boss then on ops!). Use this command: + + passwd + + NOTE: See doc/shelloboss.txt for important security issues w.r.t. giving + real shells on boss. Before you give a real shell to someone, it is a + good idea for them to read this file! + +* Now logout and back in as yourself. In general, it is safer and better to + not do things as root. In fact, many testbed programs will complain if + you invoke them as root because it makes accounting and auditing more + difficult. + +* Unix Group Membership: The Emulab account system manages both the + password file and the group file (/etc/group) on both boss and ops. If + you edit them directly, those changes will likely be lost. If you want to be + a member of any UNIX groups on boss, use our 'unixgroups' command. For + example, to add yourself to the "operator" group, do this on boss (as + yourself, not root): + + withadminprivs unixgroups -a operator + + NOTE: Your initial account created above was already placed in the wheel + and tbadmin groups. + + NOTE: Just as you need to go 'red dot' to use admin privileges on the web + interface, you must also explicitly enable them on the command line. To + do this, prefix the command you want to run with 'withadminprivs', + which can be abbreviated as 'wap'. + +* Set your path: withadminprivs and many other admin-type commands live in + /usr/testbed/sbin - you'll want to put this and /usr/testbed/bin in your + $PATH. + +Others at your site can now apply to join your project, or start their own. + +##### Step "-1" - Undoing Step 1 if necessary + +If something went wrong during Step 1, it can leave things in an inconsistent +state. Here's how to undo it without starting over from scratch. The goal is +to remove things that boss-install checks on, so it can be run again to put +the Emulab database and directories into initial conditions. + + * Remove users, groups, and directories set up by boss-install. + pw userdel -n elabman -r + pw userdel -n elabckup -r + ssh -n ops pw userdel -n elabman -r + ssh -n ops pw userdel -n elabckup -r + + ls -l /users /proj /groups + rm -r /proj/* /groups/* + mkdir /proj/cvsrepos + pw groupdel -n emulab-ops + ls -l /usr/testbed/{expwork,*/proj} + rm -r /usr/testbed/expwork/* /usr/testbed/*/proj/* + ls -l /usr/testbed/{expwork,*/proj} + + * Remove user and group from the elabman "Create First Project" pages. + set me = *your-login* + set us = testbed + pw userdel -n $me -r + pw groupdel -n $us + ls -l /users /proj /groups + + * Also remove any other users and groups you've created since then. + Otherwise you will get this message until you've cleared them: + 'Error Creating Project: Transient error; please try again later.' + tail /etc/passwd + tail /etc/group + set him = *user* + set them = *group* + pw userdel -n $him -r + pw groupdel -n $them + + * Kill the database. + mysql -e "drop database tbdb" + + * Run boss-install, checking particularly on the success of these tasks: + . Setting up database + . Setting up initial user (elabman) + . Setting up checkup user (elabckup) + . Setting up system experiments + +Now you can go back to Step 1 with the first login as 'elabman' and try again. + +##### Step 2 - Setup web sql editor + +Several of the steps below require you to add data to the database by hand. If +you're comfortable with SQL, you can do this directly with SQL queries, or you +can use the generic web-based SQL table editor provided with the testbed +software. If you plan to use the former method, you can skip this step. + +********************************** WARNING ********************************** +* Many tables depend on data in other tables, or depend on programs running * +* to effect a change. Thus, you should not edit tables other than the ones * +* described in this document. * +* You have been warned...... * +********************************** WARNING ********************************** + +First, you'll want to protect the webdb script from outside browsers. Because +of its flexibility, it would be quite dangerous if it were broken into. So, we +add an additional layer of protection by limiting the IP addresses it may be +used from. Open your httpd.conf file (located in /usr/local/etc/apache) +and find the 'Directory' directives. Add a section such as this: + + + AllowOverride None + order deny,allow + deny from all + allow from 155.99.212. + + +If you installed the testbed tree somewhere other than /usr/testbed, fix the +directory. Change the 'allow from' line to match your IP subnet (note the '.' +on the end of the address, to match the entire subnet). You can have as +many 'allow' lines as you want. Restart apache: + + sudo /usr/local/etc/rc.d/apache.sh stop + sudo /usr/local/etc/rc.d/apache.sh start + +Next, you'll need to specify which users have the ability to edit the database. +This is done with the 'dbedit' column in the users table. You can turn on +a user's dbedit bit like so: + + echo 'update users set dbedit=1 where uid=""' | mysql tbdb + +##### Step 3 - Setup switches +##### Local Only + +1) Create node types for switches + +Add entries to the node_types table for each type of switch you'll be +using. You can do this by talking to mysql directly, but is more easily +accomplished using the web interface. In 'red dot' mode, go to: + + https:///editnodetype.php3?new_type=1 + +These are the switch types currently supported: + cisco{65xx,40xx,45xx,29xx,55xx} + For example, if you had a 6509, you'd enter 'cisco6509' + If your switch runs IOS (instead of CatOS), append '-ios' to the + type. + intel + The supported type is the 510T (but just put 'intel' in the type field) + nortel1100, nortel5510 + foundry1500, foundry9604 +(Note: Case sensitive!) + +Set the "class" to 'switch' and set "processor" to whatever you used for the +"type" field. + +Most of the other columns are not important for switches (so you can set +them to 0), but putting in "Max Interfaces" (if the switch is expandable) +can be useful for your own information. + + +2a) Create interface types for switch interconnects (if any) + +If you'll be connecting the experimental switches together, you'll add +interface types for the ports you'll be using to link them together. These go +into the interface_types table. Make up something descriptive for the 'type' +column (no spaces). Make sure to set the max_speed (in Kbps) and full_duplex +(1 for full duplex, 0 for half duplex) columns correctly. As an example: + + insert into interface_types set + type='cisco_gigE',max_speed=1000000,full_duplex=1, + manufacturuer='Cisco',model='WS-X6316-GE-TX',ports=16, + connector='RJ45'; + + +2b) Create interface capabilities entries for switch interconnects (if any) + +If you added any entries to interface_types in the previous step, you also +need to add entries to the interface_capabilities table. Using the same +type name you used above, add a capkey for "protocols", usually set to +"ethernet" and another capkey for "ethernet_defspeed" to the same value you +used in the interface_types table. Do this for each type you entered in +step 2a above. For example: + + insert into interface_capabilities set + type='cisco_gigE',capkey='protocols',capval='ethernet'; + insert into interface_capabilities set + type='cisco_gigE',capkey='ethernet_defspeed',capval='1000000'; + +3) Create switches in nodes table + +Next, add the switches to the nodes table. The only necessary fields here are +node_id (choose one appropriate to the switch), type (use one of the ones you +created earlier,) and role. Role, for switches, should be either 'testswitch' +or 'ctrlswitch', depending on whether you're using it for the experimental +or control network. If it's used for both, call it a 'testswitch'. Example: + + insert into nodes set + node_id='cisco1',phys_nodeid='cisco1', + type='cisco6509',role='testswitch'; + +The node_id you select must resolve in DNS and/or boss's /etc/host file - we'll +use this name to contact the switch for SNMP. Ie., you must be able to ping the +name you select as the switch's node_id. + +4) Add state for switch interconnects + +If you'll be connecting the experimental switches together, add interfaces to +each of them. Use the interface types you created above for them, and be sure +to get the correct card and port numbers. In modular switches, such as the +Cisco Cat6500 series, use the switch's 'native' module and port numbers. For +switches that do not have expansion modules, the card number is assumed to be +0. Make sure to get the current_speed (this time, in Mbps, and note, this is +an enumerated type, not an integer, so you need to put quotes around the +speed) and duplex correct. You'll also need to set 'iface' to some string +representing the port - I tend to use 'module/port' (ie. '1/1'). Set the 'role' +of the interfaces to 'other'. + +Now, go into the wires table and add wires connecting the switch interfaces you +just created. Make sure to set the type to 'Trunk'. In this case, it doesn't +matter which switch you use as node 1 (the node_id1, card1, port1 columns), and +which you use as node 2. If you have a non-modular switch, all ports are +considered to be on card 0. + +7) Create switch stacks + +Into the switch_stack_types table, create entries for each stack in your +network. A stack is set of switches that share common VLANs. Usually, your +experimental switches will be one stack, and the control network another. You +should create a stack even if there is only one switch in it. The control +stack should be named 'Control', and the experimental stack 'Experiment' (make +sure to get the capitalization right). The 'leader' field should be set to the +node_id of the master switch in the stack. The stack_type column is used by +snmpit to determine which module to use to talk to the stack. Most sites +should set the stack type to 'generic', which handles all switch types +currently supported by Emulab. If you have all Cisco switches, and are having +problems with the 'generic' stack type, you can try setting the type to +'cisco'. Making a stack for the control network is optional, but will enable +control-net firewalls. There are a few columns in this table that you will need +to set: + +supports_private: (Cisco only) This switch can make private VLANs - probably + best to leave it as 0 +single_domain: (Cisco only) Means that all switches in the stack use VTP to + maintain a common set of VLANs. This means that we will only create VLANs + on the stack leader - if set to 0, we create VLANs on each switch + individually +snmp_community: The SNMP community string we use for read-write access to the + switch. Most switches use 'private' by default. +min_vlan: The smallest VLAN number that the testbed software will use. If left + NULL, defaults to 2. +max_vlan: Ditto, but the biggest number. Defaults to 1000. + +Finally, add switches to these stacks by putting entries in the switch_stacks +table, listing the node_id of each switch and the stack_id of the stack it +belongs to. You can leave the 'is_primary' column of these rows with its default +value (1). + +After getting this set up, run 'wap snmpit -l', and make sure it doesn't give +you any error messages. If it tells you that your type of device isn't +supported, ask Utah - we have a list of supported devices in snmpit, but they +are just the ones we've tested. Hopefully, yours will work too, we just haven't +tried it. + +Your snmpit output should look something like this: + +%wap snmpit -l +VLAN Project/Experiment VName Members +-------------------------------------------------------------------------------- +Control- myswitch:1 + +Note that 'Control-' is a truncated version of the name 'Control-hardware' that +you created in setup-network.txt . If you see any other VLANs on this list, +that's not a problem, but if you want to delete any of them, you can do that +with 'wap snmpit -o '. + +##### Step 4 - Setup control hardware +##### Local Only + +1) Power controllers + +Follow the instructions in setup-power.txt. + +Setting up the power controllers before you add your nodes makes adding +the nodes remotely much easier. The process being: power off all nodes, +then power on what you want to be the first node. It should show up in +the "new node" page on the Emulab web portal. You can then add it as "pc1". +Then power on the second node, add it, etc. + + +##### Step 5 - Images and OSids + +These will depend on the image(s) and any OSKit kernels you've recieved, or +built yourself. Since you're probably using disk images from Utah, the best +thing to do is ask us for the appropriate lines for the os_info and images +tables. Make sure to get the PXEBOOT, PXEFRISBEE, and PXEFBSD OSIDs, which are +required to load disk images. For a widearea-only testbed, no images entries +in the images table a required, but OSIDs are still useful, to determine what a +node's capabilities are. + +##### Step 6 - Setup nodes + +To add nodes to the testbed, see setup-nodes.txt . diff --git a/doc/setup-archive/setup-fs.txt b/doc/setup-archive/setup-fs.txt new file mode 100644 index 000000000..0f5148692 --- /dev/null +++ b/doc/setup-archive/setup-fs.txt @@ -0,0 +1,247 @@ +# +# EMULAB-COPYRIGHT +# Copyright (c) 2002-2006 University of Utah and the Flux Group. +# All rights reserved. +# + +##### +##### Setting up the Utah Network Testbed software +##### Most recently tested on FreeBSD 4.11. +##### + +##### Step 0 - The Goal + +In the past we have combined the filesystem function with the ops/users +functions. However, we recognize that people may have pre-existing, dedicated +(and usually very expensive :-) file server machine that they would like to +take advantage of. Or for performance reasons, you may want to separate +the filesystem service from user activities on the ops node. + +So we have made a start toward separating the two. The general guiding +principle is to keep modifications to the fileserver box to a minimum, +since the ability to control the box may be greatly reduced; for example, +the fileserver might be a NetApp box or running MacOS X or Windows. +At a bare minimum however, we still require that you be able to manipulate +the access control lists (e.g., the BSD "exports" file) from the boss node. +Optionally, you may need to be able to monitor and manipulate disk quotas. + +But that is all theory right now. In the current state of affairs, the +fileserver box still has to run FreeBSD and is assumed to be dedicated to +Emulab use (e.g., the Emulab boss node assigns uid/gids itself, with no +attempt to synchronize with pre-existing uid/gids on the fileserver). + +With this caveat in mind, we can begin. + +##### Step 1 - OS installation and filesystem setup + +Install FreeBSD on the machine you'll be using for your fs node, using the +standard FreeBSD installation process. When asked by the installer, it's best +to choose the 'Developer' distribution set - this gets you full sources. When +it asks if you want to install the ports collection, answer *no*. Do not +install any packages at this time - you'll get a chance to later. You'll +need to partition your filesystems so that you have the proper amount of +space for certain directories - see below for details. + +Make sure that you have the network correctly configured. + +The following directories will need to exist on partitions that have enough +space to hold them: + +/usr - Needs space for the ports tree and a system object tree. + At least 10GB. Be sure to build with plenty of + inodes (the ports tree itself uses about 200000, so + be safe and build with at least a million). +/usr/testbed/ - Needs space for testbed software and logs. For a dedicated + fileserver machine, this won't be very much, 100MB + should be plenty. +/users/ - Needs space for user home directories. Amount of space required + depends on how many users you expect to have. + Generally, though, we suggest that users store large + files related to their projects in the /proj directory. +/proj/ - Needs space for project files. We recommend that this be larger + than /users, to encourage people to store files here, + which aids per-project accountability. +/groups/ - Needs enough space for files shared by the sub-groups of + projects. These are primarily used by classes, if any. +/scratch/ - Optional, large filesystem of "scratch" space. The intent + is that this filesystem provides per-project space + that is not "guaranteed" (for the Utah Emulab that + means we do not back it up to tape). If used, + you would either set no quotas, or higher quotas + than for /proj. +/share/ - Exported read-only to all nodes, we use it for providing to + experimenters the source for the FreeBSD and Linux + versions we run as well as common packages and RPMs. + This could require anything from 1GB to 20GB+ depending + on what you want to make available. + +You may want to enforce quotas on the user-writable filesystems. This is the +main reason you'd want to keep them in separate filesystems (i.e., so people +can have different /users/ and /proj/ quotas.) If you do not think you will +ever use quotas, then you could make /users and /proj part of the same +filesystem. + +As mentioned /scratch is optional. If you are not providing "guarantees" +such as filesaved or RAIDed disk space and you are not using quotas, you +might as well just put all your space in /proj and not define FSDIR_SCRATCH +in the defs file. + +Note also since /share is exported read-only, FreeBSD requires that it be on +a separate filesystem from anything that is exported read-write. So while +/users, /proj and /groups can be on the same filesystem, /share cannot. + +Make sure that, no matter how you decide to partition things up, you make +symlinks to the appropriate places. ie., if you make one big filesystem called +/z that has /users, /proj, and /groups in it, make sure you: + + ln -s /z/users /users + ln -s /z/proj /proj + ... etc. + +In other words, we assume the existence of /users, /proj, /group and /share +(but not /scratch). + +Do *not* create any user accounts, Emulab does not require that its users +have login accounts on the fileserver. For the purposes of this setup, just +log in as root. You can manually add login accounts for Emulab admins later +if you desire. + +This would be a good time to reiterate that Emulab currently assumes that it +has complete control of the uid/gid namespace. However, it starts assigning +uids at 10000 and gids at 6000, so pre-existing accounts with values lower +than those should be ok. Other accounts will have to be removed or manually +synchronized with Emulab later. + +##### Step 2 - Installing packages + +To make sure that you're running with versions of software from the ports +collection that are known to work with our software, and to save you hours +of compile time, we provide pre-built binary packages of the ports required +by Emulab. + +(Do not let the names of the following tar files bother you, e.g. +"FreeBSD-4.10" or the apparent 20041102 date stamp. These are indeed +the correct files to use with FreeBSD 4.11 and with the current (May 2005) +Emulab distribution.) + +Download the packages tarball from: + + http://www.emulab.net/downloads/FreeBSD-4.10-20041102.tar.gz + +(You can use the FreeBSD 'fetch' command to download the file.) + +Now, untar this someplace (you need about 70MB of space, so don't use /tmp). +Let's say you untarred it into /usr/tmp. You would then run: + + env PKG_PATH=/usr/tmp/FreeBSD-4.10-20041102 pkg_add emulab-fs-1.4 + +Of course, if you untarred it somewhere else, put in the correct path. + +Now you need to download a ports tree that corresponds to the above +packages. We have run into many, many problems with versions of the +FreeBSD ports. So, rather than using the /usr/ports tree that comes on the +FreeBSD installation media, we use one that we've tested against. You can +grab our 'approved' copy of the ports tree from: + + http://www.emulab.net/downloads/ports-20041102.tar.gz + +Untar it, move it into place as /usr/ports (rename the old directory to +ports.old, or just remove it). NOTE: You must download and copy the ports +tree into place, even if you do not intend to install any packages yourself. + +##### Step 3 - Unpacking and running configure + +At this point, you'll need to make a 'defs' file - You will use the same +defs file on boss, ops and fs. See defs-example in the top level directory +as a starting point. You want to make sure each of the variables has a +definition that makes sense in your environment. Feel free to ask Utah if +something is not clear. Of particular note for the fs node is the setting +of FSDIR_* to match the filesystem layout from Step 1, and the setting of +FS_WITH_QUOTAS if you intend to use quotas on any filesystem. + +Unpack the testbed source, and run it's configure script. A good place to +unpack the source code is /usr/testbed/src/testbed. You will use the +--with-TBDEFS option to configure to give it the path to your defs file: + + mkdir -p /usr/testbed/src/testbed + mkdir -p /usr/testbed/obj/testbed + cd /usr/testbed/src/testbed + tar .... + cd /usr/testbed/obj/testbed + /usr/testbed/src/testbed/configure \ + --with-TBDEFS=/path/to/your/defs-file + +Typically, you would store your defs file in the source tree along with +the other defs files that came in the tarball. + +##### Step 4 - Running the fs installation script + +In the object tree you've configured (say, /usr/testbed/obj/testbed), there's +an 'install' subdirectory, with a script called 'fs-install'. Just run this +script as root (note the same package directory argument as above). + + cd install + env PKG_PATH=/usr/tmp/FreeBSD-4.10-20041102 perl fs-install + +It will take care of installing any additional ports, and doing various +other configuration of FreeBSD required to make it into an fs node. The +script is designed so that you can run it as many times as you want, and +it'll just skip stuff it's already done. If it fails, send the output to +Utah so that we can fix it up. If it succeeds, follow any other +instructions it may have. The script will tell you to reboot the machine, +but you can wait until after you do the next step to do so, if you want. + +(You may have to set the executable bit on this script, since configure won't; +we'd like to get this fixed at some point.) + +You should be aware that, among other things, this script sets password-less +'sudo' for anyone in the 'wheel' group. If you don't want this for security +reasons, you can undo them after the installation script is finished. + +##### Step 5 - Installing from source + +To install the actual testbed software, simply do a 'gmake fs-install' in your +object directory. + +(Note: If you're logged in as root, /usr/local/bin, where gmake lives on +FreeBSD, may not be in your path.) + +##### Step 6 - Quotas (optional) + +[ Note that this section is FreeBSD specific. ] + +It you are planning to run quotas for Emulab users on the fs node, you +will need to establish a default quota value for all users. Ideally, +the quota would be configurable per-user, but for now all quotas are +initialized from a "prototypical" user. Note that you can change individual +user quotas later by running edquota(8) on the 'fs' node. To establish +the default quota values, you will need a "prototype user" to which to +apply the quotas. You will probably want to add a special user, say +'elabman', for this purpose. The uid for this user should be the MIN_UNIX_UID +value specified in your defs file (10000 by default). The gid doesn't matter, +but needs to be a value that already exists in the /etc/group file. The +prefered strategy is to first add the "tbadmin" group to /etc/group with +gid 101, since this Emulab-specific group is used on the ops node as well. +Assuming you have done this, you would do: + + pw useradd elabman -u 10000 -g tbadmin -m -d /users/elabman -h - -s /bin/nologin + +Now set the quota for that user on each quota-enabled filesystem, e.g.: + + edquota -e /proj:2000000:2000000 -e /users:1000000:1000000 elabman + +would set a 1GB quota on /proj and 512MB on /users for the prototype user. +Once the prototype user quotas are established, you can do: + + edquota -p elabman 10000-15000 + +which would automatically apply the elabman quotas to any user created with +uids between 10000 and 15000, assuming that you wanted to allow up to 5000 +users. + + +##### Step 7 - Other miscellaneous things to set up + + [Nothing at this time] + +Once you're done with all of this, reboot fs. diff --git a/doc/setup-archive/setup-network.txt b/doc/setup-archive/setup-network.txt new file mode 100644 index 000000000..4d7e423a0 --- /dev/null +++ b/doc/setup-archive/setup-network.txt @@ -0,0 +1,148 @@ +# +# EMULAB-COPYRIGHT +# Copyright (c) 2002-2005, 2007 University of Utah and the Flux Group. +# All rights reserved. +# + +##### +##### Laying Out Your Network +##### + +The purpose of this document is to aid in designing and setting up the control +and experimental networks on other emulabs. + +First a bit of background: On the control network will be your control +nodes (boss, ops, any tipservers, etc.), control hardware (SNMP-controllable +devices such as power controllers and switch IP interfaces), your connection to +the outside world, and the control interfaces of your nodes. So far, we haven't +tried to distribute the control net across multiple switches, but this should +be theorically possible. You'll need to set up the VLANs, etc. on the control +net by hand. + +The experimental network will consist of one or more (this, we have tested) +switches connected with trunk (802.1q, or proprietary, like Cisco ISL) lines. +Most configuration of these switches will be taken care of by our software. +(See setup-cisco.txt for some configruation options you many want to apply to +these switches if they are Ciscos.) + +##### Splitting up the control net + +We basically have 5 VLANs on the control network: +'External' contains our connection to the outside world +'Private' contains the boss node, and our tape backup server +'Public' contains our ops node +'Control' contains the control net interfaces of all experimental nodes +'Control-hardware' contains all IP-controllable devices (namely, power + controllers and switch IP interfaces, as well as a second + interface on the boss node. This VLAN uses private IP addresses, + and does NOT contain a router interface. + +This is done for security - we route (using an 'L3 switching' module in our +control-net switch) between these VLANs, and do some firewalling between each +of them. You could also do the routing with a real router or a PC. The main +goals are: +1) Protect both control and experimental nodes from the outside world (and + vice-versa - we don't want people attacking the outside world from our nodes) +2) Protect the control nodes from the experimental nodes +3) Protect the control hardware (power controllers, etc.) from nodes + and the outside world +4) Protect the boss node (which is _not_ publicly accessible) from the ops + node (which all experimenters have shells on.) + +Now, it's entirely possible to combine these VLANs into one big one - this is +what we've done on our mini-testbed here. But, there are some serious security +implications with doing it this way - namely, that the nodes can theoretically +impersonate each other, power cycle each other, and all kinds of nasty things. + +At the very least, you should have a firewall between your testbed and the +outside world, to satisfy #1. + +It is also a good idea to separate the nodes' control net into a separate VLAN, +which satisfies #2 and #3. After all, you are giving people root access to the +experimental nodes. In situations where you are only giving access to a small +number of trusted people, this is probably not too big a deal, but once access +gets outside the small circle of your friends, or if you are allowing students +access, then taking these precautions are a very good idea. + +If possible, putting boss and ops on separate networks is also a good idea +(#4), but is probably the least important part to worry about. + +Basically, it's up to you to decide how much security you want to worry about. +I'd recommend at least going after #1, #2, and #3. We've put a lot of thought +into this, so if you're wondering how your choice here will affect other +aspects of security, ask Utah, and we can probably help you. + +Since our firewall rules change frequently as we add new services, it's best to +contact Utah and ask us for the current set. + +Note: For compatibility with Emulab's current control-network firewalling +code, and possible future improvements such as inter-experiment control +network isolation, you should make sure to name the control network +'Control' (case sensitive). In fact, we recommend keeping all 5 VLANs +named as we do, for ease of communication among testbed admins, +if nothing else. + +##### Connecting the control net to the experimental net + +In order to be able to control the experimental switches (ie. create new VLANs, +etc.) you need a way to talk to them. So, you'll need one line that goes from +your control net to the experimental net. However, you will need to be _very_ +careful that no experimental traffic can leak across this line, and that it +cannot be used as a 'back door' into the private net by nodes. + +On the control switch, it should be put into the 'control-hardware' VLAN. + +On the experimental switch, it should be in the same VLAN as the switch's IP +interface. The best thing to do is create a new VLAN, and move the switch's IP +interface into it. On CatOS, you accomplish this with: + set interface sc0 + +You will also need to give the experimental switch(es) IP addresses. With +CatOS, you do it like this: + set interface sc0 + set interface sc0 up + +##### DHCP through the router + +If your boss node is on a separate VLAN from the node control net, you'll need +to make sure that DHCP traffic can get from the control net to your boss node, +since normally, DHCP is not forwarded through routers. On Cisco routers, this +is done with the 'ip helper-address'. For example, here, the name of the +router's interface in the node control net is 'Vlan3'. So, I'd log into the +router, and run the following: + configure terminal + interface Vlan3 + ip helper-address 155.101.128.70 + +Of course, replace 'Vlan3' with the name of your router's node +control-net interface, and replace the IP address with that of your boss node. + +##### IGMP snooping on the control net + +In order for multicast to work correctly, you need to make sure that IGMP +snooping is enabled on the control switch. This is needed for frisbee, our +disk-loading system. It's up to you whether you want to enable this on the +experimental switches. In general, we recommend it so that your experimenters +can use multicast, but it does seem that unexpected or malformed multicast +packets have an easier time DOSing the switch control processor than unicast +traffic. On CatOS, the command is: + set igmp enable + +#### VTP domains + +If you have only one Cisco experimental-net switch, you don't need to have it +using VTP, which is a mechanism for keeping VLANs in sync across multiple +switches. You do this (in CatOS) with: + set vtp mode transparent + +If you have multiple experimental switches connected by trunk lines, you should +use VTP. Pick a domain name (we call ours simply 'Testbed',) and run the +following on all of your switches: + set vtp domain + +Pick one switch to be the master - it doesn't really matter which one. (See the +switch setup instructions in setup-db.txt, and make sure you name the stack +after the master switch.) On this one, run + set vtp mode server +On all the others, run + set vtp mode client diff --git a/doc/setup-archive/setup-nodes.txt b/doc/setup-archive/setup-nodes.txt new file mode 100644 index 000000000..78b3cb86e --- /dev/null +++ b/doc/setup-archive/setup-nodes.txt @@ -0,0 +1,275 @@ +# +# EMULAB-COPYRIGHT +# Copyright (c) 2002-2007 University of Utah and the Flux Group. +# All rights reserved. +# + +##### +##### Setting up nodes for use in the testbed +##### + +This file explains how to get nodes set up and added to the testbed. + +##### BIOS setup on the nodes + +First, we need to get some things set up in the nodes' BIOS. For now, just do +this on one of the nodes, you'll do the rest later. + +Booting from PXE - +The first thing we'll need to do is have the node boot from PXE on its control +network interface. This is how the testbed exercises control over what the +node will boot. In most BIOSes, this should be as simple as finding the boot +order options, and putting PXE on the top. Things can get a bit confusing if +you have more than one PXE-capable interface, because the BIOS often provides +no way of distinguishing between them - you'll have to do some trial-and error +to figure out which is which. + +Disable PXE on experimental interfaces - +Nodes will boot much quicker if you disable PXE booting (through whatever means +provided by your card) on experimental net interfaces. + +Set power-loss behavior - +Many BIOSes have an option about what to do after a power failure (which is +what it looks like to the node when it gets power cycled by a power +controller.) They're usually 'always off', 'always on', and 'last state'. +Always on is the best - last state is OK, but if someone does a 'shutdown -h' +on the node, you can't bring it back up with power cycling - you have to go +punch the power button. Just make sure they're not set to always off. + +#### Type information for the nodes + +Unless you're adding some more nodes, identical to the ones you already have, +you'll need to put some type information about them into the database. You can +do this through the web interface: log in and go into admin mode. Now, click on +the 'Node Status' link on the menu, and use the 'Create a new type' link. The +important things on this page to fill out are: (You can leave the defaults for +the rest) + +* Type - We typically name types 'pcXXX', where XXX is a short (a few + characters) string describing the nodes (such as processor + speed, chipset, etc.) eg. pc600 for 600-MHz nodes + +* Processor - Class of processor (eg. 'Pentium IV') + +* Speed - CPU speed in MHz (eg. 3000) + +* RAM - Amount of RAM in MB (eg. 256) + +* HD - Hard disk size in GB (eg. 20) + +* Max Interfaces - Maximum number of NIC ports (eg. dual port cards + count as 2) + +* Control Network - Interface number (described below) of the control network + interface. + +* Control Network Iface - Name of the control network interface under Linux - + usually, just a concatenation of 'eth' and the Control Network number + you entered above. (eg. 'eth0') + +* OSIDs - If Utah has already given you disk images (and their + associated database state), then select those at this stage. You + should have only one choice for the ImageID. For the Default OSID, + select either Linux or FreeBSD, depending on what you think your + users are likely to want by default. For the time being, both the + delay and jail OSIDs need to be FreeBSD. If Utah has not given you + images yet, come back and set the OSIDs once they have. + +* Delay Capacity - How many delay nodes this node can be. For example, + nodes with 2 experimental interfaces can be 1 delay node, nodes + with 4 experimental interfaces can be 2 delay nodes, etc. If your + nodes are not especially "beefy" then you might not want to set + this number to the maximum it could be. + +* Disk Type - FreeBSD-style disk name for the primary hard drive. + Choices are 'ad' (IDE), 'sd' (SCSI), or 'ar' (IDE RAID). + +You'll also need to add entries to the interface_types table (using the web +SQL editor, or SQL directly) for each type of network card you are using. +Notes on the columns: + +* type - Name of the FreeBSD driver for the card (common ones are 'fxp' for + Intel EtherExpress Pro 100, 'em' for Intel Pro 1000, and 'bge' for + Broadcom gigabit). + +* max_speed - The maximum speed of the interface, in Kbps. Likely to be + either 100000 for 100Mb or 1000000 for Gb. + +* full_duplex - 1 if the card can operate in full duplex, 0 otherwise. + +Note that you should only specify 1000000 (Gb) for the max_speed if the +card is Gb capable *and* connected to a Gigabit switch. If a Gb capable +card is going to run at 100000 (100Mb) instead, then it is best to add +a seperate type such as 'em100' (for the 'em' type) to describe that +configuration. This way, if you use a Gb switch in the future, you can +change the type fields in the interfaces table to be 'em' instead of 'em100'. + +As an example, to insert an entry for an "fxp" interface: + + insert into interface_types set + type='fxp',max_speed=100000,full_duplex=1, + manufacturuer='Intel',model='EtherExpress Pro',ports=1, + connector='RJ45'; + +Once you have all of the interfaces specified, you need to insert the +following interface_capabilities table entries, *for each* new interface +type you entered above: + + insert into interface_capabilities set + type='fxp',capkey='protocols',capval='ethernet'; + insert into interface_capabilities set + type='fxp',capkey='ethernet_defspeed',capval='100000'; + +Again, if the interface is Gigabit capable and connected to a Gb switch, +capval above should be 1000000. For example: + + insert into interface_capabilities set + type='em',capkey='protocols',capval='ethernet'; + insert into interface_capabilities set + type='em',capkey='ethernet_defspeed',capval='1000000'; + +In such a configuration, you will also want to add: + + insert into interface_capabilities set + type='em',capkey='ethernet_auxspeeds',capval='100000'; + +This allows a Gb 'em' interface to be configured in an experiment as 100Mb +without the use of a delay node; i.e., by setting the switch and interface +characteristics to 100Mb directly. In theory you could use this to +configure 10Mb links without a delay node as well, but in our experience, +we have had too many problems getting switch and node to communicate when +hardwiring at 10Mb. So we just use delay nodes to handle 10Mb. + + +##### Bringing up the first node + +We'll start by bringing up the first node in the testbed, to make sure things +are working, and so that you can set some initial values. If we haven't already +given it to you, ask Utah for the 'newnode' MFS. + +What's going to happen as we bring nodes up is that they should boot into +the 'newnode' MFS, which is a stripped down version of FreeBSD that runs +out of a memory filesystem. This MFS reports in to boss, informing it of +it's existence and key things such as it's MAC addresses. Do the BIOS setup +detailed above on this node, and fire it up. By the time it's got a +FreeBSD login prompt on the console, it should have reported in. This will +send mail to the local testbed-ops list. + +Now, let's take a look at the web page where nodes that have checked in, and +are awaiting creation as 'real' nodes, show up. Log into the web interface as +an admin (make sure to go 'red dot'). Now, go to the 'Add Testbed Nodes' link. +Clicking on the numeric ID next to a node will bring up a page with more +information about the node, which you can edit. You can select nodes with the +checkboxes along the left side - actions taken by the buttons below operate on +the selected nodes. + +WARNING: Nothing on this page asks for confirmation, so be careful where you +click. + +On this page, you should now see the first node you booted up, which should +have gotten the name 'pc1'. Click on the ID number (which is probably '1') to +see more detail. Make sure that the number of interfaces reported is correct. +Note that the 'Temporary IP' shown on this page is the dynamic one assigned to +the node by DHCP, from the dynamic range you set up in the defs file. If +you need to SSH into to it to check things out, until it's be really added +to the testbed, use this IP. + +Next, make sure that the 'Type' column is filled in with the one you entered +into the types table earlier. If it isn't, fix that now using the 'Set Type' +box. + +Next, set the node name to your preferred naming scheme. We strongly suggest +leaving it as-is (ie. using pcXXX to name the nodes), but if you must change +it, do so now - nodes you add later will get a name based on this one, with the +number and the end incremented. The code that does this guessing supports node +naming schemes that end in numbers, or end with '-a'. + +Now, you'll need to set the IP address for this node. Subsequent nodes will +have their IP addresses computed from this one. (ie. if you set pc1 to an +address that ends in '.1', pc3 will get address '.3'). + +The other thing to check here is to see what order the interface got detected +in. Unfortunately, FreeBSD and Linux sometimes detect them in different orders. +If you will usually be running Linux on the nodes, you probably want to +re-order them to the Linux order so that the database state will make more +sense to you. At this point, figure out the mapping from the FreeBSD order to +the Linux one, and write that down. (If necessary, you could boot the node up +from Knoppix, or some distribution's install floppy/CD to determine the Linux +order - use MAC addresses to map this to the FreeBSD one.) + +If you somehow make a mistake and need to delete a node that's been created, +use the 'deletenode' script on boss. For example, to delete pc1 from the DB: + + withadminprivs /usr/testbed/sbin/deletenode pc1 + +##### Bringing up the rest of the nodes + +Okay, now that you've got the first node up, the rest should be easy. Bring the +second node (pc2) up, just like you did the first one. Check to make sure that +it got an appropriate name and IP address, extrapolated from the first one. If +that works, start bringing the others up in order. It's important to do them in +order, because identifying which is which if you do them out of order can be +very painful! If there are some nodes you simply can't bring up, because of bad +hardware, etc. write these down, and we'll fix things up later. + +Important note: Remember the size of the dynamic range you picked for dhcp +in the defs file? Well, that will limit how many of nodes you can +bring up at a time. If you run out of IP addresses, continue on with the +nodes you have up, and repeat these steps later with the remainder. + +Okay, got all the nodes up? Good. At this point, you can fix things up for any +you had to skip, using the 'Add to Node ID suffix' box. If, for example, you +couldn't boot pc10 select all the nodes detected as pc10 and higher, and add 1 +to their node numbers. You will then want to use the 'Recalculate IPs' button +on these nodes to get their IP addresses set appropriately. + +The type for each node is supposed to get detected automatically, but this can +be a bit imprecise (ie. processor speeds are never exactly as advertised - a +2GHz processor may be 1.99 GHz.) So, if the nodes didn't get their types +detected correctly, just select them all, and use the 'set type' button. + +##### Figuring out interfaces + +If you found earlier that the FreeBSD and Linux ordering for interfaces was +different, we'll fix that up now. Use the boxes right above the 'Re-number +interfaces' button to do this. Just leave blank any interface numbers your +nodes don't have. For example, if you have two interfaces, and what FreeBSD +detects as eth0 is eth1 under Linux, and vice versa for eth1, you'd enter +'1' and '0' in the first two boxes. Select all nodes, and hit the +'re-number' button. Once you've got this sorted out, the 'Control MAC' +column should be correct. + +Now, we're going to figure out where the interfaces are plugged into your +switches - you should have entered your switches into the database as part of +setup-db.txt . If they're not already, enable all of the ports on your +experimental network that have experimental interfaces connected to them. Also, +in order to work around some strange behavior (possibly a bug?), you'll need +to place these ports in some VLAN other than VLAN #1. If you just now +enabled these interfaces, wait a few minutes to give the switches time to learn +the nodes' MAC addresses. Now, select all the nodes, and click the 'search +switch ports' button. This will grab the MAC tables from all switches you put +into the database, which we'll match up with the MACs that the nodes themselves +reported. This will take a little while, and it will report any interfaces it +failed to find. Note - if you didn't enter your control network switch into the +database, this is okay, but this step won't find any control network +interfaces. That's acceptable. But, make sure it doesn't complain about any +experimental-network interfaces. + +##### Creating the nodes + +You're finally ready to take these nodes and actually create them! (By this +point, you should have a disk image, etc. from Utah.) Select all the nodes, hit +'Create', and wait a while. This enter all of the nodes into their permanent +location in the database, and will reboot them into a 'full' FreeBSD MFS. It +also puts them into the emulab-ops/hwdown experiment, to make sure that no +experimenters get them in case something went wrong. Inspect a few to make sure +they booted right. If so, free them from the hwdown experiment with: +nfree emulab-ops hwdown pc1 pc2 pc3 ... + +At this point, they should get a disk image loaded and end up in the free pool. + +##### Serial lines and power controllers + +This node creation process doesn't handle serial lines and power controllers +yet, unfortunately. These will need entries added to the tiplines and outlets +tables, respectively. Contact Utah if you need help with this. diff --git a/doc/setup-archive/setup-nortel.txt b/doc/setup-archive/setup-nortel.txt new file mode 100644 index 000000000..5d215e451 --- /dev/null +++ b/doc/setup-archive/setup-nortel.txt @@ -0,0 +1,89 @@ +#### +#### Setting up a Nortel 5500-series switch as an experimental switch +#### + +The Emulab snmpit support works for at least the Nortel 5500 series +switches; at DETER, we use 5510-48Ts and 5530-24Ts. Keith Sklower +of UCB wrote the actual code, and can provide suggestions about +which other Nortel switches will work with the snmpit code. + +There are at least two different lines of switches with different SNMP +MIBs, so there are definitely some Nortel switches which will NOT +work. The switches that use the RAPID-CITY mib should be okay, including +the 8000-series switches, although they have not been tested. +Apparently, *none* of the Nortel switches that are provided as part +of IBM blade centers will work with the current code; if you're +interested in support, definitely contact Keith. + +All of these commands are supposed to be typed from the command +line (use the "Command Line Interface" menu option). Make sure that +you go into "config term" mode to make the changes. We usually +leave our switches configured to come up with the menu, as there +are some common functions that are much easier to do from the menu. + +#### Access to the switches + +At DETER, we've had the switches lock up infrequently. When they +lock up, they require a power cycle to come back, and often have +managed to lose all or part of their configuration. + +For this reason, we have permanent connections to the serial ports +on the switches from serial port servers, so that we can get in and +reset the configuration by hand. We also have the switches on power +controllers, so that we can power off the switches remotely. + +#### Configuration + +While I've set up Nortels at DETER several times, and they seem +to work just fine, I may have forgotten some of the details in my +rush to get things working. Here are my suggestions -- please +let me know what details I've forgotten! + +You can ask us at DETER for our 300KB+ Nortel configuration files; +for obvious reasons, we don't include them here. Oh, yeah, and +we're running SW v4.2.1.005. + +#### ToS bits + +By default, the Nortels appear to set the *IP* ToS bits to zero +for most packets. To prevent this, turn off QoS on the switch +altogether. This make sense if we want the network to appear as +though the switch is a direct connection: + + no qos if-assign port ALL + +#### Spanning tree + +I believe that both Keith and I just turned off STP via the menu +options, but from the command line it should go something like +this (YMMV): + + interface FastEthernet ALL + spanning-tree port ALL learning disable + exit + +#### Extra packets + +As above, we also want to turn off all topology discovery packets, +which will otherwise interfere with idle detection: + + no autotopology + +#### IGMP packets + +Make sure that multicast can be flooded around the switch correctly; +the Nortel documentation for this command is particularly cryptic, +but it works for us. Doing IGMP snooping apparently (as per Keith) +locked up our switches. Ugh. + + vlan igmp unknown-mcast-no-flood disable + +#### mac address aging + +While we don't have this set at DETER right now, Keith recommended +turning off mac address aging, as per the various Cisco setup +suggestions. The Nortel maximum (as per Keith) is apparently +a million; there is no way to turn it off altogether. + + mac-address-table aging-time 1000000 + diff --git a/doc/setup-archive/setup-ops.txt b/doc/setup-archive/setup-ops.txt new file mode 100644 index 000000000..4e5de6dd9 --- /dev/null +++ b/doc/setup-archive/setup-ops.txt @@ -0,0 +1,217 @@ +# +# EMULAB-COPYRIGHT +# Copyright (c) 2002-2008 University of Utah and the Flux Group. +# All rights reserved. +# + +##### +##### Setting up the Utah Network Testbed software +##### Most recently tested on FreeBSD 6.2 . +##### + +##### Step 1 - OS installation and setup + +Install FreeBSD on the machine you'll be using for your ops node, using the +standard FreeBSD installation process. When asked by the installer, it's best +to choose the 'Developer' distribution set - this gets you full sources. When +it asks if you want to install the ports collection, answer *no*. Do not +install any packages at this time - you'll get a chance to later. You'll +need to partition your filesystems so that you have the proper amount of +space for certain directories - see below for details. + +Make sure that you have the network correctly configured. + +The following directories will need to exist on partitions that have enough +space to hold them: + +/usr - Needs space for the ports tree and a system object tree. + At least 10GB. Be sure to build with plenty of + inodes (the ports tree itself uses about 200000, so + be safe and build with at least a million). +/usr/testbed/ - Needs space for testbed software and logs. Several (3-4) GB + should be enough. + +Do *not* create any user accounts yet, and just log in as root for the time +being. Our software will create user accounts later, once you get boss set +up. + +If you already created any users, then delete them with the "pw" command +and make sure the home directories for them are removed as well! + + +##### Step 1a - Shared filesystem setup for combined ops/fs machine. + +If you have a separate 'fs' node, you should have already setup your shared +filesystems and can skip to Step 2. If this node is also to be your file +server, you will also need: + +/users/ - Needs space for user home directories. Amount of space + required depends on how many users you expect to have. + Generally, though, we suggest that users store large + files related to their projects in the /proj directory. +/proj/ - Needs space for project files. We recommend that this be larger + than /users, to encourage people to store files here, + which aids per-project accountability. +/groups/ - Needs enough space for files shared by the sub-groups of + projects. These are primarily used by classes, if any. +/scratch/ - Optional, large filesystem of "scratch" space. The intent + is that this filesystem provides per-project space + that is not "guaranteed" (for the Utah Emulab that + means we do not back it up to tape). If used, + you would either set no quotas, or higher quotas + than for /proj. Most sites probably won't want this. +/share/ - Exported read-only to all nodes, we use it for providing to + experimenters the source for the FreeBSD and Linux + versions we run as well as common packages and RPMs. + This could require anything from 1GB to 20GB+ depending + on what you want to make available. + +You may want to enforce quotas on the user-writable filesystems. This is the +main reason you'd want to keep them in separate filesystems (i.e., so people +can have different /users/ and /proj/ quotas.) If you do not think you will +ever use quotas, then you could make /users and /proj part of the same +filesystem. + +As mentioned /scratch is optional. If you are not providing "guarantees" +such as filesaved or RAIDed disk space and you are not using quotas, you +might as well just put all your space in /proj and not define FSDIR_SCRATCH +in the defs file. + +Note also since /share is exported read-only, FreeBSD requires that it be on +a separate filesystem from anything that is exported read-write. So while +/users, /proj and /groups can be on the same filesystem, /share cannot. + +Make sure that, no matter how you decide to partition things up, you make +symlinks to the appropriate places. ie., if you make one big filesystem called +/z that has /users, /proj, and /groups in it, make sure you: + + ln -s /z/users /users + ln -s /z/proj /proj + ... etc. + +In other words, we assume the existence of /users, /proj, /group and /share +(but not /scratch). + +##### Step 2 - Installing packages + +To make sure that you're running with versions of software from the ports +collection that are known to work with our software, and to save you hours +of compile time, we provide pre-built binary packages of the ports required +by Emulab. + +If this is a new Emulab installation, or you do not need to maintain backward +compatibility with old OS images (i.e., you plan to build new ones) then +download the packages tarball from: + + http://www.emulab.net/downloads/FreeBSD-6.1-20080610.tar.gz + +(You can use the FreeBSD 'fetch' command to download the file.) + +If you are upgrading an existing installation, and you want to continue to +use your current OS images, then you need to download a different tarball: + + http://www.emulab.net/downloads/FreeBSD-6.1-20080610-elvincompat.tar.gz + +Now, untar this someplace (you need about 140MB of space, so don't use /tmp). +Let's say you untarred it into /usr/tmp. You would then run: + + env PKG_PATH=/usr/tmp/packages pkg_add emulab-ops-2.0 + +Of course, if you untarred it somewhere else, put in the correct path. + +We provide all of the binary packages necessary to build ops/fs/boss in our +packages tarball, making setup of an Emulab-in-Emulab very quick. You may +want to install more packages built from ports tree makefiles with downloaded +sources, but if not there is no need to install a /usr/ports tree. + +We have run into many, many problems with clashes between versions of the +FreeBSD ports. So, rather than using the /usr/ports tree that comes on the +FreeBSD installation media, we use one that we've tested against. You can +grab our 'approved' copy of the ports tree from: + + http://www.emulab.net/downloads/ports-20060921.tar.gz + +The tarball is about 40 meg; it unpacks to 175 meg. + +Untar it, move it into place as /usr/ports (rename the old directory to +ports.old, or just remove it), and install whatever ports you want to make +ops feel like 'home' (like emacs, jove, or whatever) + +##### Step 3 - Unpacking and running configure + +At this point, you'll need to make a 'defs' file - You will use the same +defs file on boss, ops and fs. See defs-example in the top level directory +as a starting point. You want to make sure each of the variables has a +definition that makes sense in your environment. Feel free to ask Utah if +something is not clear. + +Unpack the testbed source, and run it's configure script. A good place to +unpack the source code is /usr/testbed/src/testbed. You will use the +--with-TBDEFS option to configure to give it the path to your defs file: + + mkdir -p /usr/testbed/src/testbed + mkdir -p /usr/testbed/obj/testbed + cd /usr/testbed/src/testbed + tar .... + cd /usr/testbed/obj/testbed + /usr/testbed/src/testbed/configure \ + --with-TBDEFS=/path/to/your/defs-file + +Typically, you would store your defs file in the source tree along with +the other defs files that came in the tarball. + +##### Step 4 - Running the ops installation script + +In the object tree you've configured (say, /usr/testbed/obj/testbed), there's +an 'install' subdirectory, with a script called 'ops-install'. Just run this +script as root (note the same package directory argument as above). + + cd install + perl ops-install -p /usr/tmp/packages + +It will take care of installing any additional ports, and doing various +other configuration of FreeBSD required to make it into an ops node. The +script is designed so that you can run it as many times as you want, and +it'll just skip stuff it's already done. If it fails, send the output to +Utah so that we can fix it up. If it succeeds, follow any other +instructions it may have. The script will tell you to reboot the machine, +but you can wait until after you do the next step to do so, if you want. + +(You may have to set the executable bit on this script, since configure won't; +we'd like to get this fixed at some point.) + +You should be aware that, among other things, this script sets up sendmail, and +sets up password-less 'sudo' for anyone in the 'wheel' group. If you don't want +these for security reasons, you can undo them after the installation script is +finished. + +##### Step 5 - Installing from source + +To install the actual testbed software, simply do a 'gmake ops-install' in your +object directory. + +(Note: If you're logged in as root, /usr/local/bin, where gmake lives on +FreeBSD, may not be in your path.) + +If you'll be using Cyclades or similar boards to use 'ops' as a serial console +server, you'll also want to do a 'gmake tipserv-install' in the object +directory. In addition, you'll need to build a custom kernel with the +appropriate driver (you may even need to get patches from Utah for some +boards). You'll also need to set up a few things (not yet automated), +described in doc/adding_nodes.txt . + +##### Step 6 - Setting up mailing lists + +Optional. The ops-install script set up some mailing lists for the email +addresses you set up in the defs file, in /etc/mail/lists . If you want to run +these mailing lists off your ops node, you can put some people in them now. +These are just standard sendmail list files: a list of addresses and/or +archive files, one per line. If you chose to use archive files, give a path +to the file you'd like sendmail to stick all correspondence in, and create +this file. + +##### Step 7 - Other miscellaneous things to set up + + [Nothing at this time] + +Once you're done with all of this, reboot ops. diff --git a/doc/setup-archive/setup-power.txt b/doc/setup-archive/setup-power.txt new file mode 100644 index 000000000..627b5b5ec --- /dev/null +++ b/doc/setup-archive/setup-power.txt @@ -0,0 +1,120 @@ +# +# EMULAB-COPYRIGHT +# Copyright (c) 2002-2007 University of Utah and the Flux Group. +# All rights reserved. +# + +##### +##### Setting up power controllers for Emulab. +##### + +Currently we support a couple of "families" of power controllers: +the serial-controlled BayTech RPC family and the network-controlled +APC AP family. There is no guarantee that all models in these families +work, but we know that the following models do: + + Baytech RPC14 + Baytech RPC27 + APC AP9210 (deprecated) + APC AP7960 + +Basically, any unit that uses the Baytech CLI or the APC MIB should work. + + +A. node_types and nodes table entries. + +For either serial or network controlled units, you will need to add +node_type and node DB table entries. Perform the following in mysql +("mysql tbdb" on boss): + + insert into `node_types` (class,type) values ('power','APC'); + +or: + + insert into `node_types` (class,type) values ('power','RPC27'); + +depending on what you have. You will then need to add a nodes table entry +for each instance of the power controller you have. Something like: + + insert into `nodes` (node_id,type,phys_nodeid,role,priority, + status,eventstate,op_mode,allocstate) values + ('powerN','APC','powerN','powerctrl',10001, + 'down','ISUP','NONE','FREE_DIRTY'); + +'node_id' and 'phys_nodeid' should be identical and be your name of the + power controller (e.g., "power1", "power2", etc.). +'type' will be the same as what you added to node_types above. +'role' should always be 'powerctrl'. +'priority' is used to order all nodes in node listings (e.g., in web pages); + we pick a high number for power controllers so they will come after + regular nodes. Each power controller should have a different value. +All other fields should be exactly as shown. + + +B. interfaces and wires table entries. + +If you have a network power controller, you will need to add entries in the +interfaces and wires tables. For interfaces, you again need one line per +power controller: + + insert into `interfaces` (node_id,IP,mask,interface_type,iface,role) + values ('powerN','AAA.AAA.AAA.AAA','MMM.MMM.MMM.MMM','','eth0','other'); + +'node_id' matches that in the nodes table above. +'IP' and 'mask' reflect the address assigned to the power controller + (e.g., 10.11.12.8, 255.255.255.0). + +Wires table entries are strictly informational. You can put them in if you +want to keep track of the cable numbers. Though if they are properly +entered, you can use snmpit to perform some basic functions on the connected +switch ports. A wires table entry looks like: + + insert into `wires` (cable,len,type,node_id1,node_id2,card2,port2) + values (NNN,LLL,'Power','powerN','switchN',MMM,PPP); + +'cable' and 'len' are integers that describe the cable. Make them 0 and 0 + if you do not care. +'type' should be 'Power'. +'node_id1' should match 'node_id' from the interfaces/nodes tables. +'node_id2' should be the name (in the nodes table) of the switch the + power controller is connected to. +'card2' and 'port2' are integers indicating the module and port on the + switch that it is connected to. + + +C. outlets table entries. + +The final step is to add 'outlets' table entries for all nodes that are +controlled by the power controllers. For each node there should be an +entry of the form: + + insert into `outlets` (node_id,power_id,outlet) + values ('pcN','powerN',NNN); + +'node_id' is the name of the node which is being power controlled. +'power_id' is the nodes table name of the power controller that the node + is connected to. +'outlet' is an integer telling which outlet on the power controller that + the node is connected to. All power controllers we support have outlets + numbered from 1..N where N is the maximum number of outlets on the + controller (typically 8, 16, or 24). So the value of this field should + be from 1 to N. + +Note that you have not yet actually added any nodes to the testbed, so +you should use the name that you *intend* to assign to each node when added. +The power control subsystem will work without there being DB state for a +node during bootstrap, but it is important that this field eventually match +an entry in the 'nodes' table later. It is okay to use placeholder values +for 'node_id' in the outlets table and then change them later to match the +nodes table entries. + + +D. Testing + +After all the state has been added, you can use the 'power' command to +test the configuration. If you have serial consoles and have already +done the console setup, then you can "console " and then do +"power cycle " and see if the machine reboots by looking for the BIOS +screen or other bootstrap output. If you do not have serial consoles or +have console setup, you will have to be physically in front of the machine +to see if the outlet properly cycles. diff --git a/doc/setup-archive/setup-serial-and-power.txt b/doc/setup-archive/setup-serial-and-power.txt new file mode 100644 index 000000000..877d10e21 --- /dev/null +++ b/doc/setup-archive/setup-serial-and-power.txt @@ -0,0 +1,209 @@ +The following is a guide to setting up power control and serial lines. + +Overview: + +0) Setup capture logging +1) Enter all the power controller's info into the emulab DB + and "capture" startup script (if applicable). +2) Enter all the nodes' outlet info into the DB +3) Enter all the nodes' serial line info into the DB and "capture" startup +4) Enter all tipserver machines into the database. +5) Collect interface mapping from nodes + +Details: + +Steps two and three can be easily scripted (automated) if you have an +online mapping of node to outlet and node to serial line. + +Note that your boss node should *NOT* also be a tipserver node! You can +host tiplines on your users node if you like, or on other dedicated +tipserver nodes. Hosting tiplines on boss (and thus exporting them via +capture) is generally a bad idea because of security issues, not to mention +that it just will plain not work. + +0) Setup capture logging + +Before you can start capture for any power controllers or nodes, you should +setup the logging files. Capture is hardwired to log into /var/log/tiplogs. +We usually create this as a symlink to /usr/testbed/log/tiplogs (yes, yes, +we should change the path in the source...). So for each serial line server +you should: + + mkdir -p /usr/testbed/log/tiplogs + ln -s /usr/testbed/log/tiplogs /var/log/tiplogs + cp /dev/null /usr/testbed/log/tiplogs/capture.log + +To setup logging for the capture processes themselves, add the following +to each /etc/syslog.conf: + + # testbed stuff + !capture + *.* /usr/testbed/log/tiplogs/capture.log + +and then HUP syslog. To ensure the logfile gets "rolled" periodically, +put this in /etc/newsyslog.conf do: + + # testbed stuff + /usr/testbed/log/tiplogs/capture.log 664 7 1000 * Z + +*** export log filesystem to ops node. So log dir should really be a + distinct filesystem, else you export more than you intend! + +# rc.conf: added by Emulab setup: allow export of log filesystem +rpcbind_enable="YES" +nfs_server_enable="YES" +nfs_reserved_port_only="YES" +mountd_enable="YES" +mountd_flags="-r -p 900" + +# exports: added by Emulab setup: allow export of log filesystem +/var/log/tiplogs ops -ro -alldirs + +*** install boss rootkey in tipserv authorized_keys + +*** install script(s) to keep groups file in sync + +*** add NFS mount to ops fstab + + tipserv:/var/log/tiplogs /var/log/tiplogs nfs ro,-b,-i,nodev,nosuid 0 0 + + note that ops-install script assumes that ops is a tipserv machine + if mounting remote tipserv directory on ops:/var/log/tiplogs, be sure + to disable newsyslog.conf entry. + + # exports: added by Emulab setup: allow export of log filesystem +/var/log/tiplogs ops -ro -alldirs + +*** install script(s) to keep groups file in sync + +*** note that ops-install script assumes that ops is a tipserv machine + if mounting remote tipserv directory on ops:/var/log/tiplogs, be sure + to disable newsyslog.conf entry. + +1) Enter all the power controller's serial line info into the emulab DB + and add a line to the "capture" startup script. + +For each power controller, do the following: + +* Create an entry in the "nodes" table. + +insert into nodes (node_id, type, phys_nodeid, role) + values ("power", "", "power", "powerctrl"); + +Where "" is the power controller designation you pick. I suggest just +using a simple number. You can, of course, change this name if you wish, +just use the same designation below. is the type of power +controller; valid values are "RPC27" and "APC". + +If this power controller of type "APC", make sure it has an entry in your +/ets/hosts file on your boss node, and skip to step 2. + +* Create an entry in the "tiplines" table. + +insert into tiplines (tipname, node_id, server) + values ("power", "power", ""); + +Where "" is the same value as above and is the hostname +of the machine that has the serial connection to the power controller. + +* Create a capture startup line in /usr/local/etc/rc.d/capture.sh on the + machine that has the serial connection to "power". + +If the file doesn't exist yet, create it, and put + +#!/bin/sh + +at the top, then change the file's mode to 0755. + +Add a line like this to the above file: + +/usr/testbed/sbin/capture -T 15 -r -s 9600 power >/dev/null 2>&1 & + +Where is as above, and is the tty device file associated +with this power controller. To make sure the console log for the power +controller gets "rolled", add the following to /etc/newsyslog.conf: + + /usr/testbed/log/tiplogs/power.log 640 7 1000 * Z /usr/testbed/log/tiplogs/power.pid + +* Disable the status detail and confirmation prompt on the power + controller. + +Manually execute the command line you added above, then run "console +power". You will now have access to this device's serial line. +Navigate the menus to turn off status reporting and confirmation. + +2) Enter all the nodes' outlet info into the DB + +For each node, do the following: + +* insert row into outlets table + +insert into outlets values ("pc", "power", , NULL); + +Where is the testbed node unique suffix (e.g., number), is the +power controller that its connected to, and is the outlet number +on the power controller. + +* Try power cycling the node (perhaps wait to do this until you get to + the end of step 3) + +on your boss node, execute + +sudo /usr/testbed/bin/power cycle pc + +You shouldn't get any errors, and the machine should go through a cold +reboot cycle. + +3) Enter all the nodes' serial line info into the DB and "capture" startup + +This process is nearly identical to step one. + +For each node, do the following: + +* Insert tipline row into DB + +insert into tiplines (tipname, node_id, server) + values ("pc", "pc", ""); + +* Create a capture startup line in /usr/local/etc/rc.d/capture.sh on the + machine with the serial conection to "pc". + +If the file doesn't exist, follow the script creation step outlined above +in step 1. + +Open the above file, and add a line like this: + +/usr/testbed/sbin/capture -T 15 -r -s 115200 pc >/dev/null 2>&1 & + +Where is as above, and is the tty device file associated +with this pc. To make sure the console log for the node gets "rolled", +add the following to /etc/newsyslog.conf: + + /usr/testbed/log/tiplogs/pc.log 640 7 1000 * Z /usr/testbed/log/tiplogs/pc.pid + +* Test the console line + +Run "console pc" on serial 2. Hit enter a couple of times, or reboot +(using "power"!) the machine and wait to see if the PXE and MFS kernel +console messages appear. + +4) Enter all tipserver machines into the database. + +For each independent tipserver machine you have, do the following in mysql: + +insert info tipservers values (""); + +5) Collect interface mapping from nodes + XXXX this should move elsewhere XXXX + +Once you have the consoles up and running, it would be a good time to +collect interface mapping information from the nodes. Unfortunately, +FreeBSD and Linux might number interfaces in different orders and they +do give them different names. Thus, Emulab requires a mapping from +FreeBSD interface number to Linux interface number. + +You'll need to boot one of each different node type into both freebsd and +linux, recording the interface number to MAC address mapping. Use these +two mappings to correlate the FreeBSD and Linux interface numbers. You'll +use this information in conjunction with the instructions for adding nodes +in setup-nodes.txt diff --git a/doc/setup-archive/setup.txt b/doc/setup-archive/setup.txt new file mode 100644 index 000000000..66e8767f4 --- /dev/null +++ b/doc/setup-archive/setup.txt @@ -0,0 +1,355 @@ +# +# EMULAB-COPYRIGHT +# Copyright (c) 2001-2008 University of Utah and the Flux Group. +# All rights reserved. +# + +##### +##### Setting up the Utah Network Testbed software. +##### Most recently tested on FreeBSD 6.2 . +##### + +##### Important notes + +In order to save your time and ours when building and installing an +Emulab, we make some up-front requirements. We need explicit +agreement to these conditions, in advance. + +1. We need to be satisfied that you have appropriate technical +expertise, that those who have it will be directly involved, and that +"management" will exercise appropriate supervision. This may take the +form of some sort of teleconference "interview" procedure with the +project lead and your people. The ideal people to bring up an Emulab +are quality system and network administrators; good student versions +of such people can work. + +2. We must be closely involved in designing the overall network +architecture of your Emulab, and approve the planned hardware. This +aspect is not as simple technically as it appears, and includes +financial, security, and support tradeoffs. + +3. To be able to help you debug any problems you run into or answer +certain questions, we'll need to have accounts, preferably with root +access if allowed by your institution's AUP, on your Emulab's servers +('boss', 'ops', and possibly 'fs' if you separate the ops/fs functions), +and will need to be able to access the webserver on boss. This is crucial +during testbed installation and bringup; after that it's not so important, +except when you are upgrading to a new version of our software. + +4. Serial line consoles and remote power controllers must be connected +to at least two experimental nodes, so we can help debug. + + +Supported environment: +This software does make some assumptions about the environment in which it is +run. Some of the most basic ones are listed below. In general, we don't have +the resources to adapt it to every possible environment. So, you will need to +either work out a way to match the environment outlined below, or be willing to +invest some work in adapting the software. + +(0) We assume a minimum of two dedicated server machines, known hereafter +as 'boss' and 'ops', for Emulab setup. While the 'ops' server normally +hosts the Emulab shared filesystems, we also support the use of a separate, +dedicated filesystem server, hereafter referred to as 'fs'. + +(1) You will need at least two network interfaces on each experimental node: +one for the control network and one for the experimental network. The +experimental network needs to be one on which we can make VLANs with SNMP. +Currently, Emulab supports +* Cisco 6500 and 4000 series switches (though not all switches in these lines + have been tested - the 6513, 6509, 6506, 4006, and 4506 are known to work, + running both CatOS and IOS). Cisco 3750's probably work but have not been + tested. The 2950 and 2980 switches are known to work, though they are limited + to a small number (64) of VLANs. +* many Nortel switches (those with RAPID-CITY Nortel mibs, ie, 5510-24T 5510-48T + 5520-24T 5520-48T, and reasonably recent Accellars). +* some Foundry switches +* Intel 510T (probably a bit bit-rotted since we haven't used it in a long time, + but likely easy to fix) +* HP Procurve switch support has been written (by Cornell) but not yet + fully tested. +Emulab's Cisco support is the best, because those are the switches we have. +The control net must have full multicast support, including IGMP +snooping. Nodes' control network interfaces must support PXE. + +(2) We highly, highly recommend that boss, ops, fs, and all the nodes be in +publicly routed IP space. If this is not possible, then boss, ops and fs +should be given two interfaces: One in the nodes' control network, and one in +public IP space. If you must use private IP space for the nodes' control net, +we suggest using the 192.168/16 subnet, which leaves the larger 10/8 subnet +available for the experimental network. The defs-example-privatecnet file +shows an example configuration like this. + +(3) If you have a firewall, you will need to be able to get certain standard +ports through to boss and ops, such as the ports for http, https, ssh, named +(domain), and smtp. Any other strange network setup (such as NAT) between the +boss/ops/fs and the outside world will cause really big headaches. + +(4) The whole testbed should be in a domain or subdomain for which boss can be +the name server. + +(5) The nodes must be able to reach boss with DHCP requests on the control +network - this means either being in the same broadcast domain (ie. LAN), or, +if there is a router in between, the router must be capable of forwarding +DHCP/BOOTP packets. Since the nodes will DHCP from boss, it is important that +there not be another DHCP server (ie. one for another part of your lab) +answering their requests. + +(6) The boss node should have its own local disk space, for various reasons: +* For logistical reasons, /usr/testbed cannot be shared between boss and + ops, or between boss and fs, or between ops and fs. All three install + different (and alas, sometimes conflicting) subsets of the Emulab software. +* For security reasons, /usr/testbed/images, which is the home of the + "trusted" default disk images, should not be hosted on ops/fs since they + are potentially more vulnerable. +* Similarly, home directories for "real" (admin) users on boss should not be + shared with, or hosted from, ops and fs. See shellonboss.txt for details. + +##### Other docs: + +Useful summary info and diagrams can be found in "build-operate.ppt" and +"security-all.ppt" in http://www.cs.utah.edu/flux/testbed-docs/internals/ + +##### Step 1 - Set up "fs" and "ops" + +Follow the instructions in the setup-ops.txt file before the ones in this file! + +##### Step 2 - OS installation and setup + +Install FreeBSD on the machine you'll be using for your boss node, using the +standard FreeBSD installation process. When asked by the installer, it's best +to choose the 'Developer' distribution set - this gets you full sources. When +it asks if you want to install the ports collection, answer *no*. You will +be able to install ports later. + +As with setting up ops, you need to create partitions large enough: + +/usr - Needs space for the ports tree and a system object tree. + At least 10GB. Be sure to build with plenty of + inodes (the ports tree itself uses about 200000, so + be safe and build with at least a million). +/usr/testbed - Needs space for testbed software and logs, as well as + many disk images. At least 10GB, but more is better! + You may simply make sure this space is available in + /usr, and not make this a separate partition. +/var - Holds the database, so at least a few hundred MB. + + +Do *not* create any users yet, and just log in as root for the time being. +Our software will create users later, once you get boss set up. BE SURE to +give root a password and REMEMBER it! You are going to need it later. To +set the root password use "passwd root". + +If you already created any users, then delete them with the "pw" command +and make sure the home directories for them are removed as well! + +##### Step 3 - Installing packages + +Again, almost the same as on ops. Use the same tarball you downloaded before, +unpack it on this machine (in say /usr/tmp) and follow the same pkg_add +procedure, except this time, you're going to install the emulab-boss package +instead of emulab-ops: + + env PKG_PATH=/usr/tmp/packages pkg_add emulab-boss-2.0 + +Also grab a copy of our approved ports tree and install it, the same as +described in setup-ops.txt. + +##### Step 4 - Unpacking source and creating a defs file + +Unpack the source tarball: + + mkdir -p /usr/testbed/src/testbed + mkdir -p /usr/testbed/obj/testbed + cd /usr/testbed/src/testbed + tar .... + +Now, you'll need to create a 'defs file', which is used by the configure +script to describe your environment, such as the hostnames of your boss and ops +nodes, and email addresses that certain types of mail will be sent to. + +Use the 'defs-example' file in the root of our source distribution as a +template. It contains comments explaining the important variables to set. +defs-example-privatecnet is the same configuration but with an unroutable +control net. + +##### Step 5 - Configuring an object tree + +This works the same as it did on ops. Remember, you have to use the same +defs file on boss that you used on ops! + + + cd /usr/testbed/obj/testbed + /usr/testbed/src/testbed/configure \ + --with-TBDEFS=/path/to/your/defs-file + +##### Step 6 - Running the boss installation script + +Again, this works the same as it did on ops, except that you run +boss-install in the object tree, instead of ops-install. Just run this +script as root (note the same package directory argument as above). + + cd install + perl boss-install -p /usr/tmp/packages + +Like the ops-install script, boss-install sets up passwordless sudo for +anyone in the wheel group. + +Also note that boss-install (really, the installing from source step below) +requires internet access so it can grab some files + +##### Step 7 - Installing from source. + +This step is now done as part of boss-install. You can check it or fix up +problems by doing the makes yourself, as follows: + +In your object directory, do a 'gmake && gmake boss-install'. Then, as root, do +a 'gmake post-install'. The post-install target needs to run as root, so that +it can make certain scripts setuid, etc. + +##### Step 8 - Setting up root ssh from boss to ops + +This step is now done as part of boss-install/ops-install. To confirm +this, make sure this works: + + boss> sudo ssh ops ls / + +If this *FAILS*, you will need to do this by hand, typing a password: + + scp /root/.ssh/identity.pub ops:/root/.ssh/authorized_keys + +##### Step 9 - Setting up named + +The testbed software manipulates DNS zone files for two reasons. First, it +adds your nodes to them so that you don't have to. Second, it creates CNAMEs +for all the nodes in every experiment. (so that you can use, for example, +'nodeA.myexp.myproj.emulab.net' to refer to your node regardless of which +physical node it got mapped to.) + +The named_setup script does this by generating zone files - in general, it +concatenates a '.head' file, written by you, with it's own generated entries. +The main zone file is /etc/namedb/OURDOMAIN.db, where OURDOMAIN is from your +defs file. (OURDOMAIN, unless explicitly specified, is taken to be the domain +portion of BOSSNODE). We also generate reverse zone files (for inverse +lookups, ie. turning IP addresses back into names) in +/etc/named/reverse/SUBNET.db, where SUBNET is the the class-C subnet in which +the addresses reside (ie. 10.0.0.db). This value is defined in the defs +file created above, as TESTBED_NETWORK. + +boss-install makes a reasonable attempt to create a set of named config +files for your, placing them in /etc/namedb. If your testbed consists of +a single class-C network, then these files will most likely be correct, +although you want to look at them to make sure. Look at these files to make +sure: + + /etc/named/OURDOMAIN.db.head + /etc/named/reverse/SUBNET.db.head + /etc/named/named.conf + +If you have more than one class-C subnet for your testbed, you'll need a +copy of the reverse zone file for each one. You want to put boss, ops, and +any 'infrastructure' equipment (such as routers and switches) into the zone +files. These zone files do not need to include the nodes - the nodes will +be added to them automatically. Be sure to edit /etc/named/namedb.conf if +you add any reverse map files (follow the format for the existing entry). + +Once you think you've got things set up, run /usr/testbed/sbin/named_setup, +and make sure that it doesn't give you any error messages. It will generate +the following files: + + /etc/namedb/OURDOMAIN.db + /etc/namedb/reverse/SUBNET.db + +##### If you are using unroutable private IP addresses for part of the + testbed: + +In order to handle this situation, we'll need to use a feature of bind called +`views` so that the private addresses are only exposed to clients within the +testbed. See the bind documentation for more details on this feature. Note +that you'll want to make sure that loopback queries from boss itself see the +internal view - you want boss to resolve its own hostname to its private +address, not its public one. + +In order to use multiple views, we generate multiple zone files. In addition +to OURDOMAIN.db, which will be used for the 'external' view, we create +OURDOMAIN.internal.db for use with the 'internal' view. So, you'll also need +to create OURDOMAIN.internal.db.head . When we generate the zone files, only +publicly-routable addresses are put into OURDOMAIN.db . +OURDOMAIN.internal.db contains all addresses, both public and private. So, +basically, you'll want to put the public addresses for boss, ops, etc. into +OURDOMAIN.db.head, and their private addresses into +OURDOMAIN.internal.db.head . + +##### Step 10 - Other miscellaneous things to set up + +There are a few things we haven't been able to completely automate just yet, +though we hope to soon. + +hosts file - You want to put boss/ops name/IP addresses in /etc/hosts on +both boss and ops to avoid boottime circular dependencies (cause of NFS +cross mounts). This is done for you in ops-install and boss-install, but +you might want to confirm it was done properly. If you change the IP +addresses of boss/ops later, you will want to be sure to update /etc/hosts +on both machines. + +SSL certificates - Our apache config file expects to find SSL certificates +in: + /usr/local/etc/apache/ssl.crt/www..crt + /usr/local/etc/apache/ssl.key/www..key + +(where is OURDOMAIN from the configure defs file, which defaults +to boss's domain name). + +boss-install already generated a temporary no-passphrase certificate for you +and placed them in the locations specified above. However, we recommend +that you get a "real" certificate from Verisign (or one of the others). + +DHCPD - boss-install generated a dhcpd.conf.template and installed it in +/usr/local/etc (the template is derived from information you provided in +defs file). It then generated an actual dhcpd.conf file and started up +dhcpd for you. Do not edit the dhcpd.conf file directly! If you need need to +make changes, change the template instead and then run: + + /usr/testbed/sbin/dhcpd_makeconf -i -r + +tftpboot - There are a few bootloaders, mini-kernels, and MFSes that are used +to boot, reload, etc. testbed machines, which live in /tftpboot . For the time +being, these are not distributed with our source, and require some site +customizations, so ask Utah for the boot loaders, etc. + +disk images - You'll also, of course, need disk images to go on your nodes. +Right now, we have no automatic way of generating these, so you'll have to ask +Utah for some. + +locate database - It can be useful to update the 'locate' database to help you +find files as you're learning the system. This normally happens nightly, but +you can force it to happen now by running '/usr/libexec/locate.updatedb' as +root. This will take several minutes. You can then find foo.conf by running +'locate foo.conf'. + +dhclient - If you don't control the public network to which boss and ops (and +perhaps tipserv) are attached, your network admins may want you to run dhclient +on those interfaces (and you may want to as well so that your boss +/etc/resolv.conf contains the correct nameserver returned by dhclient, to avoid +configuring it statically). However, boss runs its own nameserver that +ops and the experimental nodes must contact to resolve internal names, so you +must get dhclient to correctly write /etc/resolv.conf (i.e., boss must be the +first nameserver). The easiest way to solve this problem is to add the +following line to boss's /etc/dhclient.conf: + prepend domain-name-servers 127.0.0.1 +and add the following line to ops's (and any other non-experimental nodes, such +as tipservs) /etc/dhclient.conf: + prepend domain-name-server +where is replaced by a valid IP address for your boss. Make sure to +restart dhclient on these machines since dhclient will continue to overwrite +/etc/resolv.conf incorrectly. + +##### Step 11 - Reboot boss + +Okay, go ahead and reboot boss now, and make sure it comes up okay. + +##### Step 12 - Filling the database + +See the file setup-db.txt in this directory for instructions on getting the +proper information about your site and nodes into the database. + diff --git a/doc/setup-archive/tipserver.txt b/doc/setup-archive/tipserver.txt new file mode 100644 index 000000000..4abc90198 --- /dev/null +++ b/doc/setup-archive/tipserver.txt @@ -0,0 +1,51 @@ +These are the steps we followed to create a second tip server: + +1. Load the new 4.3 image, and then neuter the tmcd/tmcc stuff. Give + it a fixed IP address (turn off DHCP). + +2. Put in testbed /etc/group file (since we are going to run + console_setup.proxy from paper), but a tbadmins only password file + since mere users will not be able to log in. + +3. Mount fs.emulab.net:/users on /users (in /etc/fstab). + +4. Create new filesystem for /usr/testbed and mount it (in /etc/fstab). + +5. Install testbed software using new "gmake tipserv-install" target. + +6. Edit /etc/rc.local to start up the /usr/site/etc/capture.rc file. + +7. Create /usr/site/etc/capture.rc. + +8. Symlink /var/log/tiplogs to /usr/testbed/log/tiplogs. + +9. Edit /etc/syslog.conf: + + !capture + *.* /usr/testbed/log/capture.log + + Must "touch" the log file so it exists. + +10. Create /dev entries for cyclades with MAKEDEV.cz. + +11. Edit /etc/remote as needed. + +12. Edit /usr/site/etc/capture.rc as needed. + +13. Make sure DB tiplines table reflects the node names/locations. + Should be entries like: + + INSERT INTO tiplines VALUES ('pc109','tipserv1.emulab.net',0,0,''); + +14. Create symlinks on plastic to NFS mounted directories on the + tipserver. eg: + + plastic:/dev/tip/pc109.acl --> + /nfs/tipserv1/dev/tip/pc109.acl + plastic:/var/log/tiplogs/pc109.run --> + /nfs/tipserv1/var/log/tiplogs/pc109.run + plastic:/var/log/tiplogs/pc109.log --> + /nfs/tipserv1/var/log/tiplogs/pc109.log + + Use amd on plastic for this since we don't hard mount cross deps. + diff --git a/doc/setup-archive/tipwithoutops.txt b/doc/setup-archive/tipwithoutops.txt new file mode 100644 index 000000000..9993f2013 --- /dev/null +++ b/doc/setup-archive/tipwithoutops.txt @@ -0,0 +1,55 @@ +; This file contains an overview of the design of tiptunnel, which allows users +; secure access to serial console lines, directly from their desktop machine + +; crb, june 5 2002 + +Tip-without-ops: + +Joe User has downloaded and installed "tiptunnel," telling it his favorite telnet client +and associating it with files of type "text/testbed-acl" in his web browser. +There is a Windows binary (and a FreeBSD binary?), as well as unix/windows source, +available for download, with adequate disclaimers all over it. + +Joe User logs into his experiment, and opens up a "detail view" on a node +assigned to his experiment. A link on the page offers "Connect to serial line." +When he clicks on the link, the server sends a file of MIME type "text/testbed-acl". +This launches tiptunnel on his machine. The file contains a server name, a port number, +a key, and a certificate's SHA-hash. + +Tiptunnel connects, via TCP/IP, to the server/port specified in the ACL file. +It is now talking to Capture. Capture, upon accepting a TCP/IP connection, +wants a secret key before it allows a client access to the serial line. +Tiptunnel sends "WANTSSL" as the secret key, and both initialize an OpenSSL +connection. Capture uses $TB/etc/capture.pem as its certificate, unless a different +certificate was specified on its command line. + +Tiptunnel looks at the certificate given by Capture, and SHA-hashes it, verifying that +it matches the hash in the ACL file. If it does not, the user is warned of a possible +man-in-the-middle attack, and tiptunnel closes. If the hash does indeed match, +the key is sent over SSL. If the key is accepted by Capture, the connection begins. + +Tiptunnel then forks. + +The parent starts listening on a local port for a TCP/IP connection. A single +connection (from localhost only) will be accepted, and traffic to/from that port +will be tunnelled through the SSL connection. Upon accepting a connection, +special telnet commands will be sent to the client, turning off line-at-a-time and +local echo. +When the connection closes, this process will exit. + +The child execs the telnet program chosen by the user, +telling it to connect to localhost: (the tiptunnel). +Joe User is now able to talk to his node's serial line. +When Joe User exits, the connection is dropped, and all processes exit. + + + + + + + + + + + + diff --git a/doc/setup-archive/update-testbed.txt b/doc/setup-archive/update-testbed.txt new file mode 100644 index 000000000..0e829abf0 --- /dev/null +++ b/doc/setup-archive/update-testbed.txt @@ -0,0 +1,77 @@ +# +# EMULAB-COPYRIGHT +# Copyright (c) 2004 University of Utah and the Flux Group. +# All rights reserved. +# + +Rough steps for upgrading your existing Emulab. + +0. "Shutdown" the testbed. Disable the web interface and prevent user + logins to "users". [ We really need a way to stop all Emulab daemons, + etc. that might be confused by DB changes. ] + +1. Look at doc/UPDATING to find out what you have to update in the + boss/ops FreeBSD environment before updating Emulab software. + Entries are timestamped, so look for things that have changed since + the last you installed. + +2. Unpack and build the Emulab software. You should do this in a directory + accessible to your tipserv, ops and boss nodes (e.g., /users) to make + the various installs easier. + +3. BACKUP YOUR DB. Do: + mysqldump tbdb > db.backup + +4. Update the DB. Look at sql/database-migrate.txt which is a list of + SQL commands to perform. In some cases, it also includes instructions + regarding operations you need to perform on your data to make it work + in the new schema. Diff the version of this file from the last release, + with the version from this release to find which ones to apply. + + If all goes well, proceed to the next step. Be aware that the + "boss-install" make target below will do further checks for missing + rows in DB tables. That target will fail, and tell you what to do to + fix your DB, in that case. It does the table check in two pieces, + so it is possible you will have to fix the DB, and restart the boss-install + twice. + + If anything goes wrong with the DB update step, contact us. You can + still abort the upgrade at this point by putting back the old db: + mysql + drop database tbdb; + create database tbdb; + quit + mysql < db.backup + and bringing the testbed back up. After this step, you are committed + to the upgrade. + +5. Login to your tipservers (if you have them), cd to your build directory + and do: + sudo gmake tipserv-install + +6. Login to your ops/users node, cd to your build directory and do: + sudo gmake ops-install + +7. Back on your boss node, cd to your build directory and do: + sudo gmake boss-install + sudo gmake post-install + +8. Install a "shared" copyright to appear on the web pages. We have not + yet integrated this step, so it is a little awkward: + + Edit www/copyright-contrib.html and fill in values for @SITENAME@, + @SITEDATES@, @SITECOPYRIGHT@. Then copy this file to copyright.html + in your installed testbed tree, e.g.: + cp copyright-contrib.html /usr/testbed/www/copyright.html + +9. Reboot tipservers, ops and boss and make sure things boot up ok. + Try creating/swapping/modifying/destroying an experiment, look + for errors. + +10.Bring up the web interface and inform users that they should reboot + their existing experiment nodes if they notice odd behaviors. + +Things should be backward compatible so you don't have to update the +standard disk images immediately. But you will want to do it. See +update-node.txt for more info about that. + -- GitLab