Commit 958831ee authored by Mike Hibler's avatar Mike Hibler
Browse files

Significant overhaul to structure.

Also fixes based on updating Leigh's elabinelab.
parent 1a4759c4
......@@ -11,78 +11,55 @@ the ops node, so pay attention!
[XXX these would benefit from breaking ops out from boss as they are different
enough that describing them with a lather-rinse-repeat process is confusing...]
==========
BEFORE YOU START: Backup if you can!
A. Things to do in advance of shutting down Emulab.
If your boss and ops are VM on Xen, you can create shadows of the disks
that you can roll back to. Really only need to backup the root disk which
has all the FreeBSD stuff. Login to the control node and:
These first few steps can be done in advance of shutting down your site.
These include making a backup, fetching the new release files and merging
in local changes, building a custom kernel (if you use one), and stashing
away state about your current packages.
# apt
sudo lvcreate -s -L 33g -n boss.backup xen-vg/boss
sudo lvcreate -s -L 33g -n ops.backup xen-vg/ops
1. BACKUP IF YOU CAN!
# cloudlab utah/clemson
sudo lvcreate -s -L 17g -n boss.backup xen-vg/boss
sudo lvcreate -s -L 17g -n ops.backup xen-vg/ops
If your boss and ops are VM on Xen, you can create shadows of the disks
that you can roll back to. Really only need to backup the root disk which
has all the FreeBSD stuff. Login to the control node and:
This will seriously degrade the performance of the upgrade process due
to the inefficiencies of disk writes when shadows are present, but it is
worth it to avoid a total screw up.
==========
# apt
sudo lvcreate -s -L 33g -n boss.backup xen-vg/boss
sudo lvcreate -s -L 33g -n ops.backup xen-vg/ops
A. Updating the base FreeBSD system with freebsd-update
# cloudlab utah/clemson
sudo lvcreate -s -L 17g -n boss.backup xen-vg/boss
sudo lvcreate -s -L 17g -n ops.backup xen-vg/ops
-1.Things you can do in advance.
This will seriously degrade the performance of the upgrade process due
to the inefficiencies of disk writes when shadows are present, but it is
worth it to avoid a total screw up.
* The "fetch" part of freebsd-update (A2 and A4)
* Build any custom kernel (A3)
* Stash state about ports/packages (B0)
2. Fetch the new release with freebsd-update.
0. If you are on the boss node, shutdown the testbed and mysqld right off
the bat. The ports upgrade is going to replace mysql and testbed daemons
will wig out if running when this happens. On ops, you just need to
shutdown mysqld:
This will not install anything, it will just fetch the new files and merge
local changes in. You can do this on both boss and ops simultaneously.
boss:
sudo /usr/testbed/sbin/testbed-control shutdown
sudo /usr/local/etc/rc.d/2.mysql-server.sh stop
sudo /usr/local/etc/rc.d/capture stop
sudo /usr/local/etc/rc.d/apache22 stop
ops:
sudo /usr/local/etc/rc.d/1.mysql-server.sh stop
sudo /usr/local/etc/rc.d/apache22 stop
1. Before installing the new binaries/libraries/etc., you might want
to back up the files that have Emulab changes just in case. Those files
are:
Do not do it too far (i.e., more than a day) in advance, since the base
system changes and your local mods may change as well. For example, new
users might be added in the interim which would invalidate your merged
changes.
/etc/hosts
/etc/ntp.conf # if you have customized it
/etc/ssh/sshd_config
/etc/ttys # if you have configured a serial console
The easiest thing to do is just:
sudo cp -rp /etc /Oetc
NOTE: if you built and installed your system from sources originally,
you may also get some conflicts with other files that record their origin
in comments. Sendmail files are one example. Use the 11.1 versions of
those to avoid future conflicts.
Before fetching, make sure your /etc/freebsd-update.conf is correct,
in particular the "Components" line.
2. I generally don't want to update /usr/src (if it exists at all I keep
it up to date with svn) or the kernel (if you have a custom kernel).
So you will want to modify /etc/freebsd-update.conf and change the
"Components" line to either:
By default it will want to update your kernel ("kernel") and source tree
("src") as well as the binaries ("world"). Life will be much easier if you
go with the flow and just let it do that. However, if you have a custom
source tree (or update it yourself with svn or git) then remove "src"
from the line:
Components world kernel # don't update src
Components world kernel # don't update src
or
If you have a custom kernel, then remove "kernel":
Components world # don't update src or kernel
Components world # don't update src or kernel
However, because you are changing major releases, rebuilding your
custom kernel (next step) will require rebuilding the entire world first,
......@@ -92,11 +69,51 @@ A. Updating the base FreeBSD system with freebsd-update
for the default GENERIC kernel, make sure to leave "kernel" in the
components above.
Once you have /etc/freebsd-update.conf squared away, do the "fetch"
part of the upgrade:
sudo freebsd-update -r 11.1-RELEASE upgrade
Since this will ask you to merge a bunch of local changes into various
files and will want to fire up an editor, you might want to make sure
you get a *real* editor by doing:
sudo -E EDITOR=emacs freebsd-update -r 11.1-RELEASE upgrade
instead. Otherwise you will probably wind up with vi.
It will crunch for a long time and then probably want you to merge
some conflicts. Here are a couple to take note of:
* /etc/ssh/sshd_config: make sure Protocol does not include 1,
otherwise it will spit out constant warnings to the console.
* /etc/ttys: for Xen VMs make sure the getty on ttyu0 is "off"
and not "onifconsole". Otherwise you will have competing gettys
on /dev/console.
NOTE: if you built and installed your system from sources originally,
you may also get some conflicts with other files where it calls out diffs
is the RCS header or in comments. Favor the newer versions of those to
hopefully avoid future conflicts.
NOTE: if it shows you a diff and asks you if something "looks reasonable"
and you answer "no", it will dump you out of the update entirely and
you have to start over. It will *not* just let you fire up the editor
and fix things!
It will then show you several potentially long lists of files that it
will be adding, deleting, etc. It uses "more" to display them, so you
can 'q' out of those without dumping out of the update entirely (the
last one will exit the update, but that is because it is done).
3. (Optional) Upgrade your custom kernel
If you have a custom kernel config, then you should build and install
a new kernel first. As mentioned in the last step, this will take a long
time because you must build (but not install) the entire world before
building the kernel. Clone the FreeBSD 11.1 source repo:
building the kernel. You can again to this on boss and ops simultaneously.
Clone the FreeBSD 11.1 source repo:
cd /usr
sudo mv src Osrc
......@@ -107,37 +124,78 @@ A. Updating the base FreeBSD system with freebsd-update
sudo make -j 8 buildworld
sudo make -j 8 buildkernel KERNCONF=CUSTOM
# XXX I wait to install the custom kernel til after doing the
# freebsd-update, since it will want to patch /boot/kernel files...
#sudo make installkernel KERNCONF=CUSTOM
4. Stash away the current set of packages you have installed.
4. Now upgrade the base system:
This will allow you to figure out the extra ports you have installed so
that you can update them later. First make a list of everything installed:
Do this on boss and then on ops. For boss:
sudo freebsd-update -r 11.1-RELEASE upgrade
mkdir ~/upgrade
cd ~/upgrade
pkg query "%n-%v %R" > boss.pkg.list
or, if you want a real editor:
This is mostly to keep track of any ports you may have installed locally.
One way to determine local installs is to see which ports did NOT come
from the Emulab repository:
sudo -E EDITOR=emacs freebsd-update -r 11.1-RELEASE upgrade
grep -v 'Emulab$' boss.pkg.list | awk '{ print $1; }' > boss.pkg.local
It will crunch for a long time and then probably want you to merge
some conflicts. Here are a couple to take note of:
This will give you the list of packages that you may need to reinstall.
* /etc/ssh/sshd_config: make sure Protocol does not include 1,
otherwise it will spit out constant warnings to the console.
You may want to list the dependencies of each to see what the top-level
packages are and just install those.
* /etc/ttys: for Xen VMs make sure the getty on ttyu0 is "off"
and not "onifconsole". Otherwise you will have competing gettys
on /dev/console.
pkg query -x "%n %v usedby=%#r" `cat boss.pkg.local` | \
grep 'usedby=0' | awk '{ print $1; }' > boss.pkg.reinstall
Now login to ops and do the same thing:
cd ~/upgrade
pkg query "%n-%v %R" > ops.pkg.list
grep -v 'Emulab$' ops.pkg.list | awk '{ print $1; }' > ops.pkg.local
pkg query -x "%n %v usedby=%#r" `cat ops.pkg.local` | \
grep 'usedby=0' | awk '{ print $1; }' > ops.pkg.reinstall
B. Updating the base FreeBSD system
1. If you are on the boss node, shutdown the testbed and some other services
right off the bat.
boss:
sudo /usr/testbed/sbin/testbed-control shutdown
sudo /usr/local/etc/rc.d/2.mysql-server.sh stop
sudo /usr/local/etc/rc.d/apache22 stop
sudo /usr/local/etc/rc.d/capture stop
ops:
sudo /usr/local/etc/rc.d/1.mysql-server.sh stop
sudo /usr/local/etc/rc.d/apache22 stop
sudo /usr/local/etc/rc.d/capture stop
It will then show you a long, long list of files that it will be adding,
deleting, etc. You can 'q' out of those.
2. Before installing the new binaries/libraries/etc., you might want to back
up the files that have Emulab changes just in case. Those files are:
5. Install the new system binaries/libraries/etc:
/etc/hosts
/etc/ntp.conf # if you have customized it
/etc/ssh/sshd_config
/etc/ttys # if you have configured a serial console
The easiest thing to do is just:
sudo cp -rp /etc /Oetc
3. Install the new system binaries/libraries/etc:
If it has been more than a day or so since you did the "upgrade"
command back in step A2, then you might consider doing it again.
Doing it again basically throws away everything it built up on the
previous run and you will have to go through all the manual merging
again. Once you are satisfied, do the install of the new binaries:
sudo /usr/sbin/freebsd-update install
After a while it will want you to reboot the new kernel. Before you reboot,
install any custom kernel:
After a while it will want you to reboot the new kernel. Before you reboot,
if you built a custom kernel back in step A3, install it now:
cd /usr/src
sudo make installkernel KERNCONF=CUSTOM
......@@ -157,6 +215,7 @@ A. Updating the base FreeBSD system with freebsd-update
Before you do this, make sure you first have access to the console!
sudo shutdown now
umount -at nfs
umount -at ufs
accton # turn off accounting that has a file open on /
......@@ -170,14 +229,15 @@ A. Updating the base FreeBSD system with freebsd-update
restarted, including some that won't work right.
boss:
sudo /usr/local/etc/rc.d/apache22 stop
sudo /usr/local/etc/rc.d/2.dhcpd.sh stop
sudo /usr/local/etc/rc.d/2.mysql-server.sh stop
sudo /usr/local/etc/rc.d/capture stop
sudo /usr/testbed/sbin/testbed-control shutdown
sudo /usr/local/etc/rc.d/apache22 stop
sudo /usr/local/etc/rc.d/2.dhcpd.sh stop
sudo /usr/local/etc/rc.d/2.mysql-server.sh stop
sudo /usr/local/etc/rc.d/capture stop
ops:
sudo /usr/local/etc/rc.d/apache22 stop
sudo /usr/local/etc/rc.d/1.mysql-server.sh stop
sudo /usr/local/etc/rc.d/apache22 stop
sudo /usr/local/etc/rc.d/1.mysql-server.sh stop
and then again run freebsd-update to finish:
......@@ -192,62 +252,30 @@ A. Updating the base FreeBSD system with freebsd-update
sudo diff -r /Oetc /etc
Of course, this will show you every change made by the update as well,
so you might just want to focus on the files above. When you are happy:
so you might just want to focus on the files listed in B2 above. When
you are happy:
sudo rm -rf /Oetc
6. The mothership may need some additional local hacks to some standard
4. The mothership may need some additional local hacks to some standard
utilities, in particular "mountd" and "pw". Both should have a patch
in the Emulab source tree patches subdir.
7. How did that work out for ya?
5. How did that work out for ya?
If all went well, skip to B (Updating ports/packages).
If all went well, skip to C (Updating ports/packages).
If that didn't work, see ~mike/upgrade-from-10.0.txt and follow steps
A1 - A10. Return here for upgrading your ports.
B. Updating ports/packages
C. Updating ports/packages
Updating the core ports from 10.3 to 11.1 is pretty easy. However, if
you installed extra ports that will require a bit more work.
0. Figure out the extra ports you have installed so that you can update
them later. First make a list of everything installed:
pkg query "%n-%v %R" > boss.pkg.list
This is mostly to keep track of any ports you may have installed locally.
One way to determine local installs is to see which ports did NOT come
from the Emulab repository:
grep -v 'Emulab$' boss.pkg.list | awk '{ print $1; }' > boss.pkg.local
This will give you the list of packages that you may need to reinstall.
You may want to list the dependencies of each to see what the top-level
packages are and just install those.
pkg query -x "%n %v usedby=%#r" `cat boss.pkg.local` | \
grep 'usedby=0' | awk '{ print $1; }' > boss.pkg.reinstall
====
IMPORTANT NOTE: for those with Moonshot chassis, you cannot use an
ipmitool port newer than 1.8.15 due to issues with "double bridged"
requests. Either ipmitool or HPE got it wrong and it doesn't behave
like ipmitool expects as of commit 6dec83ff on Sat Jul 25 13:15:41 2015.
Anyway, you may want to "pkg lock" that package in this case as the
FreeBSD 11.1 ports have a newer version.
====
For ops, those commands are:
pkg query "%n-%v %R" > ops.pkg.list
grep -v 'Emulab$' ops.pkg.list | awk '{ print $1; }' > ops.pkg.local
pkg query -x "%n %v usedby=%#r" `cat ops.pkg.local` | \
grep 'usedby=0' | awk '{ print $1; }' > ops.pkg.reinstall
0. If you forgot to save off your package info back in A4, or it has been
awhile, then you might want to go back and do that now.
1. Modify your /etc/pkg/Emulab.conf file, replacing "10.3" with "11.1" in
the "url" line:
......@@ -259,10 +287,10 @@ B. Updating ports/packages
sudo pkg unlock pkg
sudo -E ASSUME_ALWAYS_YES=true pkg upgrade -r Emulab
3. Fix perl/python paths.
3. Tweak package installs:
REALLY, REALLY IMPORTANT: the latest perl port does not install the
/usr/bin/perl link which we use heavily in Emulab scripts. Ditto for
REALLY, REALLY IMPORTANT: at some point, the perl port stopped installing
the /usr/bin/perl link which we use heavily in Emulab scripts. Ditto for
python and the /usr/local/bin/python link. Make sure those two symlinks
exist, e.g.:
......@@ -270,21 +298,39 @@ B. Updating ports/packages
If not, get them back with:
ln -sf /usr/local/bin/perl5 /usr/bin/perl
ln -sf /usr/local/bin/python2 /usr/local/bin/python
sudo ln -sf /usr/local/bin/perl5 /usr/bin/perl
sudo ln -sf /usr/local/bin/python2 /usr/local/bin/python
REALLY, REALLY IMPORTANT PART 2: Because perl changed, you will need
to make sure that the event library SWIG-generated perl module is rebuilt,
and then all the event clients. Otherwise you will get bus errors when
they all try to start. So do not skip step D2 below!
they all try to start. So do not skip step E2 below!
REALLY, REALLY IMPORTANT PART 3: For those with Moonshot chassis,
you cannot use an ipmitool port *newer* than 1.8.15 due to issues with
"double bridged" requests. Either ipmitool or HPE got it wrong and it
doesn't behave like ipmitool expects as of commit 6dec83ff on
Sat Jul 25 13:15:41 2015. Anyway, you will need to relace the standard
ipmitool install with the "emulab-ipmitool-old-1.8.15_1" package from
the emulab repository:
sudo pkg delete ipmitool
sudo pkg install -r Emulab emulab-ipmitool-old
But ONLY do this if you have Moonshot chassis.
4. Fix apache setup.
We upgraded from apache 2.2 to apache 2.4 this go round, so you will have
to tweak your /etc/rc.conf file and change any instances of "apache22" to
"apache24":
"apache24" and copy over your certificates from the 2.2 install:
sudo sed -i .bak -e 's;apache22;apache24;g' /etc/rc.conf
sudo cp -r /usr/local/etc/apache2{2,4}/ssl.crt
sudo cp -r /usr/local/etc/apache2{2,4}/ssl.key
sudo sed -i .bak -e 's;apache22;apache24;g' /etc/rc.conf
This is just the first part. One more thing will need to be done later
when updating the Emulab software.
5. Reinstall local ports.
......@@ -305,9 +351,9 @@ B. Updating ports/packages
DEFAULT_VERSIONS=perl5=5.26 python=2.7 php=5.6 mysql=5.7 apache=2.4 tcltk=8.6
DEFAULT_VERSIONS+=ssl=base
C. Repeat steps A and B for ops.
D. Repeat steps B and C for ops.
D. Update Emulab software
E. Update Emulab software
1. Make sure your Emulab sources are up to date.
......@@ -342,14 +388,11 @@ D. Update Emulab software
# on ops -- do this first
sudo gmake opsfs-install
cd apache ; sudo gmake control-install
sudo cp -r /usr/local/etc/apache2{2,4}/ssl.crt
sudo cp -r /usr/local/etc/apache2{2,4}/ssl.key
# on boss -- do this after ops
sudo /usr/local/etc/rc.d/2.mysql-server.sh start
sudo gmake all boss-install
cd apache ; sudo gmake install
sudo cp -r /usr/local/etc/apache2{2,4}/ssl.crt
sudo cp -r /usr/local/etc/apache2{2,4}/ssl.key
The reason for the ops install is that, while boss-install updates
most of the ops binaries/libraries via NFS, there are some that it
......@@ -365,18 +408,17 @@ D. Update Emulab software
not have to do #3 below. Note also that this command may take awhile
and provide no feedback.
3. Re-run the freebsd-update again to remove old shared libraries.
3. Re-enable the testbed on boss.
Now that everything has been rebuilt:
sudo /usr/local/etc/rc.d/apache22 start
sudo /usr/local/etc/rc.d/2.dhcpd.sh start
sudo /usr/testbed/sbin/testbed-control boot
sudo freebsd-update install
4. Re-run the freebsd-update again to remove old shared libraries.
4. Re-enable Emulab services on boss and make sure nothing obvious blows up:
Now that everything has been rebuilt:
sudo /usr/local/etc/rc.d/2.dhcpd.sh start
sudo /usr/local/etc/rc.d/2.mysql-server.sh start
sudo /usr/local/etc/rc.d/apache22 start
sudo /usr/testbed/sbin/testbed-control boot
sudo freebsd-update install
5. Reboot boss and ops again!
......@@ -385,7 +427,7 @@ D. Update Emulab software
sudo /usr/testbed/sbin/eventsys_start
E. Update the MFSes
F. Update the MFSes
This is not strictly part of updating the OS, but it would be good to
do this if you have not for awhile. See the instructions in
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment