Commit 095af76e authored by Mike Hibler's avatar Mike Hibler
Browse files

Assorted files contained in the "generic image" tarball.

Checked in so we can track their evolution and so you don't
have to rely on Mike to remember where he stashed them!
parent e6375a7a
This directory contains scripts/directions included in the "generic image"
tarball. It does not currently contain any instructions for Emulab-types
on how to create the actual images or the tarball. Talk to Mike about that.
META-README This file, not included in the tarball
README Directions for using the tarball
setup-images.sql SQL snippets for populating the DB with the images.
setup-mfsosids.sql
setup-o2i.sql
setup-osids.sql
Instructions for creating a site-customized image from a "generic" image
provided by Utah. The basic procedure is:
- load the image on a testbed machine
- customize both FreeBSD and Linux filesystems
- save customized image
We will provide you with a generic disk image. This disk image is one
which has both a FreeBSD partition and a Linux partition. You will need
to customize both. By convention FreeBSD is in DOS partition #1 and Linux
in DOS partition #2.
You will need to have the Emulab network-booted, memory-filesystem-based,
FreeBSD systems (hereafter known as the MFSes). If you have not done this
yet, do it now. If you do not know what I am talking about, contact
testbed-ops@emulab.net.
In particular, the customization of the disk partitions is done using a
node booted into the "admin" MFS. If you have followed the Emulab setup
directions and have added nodes to your testbed to the point where they
are now in the "hwdown" experiment, they should already be in the admin MFS.
At this point you should be able to pick one and slogin as root from your
boss machine.
If the node does not respond to slogin and you have serial consoles hooked
up, connect to the console, reboot the node and wait for the Emulab pxeboot
prompt:
Type a key for interactive mode (quick, quick!)
So hit the space bar (quick, quick!) and you go into interactive mode
where you can tell it to boot from the admin MFS:
loader:/tftpboot/freebsd
Now on with the show.
A. Load the image on a testbed machine.
We will assume the machine is called "pc1" in the following directions.
* Put the Utah-provided generic image in /usr/testbed/images on your
boss machine. The generic image will be loaded, and the custom
image saved, using this directory.
* Boot pc1 into the admin MFS as described above
* Figure out what your root disk is. You can look through the boot
time output of FreeBSD on pc1 to find it. If you missed that output,
you can ssh into pc1 and run "dmesg":
sudo ssh pc1 dmesg
If you have IDE disks, the disk you want will probably be "ad0".
If you have SCSI disks, it will be "da0". For SATA disks, it will
probably be "ad4". RAID controllers are variously things like: "ar",
"aacd", "twed", depending on the controller you have. If you cannot
find anything in the output that looks like a disk, you may have an
unsupported disk controller. Contact testbed-ops@emulab.net if this
happens (and be sure to have your "dmesg" output handy!)
If your disk is anything other than "ad0", there are several files
under both FreeBSD and Linux that will need to be "fixed". The good
news is that the post-disk-load customization pass should do this
for you. Just make sure that the Emulab database node_types table
has the correct value of disktype for each node type. But, you still
need to know the disk type for the following steps, so lets set a
shell variable:
set DSK=<your-disk-here> # e.g. "ad0", "da0", "ad4"
or
DSK=<your-disk-here>; export DSK
* Use 'imageunzip' to load the disk.
If all has gone well, the node should be up and accessible via ssh.
To copy the generic image onto the test machine disk, do the following
from boss:
sudo ssh pc1 imageunzip - /dev/$DSK \
< /usr/testbed/images/FBSD410+RHL90-GENERIC.ndz
Image loading should take anywhere from 45 seconds, to several minutes.
If the ssh returns with "Killed" then imageunzip ran out of memory.
By default, imageunzip will consume memory without bound for buffering
of pending disk writes. If it grow too big, the system will kill it.
In this case, retry the imageunzip with "-W <num-MB>" where <num-MB>
is a number of MB maximum to use for disk buffering. If the node you
Using about half of the available physical memory should be safe
(e.g., if the machine are loading has 512MB of memory, try "-W 256").
Now you can begin customizing the FreeBSD and Linux partitions.
B. Customize FreeBSD:
First, login as root from boss and set that magic disk variable:
sudo slogin pc1
set DSK=<your-disk-here> # you *will* be in csh here
and mount the FreeBSD filesystems on the disk:
mount /dev/${DSK}s1a /mnt
mount /dev/${DSK}s1e /mnt/var
mount /dev/${DSK}s1f /mnt/usr
Now you can update the necessary files as follows.
The MFS has a much scaled-down set of binaries. To get access to a more
full-featured system, you can run binaries from the disk image itself:
ldconfig /mnt/usr/lib /mnt/usr/X11R6/lib /mnt/usr/local/lib
set path=($path /mnt/sbin /mnt/bin /mnt/usr/sbin /mnt/usr/bin)
Now update the following files:
* /mnt/root/.ssh/authorized_keys
Put in local boss root pub key. Leave in Utah (Emulab) pub key if
acceptable (if you want/need our help debugging). Otherwise, remove it.
* /mnt/etc/localtime
Copy the correct file over from /mnt/usr/share/zoneinfo
* /mnt/etc/master.passwd
Change the root password. The password needs to be changed in the
etc/emulab subdirectory as well:
chroot /mnt passwd root
<set password>
cp -p /mnt/etc/master.passwd /mnt/etc/emulab/
* /mnt/etc/ssh/ssh_host*
We use the same host key for all images and all OSes. If you correctly
customized your MFSes, you have already generated a set of site-specific
host keys, and you can copy them to the disk with:
cp -p /etc/ssh/ssh_host* /mnt/etc/ssh/
and then skip to the next bullet item.
If you did NOT generate host keys for your MFSes, you can generate
keys now with:
ssh-keygen -t rsa1 -N "" -f /mnt/etc/ssh/ssh_host_key
ssh-keygen -t rsa -N "" -f /mnt/etc/ssh/ssh_host_rsa_key
ssh-keygen -t dsa -N "" -f /mnt/etc/ssh/ssh_host_dsa_key
This installs them in the disk image, you will still have to go back and
install these same keys in the sources for your frisbee/freebsd MFSes later
using the updating instructions in the README file in the MFS tarball.
So save the keys from /mnt/etc/ssh off somewhere.
* /mnt/etc/emulab/{client,emulab}.pem
These should have been created on your boss node when you did the boss
setup. So from your boss node do:
sudo scp -p /usr/testbed/etc/{client,emulab}.pem pc1:/mnt/etc/emulab/
That is it for FreeBSD. Now remount the filesystems read-only so you
can still run binaries but don't accidentally clobber anything:
cd /
mount -u -o ro /mnt/usr
mount -u -o ro /mnt/var
mount -u -o ro /mnt
and move on to updating the Linux partition on the disk.
C. Customize Linux:
Mount the Linux filesystems (recall that the FreeBSD filesystems are
still mounted on /mnt, so we use another directory):
mkdir /mnt2
mount -t ext2fs /dev/${DSK}s2 /mnt2
Now you can update the necessary files as follows.
* /mnt2/root/.ssh/authorized_keys
Copy over from the FreeBSD partition:
cp -p /mnt/root/.ssh/authorized_keys /mnt2/root/.ssh/
* /mnt2/etc/shadow
Copy in the password hash for root from the FreeBSD password file
(/mnt/etc/master.passwd) to /mnt2/etc/shadow. Then copy that file to
the emulab subdirectory:
cp -p /mnt2/etc/shadow /mnt2/etc/emulab/
* /mnt2/etc/localtime
Copy the correct file over from /mnt2/usr/share/zoneinfo
* /mnt2/etc/ssh/ssh_host*
Copy the host keys you created for FreeBSD above:
cp -p /mnt/etc/ssh/ssh_host* /mnt2/etc/ssh/
* /etc/testbed/{client,emulab}.pem
Copy over from the FreeBSD side:
cp -p /mnt/etc/emulab/*.pem /mnt2/etc/emulab/
Now unmount the Linux filesystem:
umount /mnt2
and fsck it for good luck. Actually, not only good luck but also to reset
the time stamp that forces a periodic fsck:
e2fsck /dev/${DSK}s2
D. Saving the customized image
The "whole disk" image is the one used most frequently, but we also
create single partition images of FreeBSD and Linux as well.
From boss do:
cd /usr/testbed/images
sudo ssh pc1 imagezip /dev/$DSK - > FBSD410+RHL90-STD.ndz
sudo ssh pc1 imagezip -s 1 /dev/$DSK - > FBSD410-STD.ndz
sudo ssh pc1 imagezip -s 2 /dev/$DSK - > RHL90-STD.ndz
E. Testing the image
Now that you have saved a copy of your customization, you can test out
the image and make sure it works.
NOTE: you will need console access to the node at this point, either
the VGA or the serial line depending on which version of pxeboot you
are using.
From your slogin session on the node do:
reboot
and wait for the pxeboot prompt to appear on the console as described
earlier. When the "Type a key..." message appears, hit space and then
at the prompt type:
part:1
which tells pxeboot to boot from partition 1 (aka, the FreeBSD partition).
The machine should proceed to boot all the way to a login prompt. Watch
for startup problems, like account setup or mount failures. If all goes
well, login to the console with the root password you set and reboot again.
When you get the pxeboot prompt again, type space and then:
part:2
to boot into the Linux partition. Again watch for problems. If
everything worked, skip the next step and proceed to "Installing the
images" below. If there was a catastrophic failure, you can reboot
the node into the admin MFS and reload the disk either with the image
snapshot you made or, worst case, the generic image. If you just
need to make some minor changes, make them and proceed with the next
step.
F. Recreating the image
If you need to tweak either the FreeBSD or Linux partitions, you will
need to save the image again. Doing this properly involves cleaning up
anything that the Emulab node self configuration might have done.
While you are running either FreeBSD or Linux you do the following
from the console:
shutdown now
<wait for single user root prompt>
cd /usr/local/etc/emulab
./prepare
reboot
As the node reboots, catch the pxeboot prompt and boot into the admin
MFS. Go back to the "Saving the customized image" step (D).
G. Installing the images
Once you have a working image, go back to your boss and do:
mv /proj/emulab-ops/images/*.ndz /usr/testbed/images/
and record these initial images in the Emulab database using the
SQL INSERT commands in setup-images.sql and setup-osids.sql.
As the database schema occasionally changes, it is important to
ensure that your DB matches the INSERT commands before doing them.
The table formats are listed at the beginning of those files.
You also need to create mappings from the OSIDs to the imageids
for every node type that supports booting these OSes (e.g., PCs
but not switches). You can either modify setup-o2i.sql in this
directory, modifying it as appropriate for your node_types or
you can use the web interface.
To use the web, login as the admin user and click the green dot
next to the "My Emulab.net" header. The dot should change to
red and you will be in admin mode. On the left menu bar there
should be a line "List ImageIDs or OSIDs". Click on the ImageIDs.
For both "RHL90-STD" and "FBSD410-STD", click on their names and
then choose "Edit this Image Descriptor" from the menu. Now
select the appropriate boxes in the "Node Types" line.
Dealing with SCSI disks (or RAID).
* For completeness here are the files that need to be changed if you have
other than IDE disks:
FreeBSD /etc/fstab: needs "ad0" changed to appropriate type
Linux /etc/fstab: needs "hda" changed to appropriate type
Linux /etc/lilo.conf: ditto, plus lilo must be rerun
Again, you should not need to mess with these files, the post-disk-loader
script will do it for you.
* For FreeBSD, we have four different kernels that are used for various
purposes. Only one of them is generic. So you will need to build
customized kernels in order to do "link delays", delay nodes and virtual
nodes. The image does include our versions of all these kernels, so
try those first and maybe save yourself some work. These kernels support:
disks: ad, ar, da (with ahc, ahd controllers)
network: xl, dc, fxp, em, nge
Copy /kernel.100HZ over to /kernel on a node and try booting it. If
that works and all your network interfaces were found, just create a
new image after copying /kernel.100HZ to /kernel (copy, don't move,
/kernel.100HZ needs to exist).
Otherwise, you need to build your own kernels. You can look at output from
the admin MFS (aka GENERIC) kernel if necessary. Add the necessary driver
to the various TESTBED configs in the kernel source, and rebuild and install
the kernels in the image. Mail testbed-ops@emulab.net for more info.
* For Linux, it is even more of a PITA. We currently don't even have a
generic Linux kernel in the image. So if Linux doesn't boot in the image,
you will have to configure/build a new kernel on some external machine.
If you don't have a Linux machine to do this with, contact us.
[ This file is a copy of doc/customize-image.txt in the Emulab source tree. ]
[ Last updated 11/11/04 ]
-- MySQL dump 8.23
--
-- Host: localhost Database: tbdb
---------------------------------------------------------
-- Server version 3.23.58-log
--
-- Table structure for table `images`
--
CREATE TABLE images (
imagename varchar(30) NOT NULL default '',
pid varchar(12) NOT NULL default '',
gid varchar(12) NOT NULL default '',
imageid varchar(45) NOT NULL default '',
creator varchar(8) default NULL,
created datetime default NULL,
description tinytext NOT NULL,
loadpart tinyint(4) NOT NULL default '0',
loadlength tinyint(4) NOT NULL default '0',
part1_osid varchar(35) default NULL,
part2_osid varchar(35) default NULL,
part3_osid varchar(35) default NULL,
part4_osid varchar(35) default NULL,
default_osid varchar(35) NOT NULL default '',
path tinytext,
magic tinytext,
load_address text,
frisbee_pid int(11) default '0',
load_busy tinyint(4) NOT NULL default '0',
ezid tinyint(4) NOT NULL default '0',
shared tinyint(4) NOT NULL default '0',
global tinyint(4) NOT NULL default '0',
updated datetime default NULL,
PRIMARY KEY (imagename,pid),
KEY imageid (imageid),
KEY gid (gid)
) TYPE=MyISAM;
--
-- Dumping data for table `images`
--
-- WHERE: imagename like '%-STD%'
INSERT INTO images VALUES ('FBSD410+RHL90-STD','emulab-ops','emulab-ops','emulab-ops-FBSD410+RHL90-STD','hibler','2004-11-09 00:00:00','Latest full disk image with FreeBSD 4.10 and RedHat 9.0',0,4,'emulab-ops-FBSD410-STD','emulab-ops-RHL90-STD',NULL,NULL,'emulab-ops-FBSD410-STD','/usr/testbed/images/FBSD410+RHL90-STD.ndz',NULL,'',0,1,0,0,1,NULL);
INSERT INTO images VALUES ('FBSD410-STD','emulab-ops','emulab-ops','emulab-ops-FBSD410-STD','hibler','2004-11-09 00:00:00','Testbed version of FreeBSD 4.10',1,1,'emulab-ops-FBSD410-STD',NULL,NULL,NULL,'emulab-ops-FBSD410-STD','/usr/testbed/images/FBSD410-STD.ndz',NULL,'',0,0,0,0,1,NULL);
INSERT INTO images VALUES ('RHL90-STD','emulab-ops','emulab-ops','emulab-ops-RHL90-STD','hibler','2004-11-09 00:00:00','Testbed version of RedHat Linux 9.0',2,1,NULL,'emulab-ops-RHL90-STD',NULL,NULL,'emulab-ops-RHL90-STD','/usr/testbed/images/RHL90-STD.ndz',NULL,'',0,0,0,0,1,NULL);
-- MySQL dump 8.23
--
-- Host: localhost Database: tbdb
---------------------------------------------------------
-- Server version 3.23.58-log
--
-- Table structure for table `os_info`
--
CREATE TABLE os_info (
osname varchar(20) NOT NULL default '',
pid varchar(12) NOT NULL default '',
osid varchar(35) NOT NULL default '',
creator varchar(8) default NULL,
created datetime default NULL,
description tinytext NOT NULL,
OS enum('Unknown','Linux','FreeBSD','NetBSD','OSKit','Other') NOT NULL default 'Unknown',
version varchar(12) default '',
path tinytext,
magic tinytext,
machinetype varchar(30) NOT NULL default '',
osfeatures set('ping','ssh','ipod','isup','veths','mlinks') default NULL,
ezid tinyint(4) NOT NULL default '0',
shared tinyint(4) NOT NULL default '0',
mustclean tinyint(4) NOT NULL default '1',
op_mode varchar(20) NOT NULL default 'MINIMAL',
nextosid varchar(35) default NULL,
max_concurrent int(11) default NULL,
mfs tinyint(4) NOT NULL default '0',
PRIMARY KEY (osname,pid),
KEY osid (osid),
KEY OS (OS),
KEY path (path(255))
) TYPE=MyISAM;
--
-- Dumping data for table `os_info`
--
-- WHERE: osid like "%MFS%"
INSERT INTO os_info VALUES ('FREEBSD-MFS','emulab-ops','FREEBSD-MFS','root',NULL,'FreeBSD in an MFS','FreeBSD','4.10','boss:/tftpboot/freebsd',NULL,'','ping,ssh,ipod,isup',0,1,0,'PXEFBSD',NULL,NULL,1);
INSERT INTO os_info VALUES ('FRISBEE-MFS','emulab-ops','FRISBEE-MFS','root',NULL,'Frisbee (FreeBSD) in an MFS','FreeBSD','4.10','boss:/tftpboot/frisbee',NULL,'','ping,ssh,ipod,isup',0,1,0,'RELOAD',NULL,NULL,1);
INSERT INTO os_info VALUES ('NEWNODE-MFS','emulab-ops','NEWNODE-MFS','root',NULL,'NewNode (FreeBSD) in an MFS','FreeBSD','4.10','boss:/tftpboot/freebsd.newnode',NULL,'','ping,ssh,ipod,isup',0,1,0,'PXEFBSD',NULL,NULL,1);
-- MySQL dump 8.23
--
-- Host: localhost Database: tbdb
---------------------------------------------------------
-- Server version 3.23.58-log
--
-- Table structure for table `osidtoimageid`
--
CREATE TABLE osidtoimageid (
osid varchar(35) NOT NULL default '',
type varchar(30) NOT NULL default '',
imageid varchar(45) NOT NULL default '',
PRIMARY KEY (osid,type)
) TYPE=MyISAM;
--
-- Dumping data for table `osidtoimageid`
--
-- WHERE: osid like '%-STD'
--
-- Change PCTYPE to an actual node_type in your database (e.g., "pc2800").
-- You will need a set of these lines for every node_type that can run BSD or Linux.
--
INSERT INTO osidtoimageid VALUES ('emulab-ops-FBSD410-STD','PCTYPE','emulab-ops-FBSD410-STD');
INSERT INTO osidtoimageid VALUES ('emulab-ops-RHL90-STD','PCTYPE','emulab-ops-RHL90-STD');
-- MySQL dump 8.23
--
-- Host: localhost Database: tbdb
---------------------------------------------------------
-- Server version 3.23.58-log
--
-- Table structure for table `os_info`
--
CREATE TABLE os_info (
osname varchar(20) NOT NULL default '',
pid varchar(12) NOT NULL default '',
osid varchar(35) NOT NULL default '',
creator varchar(8) default NULL,
created datetime default NULL,
description tinytext NOT NULL,
OS enum('Unknown','Linux','FreeBSD','NetBSD','OSKit','Other') NOT NULL default 'Unknown',
version varchar(12) default '',
path tinytext,
magic tinytext,
machinetype varchar(30) NOT NULL default '',
osfeatures set('ping','ssh','ipod','isup','veths','mlinks') default NULL,
ezid tinyint(4) NOT NULL default '0',
shared tinyint(4) NOT NULL default '0',
mustclean tinyint(4) NOT NULL default '1',
op_mode varchar(20) NOT NULL default 'MINIMAL',
nextosid varchar(35) default NULL,
max_concurrent int(11) default NULL,
mfs tinyint(4) NOT NULL default '0',
PRIMARY KEY (osname,pid),
KEY osid (osid),
KEY OS (OS),
KEY path (path(255))
) TYPE=MyISAM;
--
-- Dumping data for table `os_info`
--
-- WHERE: osid like '%-STD%'
INSERT INTO os_info VALUES ('RHL-STD','emulab-ops','emulab-ops-RHL-STD','root',NULL,'Default version of RedHat Linux','Linux','',NULL,'','','ping,ssh,ipod,isup',0,1,1,'NORMALv1','emulab-ops-RHL90-STD',NULL,0);
INSERT INTO os_info VALUES ('FBSD-STD','emulab-ops','emulab-ops-FBSD-STD','root',NULL,'Default version of FreeBSD','FreeBSD','',NULL,'','','ping,ssh,ipod,isup',0,1,1,'NORMALv1','emulab-ops-FBSD410-STD',NULL,0);
INSERT INTO os_info VALUES ('FBSD410-STD','emulab-ops','emulab-ops-FBSD410-STD','root','2004-11-09 00:00:00','Testbed version of FreeBSD 4.10','FreeBSD','4.10',NULL,'','','ping,ssh,ipod,isup',0,1,1,'NORMALv1',NULL,NULL,0);
INSERT INTO os_info VALUES ('RHL90-STD','emulab-ops','emulab-ops-RHL90-STD','root','2004-11-09 00:00:00','Testbed Version of RedHat Linux 9.0','Linux','9.0',NULL,'','','ping,ssh,ipod,isup',0,1,1,'NORMALv1',NULL,NULL,0);
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment