Commit 07367359 authored by Mike Hibler's avatar Mike Hibler
Browse files

Add some intro info. Still need to finish discussion of security.

parent 6b4605e1
Some notes on the steps done during a client self-configuration:
Rough order of RC files called [ and tmcc calls used ]:
I. Introduction.
Emulab uses a largely "node driven" strategy for node configuration; i.e.,
it is a pull-style process with the nodes requesting information from Emulab
Central and configuring themselves.
We use the same basic framework for configuring:
* local cluster physical and virtual nodes
* local wireless nodes
* IXP network processors
* Stargate SBCs
* remote internet nodes
* planetlab "nodes" (slivers)
and for OSes including:
* FreeBSD 4, 5, 6, and 7
* OpenBSD 3
* Redhat 7, 9 and Fedora 4, 6
* Ubuntu 7.0
* Windows XP
The self-configuration strategy requires that the OSes to be run on nodes
have Emulab client code installed to handle the self-configuration. The
configuration code is mostly perl and shell scripts, with a couple of
C and C++ programs. The code resides mostly in the /etc/emulab and
/usr/local/etc/emulab directories on the node with a single hook placed
in /etc/rc.d (or /etc/rc.local or ...) to trigger the configuration process.
In the case of virtual nodes (or "subnodes" which are otherwise dependent on
a "host"; e.g., IXPs), the configuration may be split between the physical
(host) and virtual (sub) nodes. In this situation, the physical node boots
and begins self-configuration via the scripts. Part of that self-config in
this case is to create, initialize and boot any virtual nodes that it hosts.
Typically, the physical host will perform only the initializations that the
virtual hosts themselves do not have the privilege to do; e.g., initializing
(virtual) network interfaces. Ideally, most virtual node configuration will
then be done in the context of the vnodes themselves. This requires that the
virtual nodes' filesystems contain a copy of the Emulab startup scripts.
The first step in the configuration process if for a node to identify the
so-called "boss" node (Emulab Central) and (possibly) configure the network
to talk to it. How this is done depends on the type of the node. For
cluster and wireless physical nodes which are connected directly to the
Emulab control network, the nodes DHCP to discover and configure the control
network and then (somewhat arbitrarily) use the returned name server as the
boss node. Remote physical nodes have the boss identity hardwired in (usually
on a CD, flash dongle, or floppy) and use their default IP routing setup to
reach it.
As mentioned there are a set of scripts, each responsible for initializing
some part of a node's configuration; e.g., accounts, NFS shared mount points,
network interfaces, routing, etc. The scripts are invoked with one of four
actions: boot, shutdown, reconfig or reset. The first three pretty much do
what you would expect: set things up at boot time, tear things down at
shutdown, and perform a dynamic reconfiguration while running. The last is
a special action used when creating disk images; it is invoked to cleanup
any on-disk state created by the boot or reconfig actions.
While the same core set of scripts and libraries are used for initializing
all node types, many of the scripts have large quantities of conditionalized
code for various node types. In the OS dimension, there is much less runtime
conditionalized code. There is a common set of scripts and libraries for
all OSes and distinct sets of per-OS scripts and libraries.
In general, the scripts operate in one of two ways: either they do the
appropriate setup themselves, or they create another set of shell scripts
in /var/emulab/boot which are invoked to do the appropriate setup. The
exact intent of the latter method is lost in the mists of time, but I think
we had envisioned a more efficient boot path, with nodes contacting Emulab
Central (see the next paragraph) only at initial boot up and then creating
the "fast path" boot scripts to use for future reboots. However, this is
not what we do now. Reboots and reconfigurations always re-contact Emulab
Most of the configuration information is passed to the nodes through the
Testbed Master Control Protocol via the programs tmcc (the client) and
tmcd (the server). Note that we generally refer to the protocol as TMCC
or TMCD rather than TMCP, because...well, just because that is what we do!
TMCC/TMCD/TMCP is a custom ASCII-based, total hack protocol. The API of
TMCC is eclectic. For the most part it consists of single word requests
about a particular type of information for a single node (e.g., "ifconfig",
"routes", "accounts"). The returned information is in one or more
easily-parsable lines of KEY=VALUE pairs. Typically, the target node
of the command is implied by the IP address from which the request comes,
though the target may also be explicitly specified.
Evolution of the API is largely "convenience driven." If there is something
small we need quickly, we tend to add a command to TMCC in the most
straightforward manner. TMCC is by no means the future of node-configuration
The tmcc client is just a mechanism for returning DB information, it does
not perform any actions on its own. For example you use tmcc to request
information about which user accounts should be set up on a node, it does
not actually modify the password file. Invocations of tmcc are made from
shell/perl scripts which do the actual client-side customization. The
client can communicate with the server via UDP, TCP or SSL over TCP.
Some commands can only be performed over certain transports. For example,
account info can only be returned via SSL+TCP since a password hash is
part of the returned info.
Security in TMCC. You might want to sit down for this. Authentication of
node TMCC requests to Emulab Central is provided in a couple of ways.
II. Rough order of RC files called [ and tmcc calls used ].
* identify boss [ bossinfo ]
* rc.{linux,freebsd}
* rc.kname
* tbshutdown (a daemon)
* run tbshutdown (a daemon)
Enter TBSETUP state
Node enters the TBSETUP state
* bootsetup [ status ]
* get all tmcc info [ fullconfig ]
......@@ -44,7 +146,7 @@ Rough order of RC files called [ and tmcc calls used ]:
* rc.startcmd [ startup ]
* bootvnodes [ vnodelist ]
TMCC calls:
III. TMCC calls:
Return info about the "boss" server to which to make TMCC calls.
......@@ -152,15 +254,22 @@ hostnames:
Return a list of tarballs to download and install on the node,
one line per tarball. Tarball path is absolute and typically on a
shared NFS filesystem.
one line per tarball. The returned tarball path is an absolute path
on the server.
DIR=<dir to "cd" to before install> TARBALL=<tarfile>
DIR=<dir to "cd" to before install> TARBALL=<tarball filename>
The tarball itself can be fetched either across NFS (if on a node-shared
filesystem) of via HTTPS from the Emulab boss (see the os/install-tarfile
Return a list of RPMs to download and install on the node,
one line per RPM. RPM path is absolute and typically on a
shared NFS filesystem.
Return a list of RPMs to download and install on the node, one line
per RPM. The returned RPM path is an absolute path on the server.
RPM=<RPM filename>
The RPM itself can be fetched either across NFS (if on a node-shared
filesystem) of via HTTPS from the Emulab boss (see the os/install-rpm
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment