The New "pc3000" Nodes
Contents
Machines and Interfaces
The new "pc3000" machines are
Dell PowerEdge 2850s
with a single 3GHz processor, 2GB of RAM, and 2 10,000 RPM 146GB SCSI disks.
Each has 4 available Gigabit experimental net interfaces,
however due to lack of Gb switch ports, not all are usable at Gb speeds.
Instead, they are a mix of 100Mb and 1Gb as follows:
# of Machines Gb ports 100Mb ports Machine names
40 4 0 pc201-pc240
20 3 1 pc241-pc260
40 2 2 pc281-pc320
60 1 3 pc261-pc280, pc321-pc360
Note that all 160 machines have at least 1 Gb interface and all can be
used for 100Mb as well.
There is also a 5th Gb interface on each machine, that is used to pairwise
connect adjacent machines. For example, pc201 is connected directly to pc202
via a cross over cable. These interfaces are not yet available, pending
further support in the resource mapper.
All machines have serial console lines accessible from users.emulab.net via
the "console" command or remotely via the
tiptunnel interface.
Images and Kernel Support
Currently, the Emulab standard FreeBSD (FBSD410-STD, FBSD54-STD),
Linux (RHL90-STD, FC4-STD, "legacy" RHL73-STD),
and Windows (WINXP-UPDATE-pc3000, WINXP-SP{0,1,2}-pc3000)
images will run on the new machines.
If you have built custom images based on our standard images before
Sept 2 2005, they will likely not work on the new machines due to a lack
of the correct disk driver. You will either need to re-customize based
on the current images or modify your existing image to add the correct
SCSI driver. For BSD, add the following to your kernel config file:
device mpt
For Linux, add the following to your .config:
CONFIG_FUSION=y
CONFIG_FUSION_BOOT=y
CONFIG_FUSION_MAX_SGE=40
CONFIG_FUSION_ISENSE=m
NOTE: The default Pro/1000 ethernet driver (e1000) included in
the RedHat 9.0 kernel sources does not work on the pc3000s. This includes
the emulab-patched RHL90 source tree! When creating a custom RH9-era kernel,
you must build and install an updated version of the e1000 driver module.
You can get it here: /share/linux/e1000-5.5.4.tar.gz .
Please read the README inside this tarball for build/install
instructions.
For Windows you will need to re-customize based on our current -pc3000
images.
Caveats
- No bandwidth shaping between 100Mb and 1000Mb.
We have not yet looked at providing bandwidth shaping values between
100Mb and 1000Mb.
- The "direct connect" 5th interface is not usable.
As mentioned, we need modifications to the resource mapper
and there are also security issues to be resolved.
- Interswitch bandwidth may limit topologies in non-intuitive ways.
Since the Gb interfaces and the 100Mb interfaces are on different
switches, and the new machines are on different switches than the old
pc600 and pc850 machines, the bandwidth of the interswitch links can
have a significant impact on whether an experimental topology can map
or not.
- No vnode characterization has been done.
The so-called "colocation factor" has been arbitrarily set to 75 right
now. It will likely be clamped down considerably.
- There might be assorted failures due to scaling issues.
Experiments of over 150 machines, which were hard to do before,
may reveal further scaling issues that will need to be addressed.
Expect an increase in unexpected swapin failures as a result.
- These machines are radically different than the old machines.
Since these machines are so much faster, have so much more and faster
memory, have so much more and faster disk, results of your experiments
on these machines could be dramatically different. That is, if your
applications were CPU, memory, or disk intensive. If you are
interested in reproducing old results, you might want to
limit your experiments to using only pc600 and pc850 nodes.
- Hyperthreading is available, but SMP support is
not currently compiled in to the stock Emulab images (with the
exception of RHL73-STD). If you want it, you
will have to compile your own kernel with SMP enabled and create a disk
image. Note that FreeBSD 4.x SMP kernels will panic on a uniprocessor
machine, so a combined 4.x image that runs on all Emulab nodes is not
currently possible. We plan to upgrade our Linux and FreeBSD 5 images
to support SMP in the near future as well as address the 4.x SMP issue
for standard images.
- The nodes run in 32-bit mode.
The processors in these machines are the Intel EM64T Xeons, but we have
not yet created 64-bit native versions of FreeBSD, Linux or Windows.
- The network interfaces use "interrupt throttling."
The Intel Pro1000 cards used for the experimental interfaces in
these machines default to using interrupt throttling. This will
add latency to packet delivery and reception. See the
knowledge base entry for how to change or disable this feature.