GitLab

We have had issues with uploading images to boss where they are then written
across NFS to ops. That seems to be a network hop too far on CloudLab Utah
where we have a 10Gb control network. We get occasional transcient timeouts
from somewhere in the TCP code. With the convoluted path through real and
virtual NICs, some with offloading, some without, packets wind up getting
out of order and someone gets far enough behind to cause problems.

So we work around it.

If IMAGEUPLOADTOFS is defined in the defs-* file, we will run a frisbee
master server on the fs (ops) node and the image creation path directs the
nodes to use that server. There is a new hack configuration for the master
server "upload-only" which is extremely specific to ops: it validates the
upload with the boss master server and, if allowed, fires up an upload
server for the client to talk to. The image will thus be directly uploaded
to the local (ZFS) /proj or /groups filesystems on ops. This seems to be
enough to get around the problem.

Note that we could allow this master server to serve downloads as well to
avoid the analogous problem in that direction, but this to date has not
been a problem.

NOTE: the ops node must be in the nodes table in the DB or else boss will
not validate proxied requests from it. The standard install procedure is
supposed to add ops, but we have a couple of clusters where it is not in
the table!

There are three pieces here, a change to the frisbee protocol itself, an
Emulab event component to get status back to the portal, and the surrounding
infrastructure to make it all work.

Frisbee heartbeat messages:

Added a new message type to the frisbee protocol, "Progress". In theory it
operates by having the server send a multicast progress request to its clients
which includes an interval at which to report (or "just once") and an
indication of what to report (nothing, progress summary, or full stats). The
client then sends unicast "fire and forget" UDP replies according to that
schedule. However, I took a shortcut for the moment and just added a command
line option to the client to tell it to report a summary at the indicated
interval (-H <interval>).  So the server never sends requests.

This is implemented in the client by a fourth thread since I wanted it to
operate independent of packet reception (which would cause clients to report
in a highly synchronized fashion due to multicast). The server instance just
logs progress reports into its log.

This protocol addition should be fully backward compatible as both client and
server ignore (but log) unknown messages.

Emulab progress report events:

When this is compiled in (-DEMULAB_EVENTS) and turned on (-E <server>), the
frisbee server instances will send a FRISBEEPROGRESS event to the indicated
event server for every progress report it receives (in addition to logging the
events to its own log). Right now it will create an event with key/value pairs
for the information in a client summary reply:

TSTAMP is the client's time at which it sends the event. Could be used by the
received to determine latency of the report if it cared (and if it assumed
that the clocks are in sync). We don't care about this.

SEQUENCE is the report number. Again, could be used by the receiver, in this
case to detect loss, if it cared. We don't.

CHUNKS_RECV is complete chunks that the client has received from the network.
CHUNKS_DECOMP is chunks decompressed by the client.  BYTES_WRITTEN is bytes
written to disk by the client.

Any of the three can be used by the event receiver as an indication of life
and/or progress. However, only the last would be a reasonable indicator of
time remaining since it is the last (and slowest) phase of imaging. To
estimate time remaining we could compare that value to the amount of
uncompressed data that is in the image. This makes the sketchy assumptions
that time for writes to the disk are uniform and that the number and distance
of seeks is uniform, but it is better than a sharp stick in the eye.

Emulab infrastructure:

There is a new sitevar "images/frisbee/heartbeat" which can be set to a
non-zero value to tell the frisbee MFS to fire off frisbee with -H <value>
and thus make reports. The default value of zero means to not make reports.
The tmcd "loadinfo" command sends this through via the HEARTBEAT=<value>
param.

REQUIRED A TMCD VERSION BUMP TO 41.

Finally just had to add a target for the mfs install since there
seemed to be no way to make CLIENT_BINDIR work for both /usr/local/bin
and /etc/testbed. Now Leigh and I can stop overwriting each others' commits.

This officially drops the pretense that fs nodes can operate with minimal
Emulab software. If you have a seperate fs node, it had better be dedicated
to Emulab!

However, it still doesn't do everything. In particular, accounts are not
installed. This has never been needed for serving NFS, but is needed for
the samba stuff to work correctly.

Also, you cannot do an fs node software install from boss yet as we do not
mount fs filesystems on boss. You really cannot do a full ops install from
boss either since we don't mount ops' /usr/local/etc/emulab directory.

Erroneously skipped installing dijkstra on Windows.

This commit is intended to makes the license status of Emulab and
ProtoGENI source files more clear.  It replaces license symbols like
"EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
blocks that contain actual license statements.

This change was driven by the fact that today, most people acquire and
track Emulab and ProtoGENI sources via git.

Before the Emulab source code was kept in git, the Flux Research Group
at the University of Utah would roll distributions by making tar
files.  As part of that process, the Flux Group would replace the
license symbols in the source files with actual license statements.

When the Flux Group moved to git, people outside of the group started
to see the source files with the "unexpanded" symbols.  This meant
that people acquired source files without actual license statements in
them.  All the relevant files had Utah *copyright* statements in them,
but without the expanded *license* statements, the licensing status of
the source files was unclear.

This commit is intended to clear up that confusion.

Most Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the Affero GNU General Public License, version 3
(AGPLv3).

Most Utah-copyrighted files related to ProtoGENI are distributed under
the terms of the GENI Public License, which is a BSD-like open-source
license.

Some Utah-copyrighted files in the Emulab source tree are distributed
under the terms of the GNU Lesser General Public License, version 2.1
(LGPL).

Mostly added a taget for the newnode MFS.

Windows 7 under Cygwin 1.7.  Mostly tweaks to makefiles to recognize Win7 in
addition to Windows XP.

New directory for Win7-specific files: clientside/tmcc/cygwinseven

Windows 7 under Cygwin 1.7.  Mostly tweaks to makefiles to recognize Win7 in
addition to Windows XP.

New directory for Win7-specific files: clientside/tmcc/cygwinseven

Also, nit: no need for WITH_EXT4FS since that is part of the default EXT
support.

the os subdir into the clientside/os dir.

Mike, use the --follow argument to git log to traverse across the mv.

avoid confusion and bad things happening when someome does the wrong
kind of install. Its happened.