GitLab

Lahey.

Power saving turns off nodes that have been sitting in PXEWAIT (and
are thus free) for more then a set amount of time (see sitevar
general/idlepower_idletime, which defaults to 3600 seconds).

The driver script is tbsetup/idlepower.in and needs to be added to
/etc/crontab at sites that want to do this. Even so, operation is
enabled by the sitevar general/idlepower_enable. Each time it runs, it
checks for nodes that need to be turned off, and then calls power.
Note: This should be a daemon not a cron job.

To be considered for power saving, you must add an attribute to the
node_type_attributes table called 'idlepower_enable', set to 1.

Locally, I hacked up stated and power to make the state transitions
legal so that stated does not whine. I added POWEROFF as a valid
transition from any state, to opmodes NORMAL, NORMALv1, and NORMALv2.
Barry's original patch already had a state transition for PXEKERNEL.
In power, I added code to look at the actual operation, and in the
case of "on", do not send an event if the node is not in POWEROFF,
since a user can foolishly say power on anytime, and the node is on
nothing is every going to change, and the state transition would be
wrong.

node_reboot takes of powering nodes on, when they are in POWEROFF.

Barry on copyright issues:
 "I'm not sure those rights are mine to grant! Remember that this code
 came originally from Kevin Lahey (kml@patheticgeek.net) and
 originated at DETER (although he's apparently not there anymore). I
 don't foresee a problem from our point of view (but I'll double
 check, of course). Shall I try to contact Kevin try to sort this mess
 out, or do you think it's better to coordinate from your end?"

libraries from snmpit_test directory, See previous commit.

for LM.

Added support for power control and status on Linux NetworkX ICE Boxes.
Note that the 'reset' command is tied to the node's hardware reset line
through the special ICE Box connector on the back of the node rather.
than a simple power cycle, so this needs to be connected.

testing. Changes include:

* syncvlans option that compares the switches and the DB and makes
  everything consistent, including creating backing DB objects for the
  dozens of hand created vlans, since all vlans must have a DB object
  so that vlan tag reservations can be consistent across all stacks.

* Clean up the tag stuff a bit. No longer store tag in the
  lan_attributes table since that is a transient table, prone to
  getting out of sync with reality.

* Changes to how the set of devices in a stack is chosen; ensure that
  we do not reference switches that are not needed (no ports on that
  switch and does not traverse a trunk on that switch). This should
  allow us to move the Backbone switches back into the Experimental
  stack, without fear that normal swapin will block cause the long
  distance links to those switches is down.

The big backward compatibility issue is that we no longer store running
frisbeed info in the DB.  This means that loadinfo could not return
address:port info to clients and thus old frisbee MFSes could no longer
work.  While not a show stopper to require people to update their MFS first,
I made a token effort to implement backward compat as follows.

When an old frisbee MFS does "tmcc loadinfo" (as identified by a tmcd
version < 33), tmcd will invoke "frisbeehelper" to startup a daemon.
Sound like frisbeelauncher?  Well sorta, but vastly simplified and I only
want this to be temporary.  The helper just uses the frisbee client to make
a "proxy" request to the localhost master server.  The Emulab configuration
of the master server now allows requests from localhost to proxy for another
node.

frisbeehelper is also used by webfrisbeekiller to kill a running daemon
(yes, just like frisbeelauncher).  It makes a proxy status request on
localhost and uses the returned info to identify the particular instance
and kill it.

No more frisbeelauncher or assorted subboss frisbee stuff.

libosload_new contains the new libosload.  It provides the same
API as the original libosload, but the functionality has been
split out into well-defined methods that can be overridden by
per-node type subclasses in small chunks.  This is all protected
by the NewOsload feature.

libosload_switch.pm.in and libossetup_switch.pm.in contain the HP
switch support, which requires the new libosload and new
libossetup.  libossetup_switch doesn't do anything too radical, but
it does support running a node it is "lighting up" through multiple
operations -- i.e., first a RELOAD, then a RECONFIG.  This was
necessary because switches require a config to be pushed to them
(well, they do in the way I elected to support them), and they
cannot fetch configuration in the normal Emulab way via tmcd.

Eventually, that multi-op support will be factored out into
libossetup more generically so that other node types which require
push-based configuration can leverage it.

libosload_switch contains the HP switch loading code.  It's built
around expect-like interaction on the serial console.  It supports
xmodem and tftp uploads, but tries to only flash the switch lazily.
Thus, if the switch is already loaded with an image and a user
(or the reload daemon, or anybody) reloads it with the same image,
it won't actually get reloaded.  You can now force this reload
with os_load.

libosload_switch also contains reconfig code (i.e., to wipe
switch configs and recreate based on which experiment it is in).
libossetup_switch directly invokes this instead of going through
libosload, since configuring is not really part of loading.  It
is just convenient since configuration also happens over the
serial console, and all the osload code for the switch can be
reused.

Also, of course, there are miscellaneous changes to support the
new libosload in os_load itself.

starting any one of our scripts can take a second or two. That time is
spent including and compiling 10000s of thousands of lines of perl
code, both from our libraries and from the perl libraries.

Mostly this is just a maintenance thing; we just never thought about
it much and we have a lot more code these days.

So I have done two things.

1) I have used SelfLoader() on some of our biggest perl modules.
   SelfLoader delays compilation until code is used. This is not as
   good as AutoLoader() though, and so I did it with just a few 
   modules (the biggest ones).

2) Mostly I reorganized things:

  a) Split libdb into an EmulabConstants module and all the rest of
     the code, which is slowly getting phased out.

  b) Move little things around to avoid including libdb or Experiment
     (the biggest files).

  c) Change "use foo" in many places to a "require foo" in the
     function that actually uses that module. This was really a big
     win cause we have dozens of cases where we would include a
     module, but use it in only one place and typically not all.

Most things are now starting up in 1/3 the time. I am hoping this will
help to reduce the load spiking we see on boss, and also help with the
upcoming Geni tutorial (which kill boss last time).

tbsetup/GNUmakefile.in:add snmpit_apcon.pm to LIB_STUFF. snmpit_apcon.pm: add response to asking about whether adding remote keys into trustedhosts for the first time ssh connection.

Subbosses authenticate to the XML-RPC server as elabman, which means the
resulting server process runs as the elabman user.  Unfortunately, this
doesn't work well when the subboss wants to launch a frisbeed for an
image for which elabman doesn't have read permission (like images under
/proj).

To fix this, a setuid wrapper script is run instead of trying to run
frisbeelauncher directly.  This script makes sure the calling user is
elabman, and then becomes root and execs frisbee_launcher.

Note that command line arguments for ptopgen are tweaked slightly so that after '-g' for GENI, you must enter a version number. Changed invocation in GeniCM. Also, tweaked rspec definitions to make it conform to reality on the ground inside of our system as it gets implemented.

land in hwdown.

Currently, if a node fails to boot in os_setup and the node is running
a system image, it is moved into hwdown. 99% of the time this is
wasted work; the node did not fail for hardware reasons, but for some
other reason that is transient.

The new approach is to move the node into another holding experiment,
emulab-ops/hwcheckup. The daemon watches that experiment, and nodes
that land in it are freshly reloaded with the default image and
rebooted. If the node reboots okay after reload, it is released back
into the free pool. If it fails any part of the reload/reboot, it is
officially moved into hwdown.

Another possible use; if you have a suspect node, you go wiggle some
hardware, and instead of releasing it into the free pool, you move it
into hwcheckup, to see if it reloads/reboots. If not, it lands in
hwdown again. Then you break out the hammer.

Most of the changes in Node.pm, libdb.pm, and os_setup are
organizational changes to make the code cleaner.

experiment in regression mode, and then compare the results.

its really a hugely stripped down Emulab boss install, using a very
short version of install/boss-install to get a few things into place.

I refactored a few things in both the protogeni code and the Emulab
code, and whacked a bunch of makefiles and configure stuff. The result
is that we only need to install about 10-12 files from the Emulab
code, plus the protogeni code. Quite manageable, if you don't mind
that it requires FreeBSD 6.X ... Still, I think it satisfies the
requirement that we have a packaged clearinghouse that can be run
standalone from a running Emulab site.

Instead of passing along a set of arguments that needs to be turned
into command line arguments to the proxy, pass along an xmldoc
representing the arguments. This xmldoc is passed through the rpc
server to the snmpit proxy, where it reconstructs the arguments. This
avoids all that cruftiness in the rpc server (also error prone) and
make it easier to add new remote methods, say for supporting eventual
elabibelab firewalls. Note that there are currently two versions of
the proxy and two methods in the rpc server, so that existing emulabs
will work.

I also added support for port enable/disable/etc from the inner elab.
There is also the beginning of firewall support. I pass the stack
module argument along, but currently not actually doing control stack
operations from the proxy. Needs more work.

Oh, I copied the cvs repo file for the original proxy so that we do
not lose the cvs history.

See corresponding commitlog for snmpit for description of other
changes.

a locally modifed version of CVS that does a couple of extra things. The
hacks are quite simple and currently on users. Briefly, there is a new
config file option to specify a program to run when the tag operation is
complete (similar to a module hook for rtag), and in tag.c I run that hook.

Check out a sandbox as before:

  cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo rtag mytag XXXXX

To create a new template (along the trunk), use the tag operation instead:

	cd sandbox
	cvs tag mtag

and what for the email that says the template is created. You can
continue to work along the trunk in this manner, although note that
the .template file is changing (by the backend commit), and you might
find it less distriacting to do a cvs update each time.

To work along a branch:

	cd sandbox
	cvs tag -r existing_tag -b newtag1
        cvs update -r newtag
        make changes and commits
	cvs tag newtag2

When working along branched you will want to do a cvs update to get
the .template file in sync.

"rtag" directive to initiate template modify operations. So, to get started
you do a checkout:

  cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo checkout XXXXX

where XXXXX is the part of the guid (10000/1) before the slash. Might try
and roll all templates into a single project wide repo at some point, to
avoid the extraneous path stuff, but didn't want to worry that just yet.

Okay, so have a checkout. You can work along the trunk, doing commits. To
create a new template (a modify of the existing template), you tag the tree
using rtag:

  cvs -d ops.emulab.net:/proj/$pid/templates/XXXXX/cvsrepo rtag mytag XXXXX

A template modify is started at the end, and you should probably wait for
email before continuing. Eventually I will need to add locking of some
kind, but I have to do the modify in the background, or else I get deadlock
cause cvs keeps the repo locked, and the modify also needs to access it.

Each time you tag along the trunk, you get a modified template, which in
the history diagram looks like:

  10000/1 --> 10000/2 --> 10000/3 ...

If you want to branch, say at 10000/2 you can create a branch tag using rtag:

  cvs -d [cut] rtag -r T10000/2 -b mytag2 XXXXX

You can also use your own tags for -r option, but I also create a TXXXXX/YY
tag at each template modify, which is easy to remember.

Then update your sandbox to the new branch, commit changes along that
branch, and then later use rtag again to initiate a template modify
operation:

  cvs update -r mytag2
  cvs commit ...
  cvs -d [cut] rtag -r mytag2 mytag3 XXXXX

And now the history diagram looks like:

  10000/1 --> 10000/2 --> 10000/3 ...
                |
                |
                -> 10000/4 ...

You should be able to mix interaction via the web with interaction via the
cvs interface. I've tested it, although not extensively.

described is the one exported to ops via the XMLRPC interface. This is
just playing aroundl no doubt this stuff is going to change.

* template_checkout guid/vers

  Checkout a copy of the template to the current working directory.

* template_commit

  Modify the previous template checkout, using the nsfile contained in
  the tbdata directory (subdir of the current directory). In other words,
  the current template is modified, creating a new template in the
  current working directory (the current directory refers to the new
  template).

  The datastore subdir is imported into the new template, but that is
  the only directory that is imported at present. Might change that.

So this sounds much cooler then it really is. Why?

* This only works from ops.

* The "current directory" must be one of the standard approved directories
  (/proj, /users, /groups).

* Cause, boss reads and writes that directory via NFS, as told to it
  by the xmlrpc client.

At some point in the future it would be nice to support something
fancier, using a custom transport, but lets see how this goes.

* Records are now "help open" when a run is stopped. When the next run
  is started, a check is made to see if the files
  (/project/$pid/exp/$eid) have changed, and if so a new version of the
  archive is committed before the next run is started.

* Change the way swapmod is handled within an instance. A new option
  on the ShowExp page called Modify Resources. The intent is to allow
  an instance to be modified without having to start and stop runs,
  which tends to clutter things up, according to our user base. So, if
  you are within a run, that run is reset (reused) after the swapmod is
  finished. You can do this as many times as you like. If you are
  between runs (last operation was a stoprun), do the swapmod and then
  "speculatively" start a new run. Subsequent modifies reuse the that
  run again, as above.

  I think this is what Kevin was after ... there are some UI issues
  that may need to be resolved, will wait to hear what people have to
  say.

* Revising a record is now supported. Export, change in place, and
  then use the Revise link on the ShowRun page. Currently this has to
  happen from the export directory on ops, but eventually allow an
  upload (to correspond to downloaded exports)

* Check to see if export already exists, and give warning. Added a
  checkbox that allows user to overwrite the export.

* A bunch of minor UI changes to the various template pages.

have to re-run the swig-wrappers target in tools/rmanage/GNUmakefile to
generate the wrapper and perl module; this must of course be done when
changes are made to the rmcp libs.

  * GNUmakefile.in, configure, configure.in: add tools/rmanage
  * tbsetup/GNUmakefile.in, tbsetup/power*.in: add rmcp to power command
  * tools/GNUmakefile.in: add rmanage
  * tools/rmanage/*.c,*.h: bugfixes, swig helper methods, etc.
  * tools/rmanage/rmcp.i: swig import control file
  * tools/rmanage/rmcp.pm,rmcp_wrap.c: rmcp wrapper/module generated by swig

	node_attributes should only be installed in one place (sbin)
	node_update and node_control are in bin