GitLab

hardwired the order of LOADINFO args.

rc.frisbee has since been fixed.

node type as ASF support in its broadcom (bge) control net interface.

Also clean up some overzealous quoting in rc.frisbee (lets hope I have not
been under-zealous!)

widearea reloading support.

* New slot in the images table to store an access key which remote
  sites must provide in order to download an image (via https).

* tmcd returns a different kind of ADDRESS field from doloadinfo.
  Instead of the multicast stuff, return a URL that points to boss'
  web server. The URL is of the form:

   https://www.myemulab.net/spewimage.php?imageid=10013&access_key=abcdef

  which as you can see is fully specified; the client does not need
  to know anything else.

* New webpage and backend scripts appropriately called "spewimage"
  which also includes support for the http HEAD request (from wget) to
  avoid downloading images that are already on the node. I just
  learned about this HEAD request stuff today ... but otherwise these
  operate as expected, spewing the image if the access key is provided.

* Changes to rc.frisbee to deal with remote loading. In addition to
  URL support, I also added support for simple paths, the intent being
  that we will probably distribute images offline (say, at night) so
  that when a node reboots it doesn't actually have to wait 60 minutes
  for an image to download. I have not added any server side support
  for this yet though. Maybe later this week.

* Other bits and pieces and fixes to make this work.

* Add a bootwhat command that can be used instead of the bootinfo
  protocol, which is not appropriate for widearea cause its UDP. We
  lose the ability to have nodes "pxewait", but that is not actually
  necessary for widearea nodes since they are always allocated and up.

  Rather then duplicate code, I reorganized the bootinfo code so that
  I can link in the guts of it. There is some hackery to deal with the
  events that bootinfo sends, but so be it.

* Initial support for setting up accounts for the CMU widearea nodes,
  adding yet another way to specify accounts for a node. In this case,
  its a new node_type_attributes slot called "project_accounts" that
  is a comma separated list of group idx's. This is just a temporary
  measure to make sure that accounts are always set up on the nodes,
  the way they are for RON nodes. This will all need to change since
  we want to also create accounts on the fly for people creating
  experiments using the virtual nodes on each widearea node, but I
  have not figured that part out yet.

jailed vnodes if it exists.  Used to size their virtual disks.

of an image and return that in the loadinfo.

relevant email:

Date: Wed, 18 Apr 2007 21:20:37 -0700
From: Jason Shupe <jshupe@ISI.EDU>
To: Testbed Ops <testbed-ops@emulab.net>
Subject: [patches] tmcd.c (Jason) and elabinelab.in (Keith)

Included in this email are my description of the problem, and my patches
to tmcd.c, followed by more descriptions of the problem and Keith's
patch to elabinelab.in.  I apologize in advance for misquoting,
changing and other wise abusing Keith's prose.

An elab in elab experiment was started from the DeterTest project.  A
simple inner experiment was then started from the emulab-ops project.
During experiment swap in the program agent would fail to start.  If the
same simple inner experiment was started under the DeterTest project or
on the main testbed it would start normally.

It turned out that Keith's account (among others) wasn't getting created
on the inner experimental node.

tmcd was only sending a subset of accounts to the experimental nodes.
By digging through the database queries from tmcd.c I noticed one of the
database responses contained a NULL in the g.unix_gid field.  By
removing the only user from the emulab-ops sub group 'ops-test' it was
then possible to successfully swap in the inner experiment.

I've included two different versions of an untested tmcd.c patch.  Both
versions include changes only to the mysql statement.  Both versions of
the modified mysql statements were tested on the elab in elab database
after the only member of emulab-ops was re-added to the 'ops-test'
group.  Both queries returned all results of the original statements
except the offending record with the 'NULL' value for g.unix_gid.

The first patch directly excludes the offending record(s), and the
second patch simple changes the _left join_'s to just _join_'s (Keith's
suggestion) which also produces the same result for the data set tested.

Ted reminded me that "is not NULL" is better than my initial "!='NULL'",
which also produces the same results.  Other suggestions on this end
include specifically using "inner join", and to use both "inner join"
and "is not NULL".

Date: Tue, 24 Apr 2007 14:44:08 -0700
From: Leigh Stoller <stoller@flux.utah.edu>
Subject: Re: [patches] tmcd.c (Jason) and elabinelab.in (Keith)

Well, unix_gid is not supposed to be null, so we should fix that problem
instead, I would think.

Date: Wed, 25 Apr 2007 00:35:30 -0700 (PDT)
From: Keith Sklower <sklower@vangogh.CS.Berkeley.EDU>
Subject: Re: [Deter-ops] [patches] tmcd.c (Jason) and elabinelab.in (Keith)

It became null because of using an outer join instead of an inner join.

I'll repeat the condition:
1.) the DETER emulab-ops has subgroups
2.) the inner elab group membership table and references to a group
    which was not inherited from the outer boss [pid=emulab-ops,
    gid=test-grup, uid=jhickey]

So, my initial proposal was to be a bit tidier in specifying what
group membership entries should be subsetted.

(the was a phrase which intended to catch the group membership
for anybody currently active in emulab-ops, but it was too encompassing).

Date: Wed, 25 Apr 2007 13:15:51 -0600
From: Robert P Ricci <ricci@cs.utah.edu>
Subject: Re: [Deter-ops] [patches] tmcd.c (Jason) and elabinelab.in (Keith)

I guess, then, I will commit both proposed changes to tmcd - both to
make the existing join more 'correct', and to guard against other ways
(ie. bad/inconsistent DB state) the gid might show up as null.

Previously it was not being filled in and would wind up '0'.  Since it was
part of the primary key along with the service, this basically meant that
only one experiment at a time could successfully use the sync-server; i.e.,
only one could have (0, 'emulab-syncd') as their primary key.

Also, attempt reconnect once if DB access fails:
>               /*
>                * Try once to reconnect.  In theory, the caller (client)
>                * will retry the tmcc call and we will reconnect and
>                * everything will be fine.  The problem is that the
>                * client may get a different tmcd process each time,
>                * and every one of those will fail once before
>                * reconnecting.  Hence, the client could wind up failing
>                * even if it retried.
>                */

interfaces.  Should be backwards compat with old PLAB-PLC client scripts.

I commented out the client side stuff in the event directory though since
I initially trying to get boss running on the new system.

yesterday Dave said he needs it now.

Currently the accounts that are returned to ron nodes are determined
by looking for the node type in the pcremote_ok slot of the projects
table. There is no per-user subgroup setting; everyone gets the group
of their project.

Dave needs to be able to assign remote users to specific subgroups
within a variety of projects, and I did not want to hack up the
pcremote_ok mechanism any further. So, new way to do this, that might
replace pcremote_ok at some point, but for now will just override it.

I am using the node_attributes table to drive what groups get
installed, and thus what users get accounts and a grouplist.

	inset into node_attributes values
	           ('ronXXX', "dp_projects", "10000,10001,...");

where the group list is specified as a list of gid_idx's. This works
out very nicely cause I can use a subquery and FIND_IN_SET clause, and
so the changes to tmcd where actually pretty easy.

file: we return info for boss/ops in the "hostnames" command, but only if
the command would have otherwise returned something.

This is because the default hosts file we distribute in the rootball already
has boss/ops in it.  But if, during bootup, tmcd returns hostname info, that
hosts file gets overwritten and boss/ops info gets lost.

I could just grep out the info from the original hosts file and transfer it
to the one we are building, but what do I grep for: "boss", "ops",
"emulab.net"?  Don't really want to hardwire any of those in the client-side
script.

By returning this info via tmcd, I also don't have to modify the client-side
and thus don't need to build a new rootball!

methods.

Change a bunch of queries that join on the users table to use the idx
instead of the uid. This will need to be completed before we can really
archive users away, but close at this point.

node_type_attributes can now include a row with attrkey=use_acpi which can
either be set to:

  '1', if the FreeBSD image absolutely must use ACPI on this node_type
       in order to boot (e.g., our Dell SC440 machine)

  '0', if the FreeBSD image absolutely must NOT use ACPI on the node_type
       in order to boot (e.g., our pc2000s, possibly due to the IXP cards).

Otherwise, just don't set this attribute.

This value is queried by rc.frisbee and passed to slicefix after the disk
is loaded so that it can change /boot/loader.conf to enable/disable ACPI.

Note that ACPI (or lack thereof) has never been an issue with Linux or
Windows, so this is only meaningful to FreeBSD >= 5.0.

Note also that this does nothing for MFSes.  On machines that must use ACPI,
you need to use a special MFS for that node type (set via 'adminmfs_osid'
and 'diskloadmfs_osid' node_type_attributes).

ElabInElab experiments, so as not to consume an interface.

I actually modified Keith's changes so that we can dynmaically choose
the single or dual model in the NS file (Keith's changes hardwired the
system to only single control network) since the single network model
has the disadvantage of not allowing new nodes to be added to an inner
elab, which would make it impossible to test some things (like the
newnode path!).

To choose the single control network model, place this in your NS file:

	tb-elabinelab-singlenet

other it defaults to the older dual network model.

is a member is 'ipv4' then we report the control net IP rather than the
(non-existent in this case) experimental net IP.

ipv4 protocol is only used in PELAB to group planetlab nodes into a LAN so
that link tracing can be performed.  Now these nodes in this LAN will be able
to name each other as "planet-1", etc.

Two-day boondoggle to support "/scratch", an optional large, shared filesystem
for users.  To do this, I needed to find all the instances where /proj is used
and behave accordingly.  The boondoggle part was the decision to gather up all
the hardwired instances of shared directory names ("/proj", "/users", etc.)
so that they are set in a common place (via unexposed configure variables).
This is a boondoggle because:

1. I didn't change the client-side scripts.  They need a different mechanism
   (e.g., tmcd) to get the info, configure is the wrong way.

2. Even if I had done #1 it is likely--no, certain--that something would
   fail if you tried to rename "/proj" to be "/mike".  These names are just
   too ingrained.

3. We may not even use "/scratch" as it turns out.

Note, I also didn't fix any of the .html documentation.  Anyway, it is done.
To maintain my illusion in the future you should:

1. Have perl scripts include "use libtestbed" and use the defined PROJRO...

rejecting UDP requests as NON-SSL, due to removing some code that used to
skip around the TCP checks.

that clients and servers can avoid using hardwired ports on those
experimental nodes. I have added the following tmcd operation:

	tmcc portregister <service> [<port>]

where we assume its the control network IP (from the DB), and the pid/eid
of the node the experiment belongs to. The given port is entered into
the port_registration table for the experiment, using the service as the
tag. Supplying port=0 clears the registration from the table.

When called like:

	tmcc portregister <service>

we return the registered port, or nothing.

I hacked up a little C library module in libtb so that there is something
that looks like a C interface to this:

 	int
 	PortRegister(char *service, int port);

 	int
 	PortLookup(char *service, char *hostname, int namelen, int *port);

The above routines call out to tmcc of course.

Lastly, I changed the sync server and client to use the new port
registration, via the library calls above.

There are other emulab services that need to be changed as well, but
they can be done on an as needed basis.

Send along the IP address of the plab node in the return string from
the 'plabconfig' command.  We can't trust that the node will have
a resolvable hostname (or have working DNS even..), so slap down the IP
we have on record in the DB into a file.  This will be used by the
event proxy, which needs to know the node's routable IP in order to
subscripe to elvind on ops properly.

table, into a new table called node_type_attributes, which is intended
to be a more extensible way of describing nodes.

The only things left in the node_types table will be type,class and the
various isXXX boolean flags, since we use those in numerous joins all over
the system (ie: when discriminating amongst nodes).

For the most part, all of that other stuff is rarely used, or used in
contexts where the information is needed, but not for type descrimination.
Still, it made for a lot of queries to change!

Along the way I added a NodeType library module that represents the type
info as a perl object. I also beefed up the existing Node module, and
started using it in more places. I also added an Interfaces module, but I
have not done much with that yet.

I have not yet removed all the slots from the node_types table; I plan to
run the new code for a few days and then remove the slots.

Example using the new NodeType object:

	use NodeType;

	my $typeinfo = NodeType->Lookup($type);

        if ($typeinfo->control_interface(\$control_iface) ||
            !$control_iface) {
  	    warn "No control interface for $type is defined in the DB!\n";
        }

or using the Node:

	use Node;

        my $nodeobject = Node->Lookup($node_id);
        my $imageable  = $nodeobject->NodeTypeInfo()->imageable();
or
        my $rebootable = $nodeobject->isrebootable();
or
        $nodeobject->NodeTypeAttribute("control_interface", \$control_iface);

Lots of way to accomplish the same thing, but the main point is that the
Node is able to override the NodeType (if it wants to), which I think is
necessary for flexibly describing one/two of a kind things like switches, etc.

Get rid of a gratuitous IFCONFIG complaint.

ready for prime time yet.

upgrade.

Remove "delated" sql statement modifiers from all queries for now as delayed
insertion/replacement is apparently broken in our version of mysqld (5.0.20).
The problems are supposedly fixed in v 5.0.23, so we may be able to revert this
change.

Actually, most of the changes here were just to generalize the "virtual
interface" state in the DB.  Other than the client-side scripts, there
is very little specific here specific to tagged VLANs.

In fact, you cannot specify "vlan" as a type yet as we haven't done the
snmpit support for setting up the switches.

For more info see bas:~mike/flux/doc/testbed-virtinterfaces.txt, which I
will integrate into the knowledge base and the Emulab doc at some point.

Should help prevent our TCP tmcd from "wedging".

Fixup the plabconfig command in tmcd to return immediately if the calling node
is not a plab vserver (sliver).