GitLab

This checkin adds the necessary NS and client-side changes.

You get such a firewall by creating a firewall object and doing:

	$fw set-type ipfw2-vlan

In addition to the usual firewall setup, it sets the firewall node command
line to boot "/kernel.fw" which is an IPFW2-enabled kernel with a custom
bridge hack.

The client-side setup for firewalled nodes is easy: do nothing.

The client-side setup for the firewall is more involved, using vlan devices
and bridging and all sorts of geeky magic.

Note finally that I don't yet have a decent set of default rules for anything
other than a completely open firewall.  The rules might be slightly different
than for the "software" firewall since they are applied at layer2 (and we want
them just to be applied at layer2 and not multiple times)

Add elab_in_elab boolean to experiments table. Add inner_elab_role
to virt_nodes table, which is one of boss,ops,node.

       nextosid mechinism of 1.114 making it possible to map a generic *-STD
       OSID based on the time in which an experiment is created.  This
       provides backward compatibility for old experiments when the standard
       images are changed.

       The osid_map table lookup is triggered when the value of the nextosid
       field is set to 'MAP:osid_map'.  The nextosid also continues to behave
       as before: if it contains a valid osid, that OSID value is used to map
       independent of the experiment creation time.  The two styles can also
       be mixed, for example FBSD-JAIL has a nextosid of FBSD-STD which in
       turn is looked up and redirects to the osid_map and selects one of
       FBSD47-STD or FBSD410-STD depending on the time.

	CREATE TABLE osid_map (
	  osid varchar(35) NOT NULL default '',
	  btime datetime NOT NULL default '1000-01-01 00:00:00',
	  etime datetime NOT NULL default '9999-12-31 23:59:59',
	  nextosid varchar(35) default NULL,
	  PRIMARY KEY  (osid,btime,etime)
	) TYPE=MyISAM;

       Yeah, yeah, I'm using another magic date as a sentinel value.
       Tell ya what, in 7995 years, find out where I'm buried, dig me up,
       and kick my ass for being so short-sighted...

       The following commands are not strictly needed, they just give
       an example, default population of the table.  They cause the standard
       images to be revectored through the table and then remapped, based on
       two time ranges, to the exact same image.  Obviously, the second set
       would normally be mapped to a different set of images (say RHL90 and
       FBSD410):

	INSERT INTO osid_map (osid,etime,nextosid) VALUES \
	  ('RHL-STD','2004-09-08 08:59:59','emulab-ops-RHL73-STD');
	INSERT INTO osid_map (osid,etime,nextosid) VALUES \
	  ('FBSD-STD','2004-09-08 08:59:59','emulab-ops-FBSD47-STD');

	INSERT INTO osid_map (osid,btime,nextosid) VALUES \
	  ('RHL-STD','2004-09-08 09:00:00','emulab-ops-RHL73-STD');
	INSERT INTO osid_map (osid,btime,nextosid) VALUES \
	  ('FBSD-STD','2004-09-08 09:00:00','emulab-ops-FBSD47-STD');

	UPDATE os_info SET nextosid='MAP:osid_map' \
	  WHERE osname IN ('RHL-STD','FBSD-STD');

* SSL based server (sslxmlrpc_server.py) that wraps the existing Python
  classes (what we export via the existing ssh XMLRPC server). I also have a
  demo client that is analogous the ssh demo client (sslxmlrpc_client.py).
  This client looks for an ssl cert in the user's .ssl directory, or you can
  specify one on the command line. The demo client is installed on ops, and
  is in the downloads directory with the rest of the xmlrpc stuff we export
  to users. The server runs as root, forking a child for each connection and
  logs connections to /usr/testbed/log/sslxmlrpc.log via syslog.

* New script (mkusercert) generates SSL certs for users. Two modes of
  operation; when called from the account creation path, generates a
  unencrypted private key and certificate for use on Emulab nodes (this is
  analagous to the unencrypted SSH key we generate for users). The other mode
  of operation is used to generate an encrypted private key so that the user
  can drag a certificate to their home/desktop machine.

* New webpage (gensslcert.php3) linked in from the My Emulab page that
  allows users to create a certificate. The user is prompted for a pass
  phrase to encrypt the private key, as well as the user's current Emulab
  login password. mkusercert is called to generate the certificate, and the
  result is stored in the user's ~/.ssl directory, and spit back to the user
  as a text file that can be downloaded and placed in the users homedir on
  their local machine.

* The server needs to associate a certificate with a user so that it can
  flip to that user in the child after it forks. To do that, I have stored
  the uid of the user in the certificate. When a connection comes in, I grab
  the uid out of the certificate and check it against the DB. If there is a
  match (see below) the child does the usual setgid,setgroups,setuid to the
  user, instantiates the Emulab server class, and dispatches the method. At
  the moment, only one request per connection is dispatched. I'm not sure
  how to do a persistant connection on the SSL path, but probably not a big
  deal right now.

* New DB table user_sslcerts that stores the PEM formatted certificates and
  private keys, as well as the serial number of the certificate, for each
  user. I also mark if the private key is encrypted or not, although not
  making any use of this data. At the moment, each user is allowed to get
  one unencrypted cert/key pair and one encrypted cert/key pair. No real
  reason except that I do not want to spend too much time on this until we
  see how/if it gets used. Anyway, the serial number is used as a crude form
  of certificate revocation. When the connection is made, I suck the serial
  number and uid out of the certificate, and look for a match in the table.
  If cert serial number does not match, the connection is rejected. In other
  words, revoking a certificate just means removing its entry from the DB
  for that user. I could also compare the certificate itself, but I am not
  sure what purpose that would serve since that is what the SSL handshake is
  supposed to take of, right?

* Updated the documentation for the XMLRPC server to mention the existence
  of the SSL server and client, with a pointer into the downloads directory
  where users can pick up the client.

       and default_firewall_rules.

1.269: Add new table to generate a per virt_lan index for use with
       veth vlan tags. This would be so much easier if the virt_lans
       table had been split into virt_lans and virt_lan_members.
       Anyway, this table might someday become the per-lan table, with a
       table of member settings. This would reduce the incredible amount of
       duplicate info in virt_lans!

	CREATE TABLE virt_lan_lans (
	  pid varchar(12) NOT NULL default '',
	  eid varchar(32) NOT NULL default '',
	  idx int(11) NOT NULL auto_increment,
	  vname varchar(32) NOT NULL default '',
	  PRIMARY KEY  (pid,eid,idx),
	  UNIQUE KEY vname (pid,eid,vname)
	) TYPE=MyISAM;

       This arrangement will provide a unique index per virt_lan, within
       each pid,eid. That is, it starts from 1 for each pid,eid. That is
       necessary since the limit is 16 bits, so a global index would
       quickly overflow. The above table is populated with:

	insert into virt_lan_lans (pid, eid, vname)
            select distinct pid,eid,vname from virt_lans;

	set g1 [new EventGroup $ns]
	$g1 add  $link0 $link1
	$ns at 60.0 "$g1 down"

See the new advanced tutorial section on event groups for a better
example.

Changed tbreport to dump the event groups table when in summary mode.
At the same time, I changed tbreport to use the recently added
virt_lans:vnode and ip slots, decprecating virt_nodes:ips in one more
place. I also changed the web interface to always dump the event and
event group summaries.

The parser gets a new file (event.tcl), and the "at" method deals with
event group events by expanding them inline into individual events
sent to each member. For some agents, this is unavoidable; traffic
generators get the initial params in the event, so it is not possible
to send a single event to all members of the group. Same goes for
program objects, although program objects do default to the initial
command now, at least on new images.

Changed the event scheduler to load the event groups table. The
current operation is that the scheduler expands events sent to a
group, into a set of distinct events sent to each member of the
group. At some point we proably want to optimize this by telling the
agents (running on the nodes) what groups they are members of.

Other News: Added a "mustdelay" slot to the virt_lans table so the
parser can tell assign_wrapper that a link needs to be delayed, say if
there are events or if the link is red/gred. Previously,
assign_wrapper tried to figure this out by looking at the event list,
etc. I have removed that code; see database-migrate for instructions
on how to initialize this slot in existing experiments. assign_wrapper
is free to ignore or insert delays anyway, but having the parser do
this makes more sense.

I also made some "rename" changes to the parser wrt queues and lans
and links. Not really necessary, but I got sidetracked (for several
hours!) trying to understand that rename stuff a little better, and
now I do.

this field but forgot to removed it from the create file (it never
went into the migrate file).

node_types:isphantom which I added in 1.256 since I decided not to use
it; just use isdynamic instead. Add an isjailed slot to node_types
rather then hardwiring that fact to "pcvm" node type.

a legal value; when interfaces are not in use, set the speed to 0.

so that user can select them (like jail/delay osname)

of the database).

when doing modelnet. Sure wish we had a better way to do this!

happy.

       that an experiment prefers to use modelnet for traffic shaping.
       At some future time maybe we can mix modelnet with delay nodes,
       but please do not hold your breath waiting.

	alter table experiments add usemodelnet tinyint(1) NOT NULL \
		default '0' after useprepass;

       Add modelnet_cores and modelnet_edges variables to hold user
       tunable number of physical core nodes and edge nodes to throw
       at a virtual topology.

	alter table experiments add modelnet_cores tinyint(4) unsigned \
		NOT NULL default '0' after usemodelnet;
        alter table experiments add modelnet_edges tinyint(4) unsigned \
		NOT NULL default '0' after modelnet_cores;

       Add ismodelnet and isphantom slots to node_types table. ismodelnet
       is set for modelnet nodes (which are also going to be isvirt).  Add
       isphantom to mark nodes that do no get underlying physical nodes;
       there will be an entry in the reserved table but no entries in the
       nodes table. Initially, modelnet virtual nodes will all be phantoms.
       Will possibly change later.

	alter table node_types add ismodelnet tinyint(1) NOT NULL \
		default '0' after isvirtnode;
	alter table node_types add isphantom tinyint(1) NOT NULL \
		default '0' after ismodelnet;

       Not sure that isphantom is going to work cause of the number of
       queries that join reserved with the nodes table. I might end up
       making nodes dynamic instead, creating and detroying them on the
       fly. I would prefer that isphantom works, but I am willing to live
       with isdynamic since we can use that for all of our virtual nodes.

	alter table node_types add isdynamic tinyint(1) NOT NULL \
		default '0' after isphantom;

off linktest async via the web interface.

even if the disk is up to date.

table - this will allow us to place them on the map while they are still
in limbo.

 ALTER TABLE new_nodes ADD COLUMN (floor varchar(32) default NULL,
    building varchar(32) default NULL,
    loc_x int(10) unsigned NOT NULL default '0',
    loc_y int(10) unsigned NOT NULL default '0',
    contact tinytext,
    phone tinytext,
    room varchar(32) default NULL);

number in database-migrate.

This is rather primitive still; We just need a place to define
buildings and floors in buildings, so that we do not hardwire them
into the code. This can get arbitrarily complicated, but not until we
need it.

	CREATE TABLE buildings (
	  building varchar(32) NOT NULL default '',
	  image_path tinytext,
	  title tinytext NOT NULL,
	  PRIMARY KEY  (building)
	) TYPE=MyISAM;

The image_path is optional for buildings. The title is a string to
print along with the images (Merril Engineering Building).

	CREATE TABLE floorimages (
	  building varchar(32) NOT NULL default '',
	  floor varchar(32) NOT NULL default '',
	  image_path tinytext,
	  thumb_path tinytext,
	  x1 int(6) NOT NULL default '0',
	  y1 int(6) NOT NULL default '0',
	  x2 int(6) NOT NULL default '0',
	  y2 int(6) NOT NULL default '0',
	  PRIMARY KEY  (building, floor)
	) TYPE=MyISAM;

The image_path is not optional; it is either an absolute path or a
filename in $TB/www. The thumb_path is for a tiny view of it. Floor is
something like 1, 2, 3 but could also be basement, lobby, penthouse,
etc. The x,y coordinates are intended to be bounding box coords of the
"interesting" part of the image so that it is easier to scale specific
entries from the location_info table. But, not really sure about this
yet; needs more thought and some investigation about appropriate ways
to store coordinate values like this.

virt_lans and virt_nodes. The intent is to migrate away from
the convention we use in virt_nodes:ips and virt_lans:member
to a more acceptable representation (one that does not rely
on textual conventions like space separated lists of colon
seperated entities. Instead:

	virt_nodes
		vname:	nodeA
        virt_lans
		vname:  link0
		vport:  0
		ip:     1.1.1.2

		vname:  link1
		vport:  1
		ip:     1.1.2.2

	alter table virt_lans add vnode varchar(32) NOT NULL default '' \
		after vname;
	alter table virt_lans add vport tinyint(3) NOT NULL default '0' \
		after vnode;
        alter table virt_lans add ip varchar(15) NOT NULL default '' \
		after vport;

Then run this script to update these new fields from the
existing ips,member slots. This must be run after installing
the parser changes, or you can just run it again.

	./ipmember.pl

This is a transitional phase; the old slots will be left in place
until they are no longer used, at which time we will also add a
unique key to the table (pid,eid,vname,vnode,vport). assign_wrapper
will be the hardest to change, but other scripts should be easy.

Whats vport about? Rather then rely on IP addresses to form a
unique key, we use vport (a small integer) so that we can delay the
IP assignment until later (after initial DB insertion).

passing this to the newnode client, web page, and viewing and
editing it.

on and off the assign prepass on a per-experiment basis.

No parser interface yet - for use only with manual tweaking for now.

'loadinfo' request.

estimated bandwidth that will be used on the link/LAN. This can then
be passed to assign, to do less conservative bandwidth allocation.

see (or remember) any reason for this slot to be 20 chars, when the
name of every other vname slot is 32. I looked in the event scheduler
and there do not appear to be any problems there with bumping it. Note
that I choose 64 cause we tend to construct agent names that might be
longer then 32 since they are based on real vnames (lan0, node0, etc).