GitLab

use/test the event system while logins are turned off.

* The per-experiment event scheduler now runs on ops instead of boss.
  Boss still runs elvind and uses events internally, but the user part
  of the event system has moved.

* Part of the guts of eventsys_control moved to new script, eventsys.proxy,
  which runs on ops and fires off the event scheduler. The only tricky part
  of this is that the scheduler runs as the user, but killing it has to be
  done as root since a different person might swap out the experiment. So,
  the proxy is a perl wrapper invoked from a root ssh from boss, which
  forks, writes the pid file into /var/run/emulab/evsched/$pid_$eid.pid,
  then flips to the user and execs the event scheduler (which is careful
  not to fork). Obviously, if the kill is done as root, the pid file has to
  be stored someplace the user is not allowed to write.

* The event scheduler has been rewritten to use Tim's C++ interface to the
  sshxmlrpc server on boss. Actually, I reorg'ed the scheduler so that it
  can be built either as a mysql client, or as RPC client. Note that it can
  also be built to use the SSL version of the XMLRPC server, but that will
  not go live until I finish the server stuff up. Also some goo for dealing
  with building the scheduler with C++.

* Changes to several makefiles to install the ops binaries over NFS to
  /usr/testbed/opsdir. Makes life easier, but only if boss and ops are
  running the same OS. For now, using static linking on the event scheduler
  until ops upgraded to same rev as boss.

* All of the event clients got little tweaks for dealing with the new CNAME
  for the event system server (event-sever). Will need to build new images
  at some point. Old images and clients will continue to work cause of an
  inetd hack on boss that uses netcat to transparently redirect elvind
  connections to ops.

* Note that eventdebug needs some explaining. In order to make the inetd
  redirect work, elvind cannot be listening on the standard port. So, the
  boss event system uses an alternate port since there are just a few
  subsystems on boss that use the server, and its easy to propogate changes
  on boss. Anyway, the default for eventdebug is to connect to the standard
  port on localhost, which means it will work as expected on ops, but will
  require -b argument on boss.

* Linktest changes were slightly more involved. No longer run linktest on
  boss when called from the experiment swapin path, but ssh over to ops to
  fire it off. This is done as the user of course, and there are some
  tricks to make it possible to kill a running linktest and its ssh when
  experiment swapin is canceled (or from the command line) by forcing
  allocation of a tty. I will probably revisit this at some point, but I
  did not want to spend a bunch of time on linktest.

* The upgrade path detailed in doc/UPDATING is necessarily complicated and
  bound to cause consternation at remote sites doing an upgrade.

Add a minor print state matent to eventsys_start.

Add elvind-inetd.conf for local inetd startup that redirects event
traffic from old images over to event server on ops. Note that this
will not be needed on new testbeds. Also note that this makefile install
is not tied into the toplevel install; it is for the metaports install.

Add local inetd startup for redirecting event sys traffic to ops using
netcat. Hmm, this is not something we need for new testbeds; will need
to revisit this later.

passwords can be wiped from the adit email messages.

path to the experiments logs directory (exp/$eid/logs/linktest.log).

This starts with a new option to newnode, -o, that tells it it's
running on ops. This reports some slightly different information
to the checkin page.

The checkin page and the backend newnode script then take this extra
information into account, and deal with ops nodes slightly
differently.

like this:

* Listen for connections on port 3069. The server requires client
  authentication, and will fail if a certificate is not provided by
  the client.

* Once the certificate is accepted, the server forks a new child.

* The child looks inside the certificate to get the CN field of the
  Distinguished Name (subject). The CN field must hold the uid of the
  user, which is checked against the DB for a matching user. We get
  the groupslist from the DB, and do a setgid,setgroups,setuid to flip
  to the user in the child.

* A instance of the emulabserver class is created, and the request is
  dispatched.

I added an sslxmlrpc_client.py script that mirrors the ssh version of
the client script. I could probably roll these into one, but decided
not to to avoid confusing people who might download it.

classes are now used from SocketServer.ForkingMixIn, and its easier to
move the little global state we have into the classes, rather then try
to refresh at each fork.

Overview of simply firewall setup.

Experimentor specifies in their ns file:

     set fw [new Firewall $ns]
     $fw style <open|closed|basic>

to set up an "open" ("allow any"), "closed" ("deny any"), or "basic"
(allow ICMP and ssh) firewall.  "basic is the default.  Additional rules
can be added with:

     $fw add-rule <IPFW format rule>
     $fw add-numbered-rule <1-50000> <IPFW format rule>

where the former implicitly numbers rules such that the firewall processes
them in the order given in the NS file.  The latter allows explicit
specification of the numbering.  Currently the rules are fixed strings,
there is no variable substitution.  There is also no syntax checking done
on the rules at parse time.

We allocate an extra node to the experiment to serve as a firewall.
Currently that node runs FreeBSD and uses IPFW.  In the initial configuration,
all other nodes in the experiment will just be setup with a default route
that points to the firewall node.  So all outbound traffic will pass through
it.  Inbound traffic will still travel straight to the node.  This should
prevent nodes from accidentally initiating attacks on the outside world.
Long term we will of course enforce the firewall on all traffic, that should
not have any effect on the NS syntax above.

When a node boots, there will be an rc.firewall script that checks to see
if there is a firewall for the experiment and if so, which node it is.
This is done with the TMCD "firewallinfo" command which returns:

      TYPE=none

      TYPE=remote FWIP=N.N.N.N

      TYPE=<fwtype> STYLE=<fwstyle> IN_IF=<macaddr> OUT_IF=<macaddr>
      RULENO=<num> RULE="<ipfw command string>"
      RULENO=...
      ...

In the case of no firewall we get back TYPE=none, and we continue as normal.
Otherwise, there are two types of replies, one for a node that is being
firewalled (TYPE=remote) and one for a node that is a firewall
(TYPE=<fwtype> + RULES).

In the TYPE=remote case, the firewall node indicated by FWIP.  This is
the address we use for the default route.

For TYPE=<fwtype>, we are the firewall, and we get STYLE and IN_IF/OUT_IF
info.  Here TYPE indicates whether we should use ipfw or whatever.
For now it is always ipfw.  IN_IF and OUT_IF may someday indicate the
interfaces to use for the internal and external connections, right now
both will indicate the control net interface.  So, after ensuring that
the ipfw modules is loaded, we grab the provided RULE info, which includes
both per-experiment and default rules, and setup ipfw.

Issues to resolve:
       - synchronization: how to ensure firewall comes up first
       - how to better implement the firewalling
         (i.e., without the cooperation of the nodes)
       - support the equiv of linkdelays (on-node firewalling)?
       - allow firewalls within experiments?
         (ie., on experimental interfaces)
       - dynamic changing of firewall rules via events?
       - how to show firewall state in various web pages

       and default_firewall_rules.

list of lists instead of a big long string to be parsed (which I did
cause I had forgotten about Tim's C++ RPC library).

the server.

A symlink command was inverted.

the sync-server is running.  Basically, its a quick way to determine
which node is the "master" when you are using a single startup script
on all of the nodes.

based server. Now I need to actually fill in some details ...

(recently changed) FreeBSD version of the function.

Also added control-install target to indicate what needs to be done
on the control-node.

startup script installed.

version number.

Also add a phase to ops-install to install the elvind config file.

pass a string with embedded quotes, perl hands it off to a shell which
results in an extra shell wrapper for every ssh. Well, with SSH_ARGS
the way they are, that means we always get the extra shell. While this
is a nice optimization it actually matters mostly from
linktest_control where I want to make sure that the child that invokes
linktest on ops (an ssh) can be killed directly with TERM (say, if
user cancels linktest) without worrying about intervening shell
screwing with signals, etc.

people who are not logged in yet.

able to scp some stuff over once this is done.

Install the new elvind config file.

Fix a missing comma.

check.  There is now a per-command flag indicating whether the calling
node must be allocated to an experiment to use the command.

* Add EventRegiser() function to wrap up event_register. Takes a host
  and a port number, and constructs the URL for the caller. Not really
  a big change of course.

* Change call to event_register(); Now connect on localhost:BOSSEVENTPORT.

run through configure.