1. 22 Sep, 2004 1 commit
  2. 13 Sep, 2004 1 commit
  3. 10 Sep, 2004 1 commit
  4. 07 Sep, 2004 1 commit
  5. 02 Sep, 2004 1 commit
  6. 01 Sep, 2004 5 commits
    • Leigh B. Stoller's avatar
      Fix minor typo. · 46f7b3f2
      Leigh B. Stoller authored
      46f7b3f2
    • Leigh B. Stoller's avatar
      e2f62c8e
    • Leigh B. Stoller's avatar
      Reorder syslog ststements slightly. · 37a19abb
      Leigh B. Stoller authored
      37a19abb
    • Leigh B. Stoller's avatar
      SSL version of the XMLRPC server. · a9c1045e
      Leigh B. Stoller authored
      * SSL based server (sslxmlrpc_server.py) that wraps the existing Python
        classes (what we export via the existing ssh XMLRPC server). I also have a
        demo client that is analogous the ssh demo client (sslxmlrpc_client.py).
        This client looks for an ssl cert in the user's .ssl directory, or you can
        specify one on the command line. The demo client is installed on ops, and
        is in the downloads directory with the rest of the xmlrpc stuff we export
        to users. The server runs as root, forking a child for each connection and
        logs connections to /usr/testbed/log/sslxmlrpc.log via syslog.
      
      * New script (mkusercert) generates SSL certs for users. Two modes of
        operation; when called from the account creation path, generates a
        unencrypted private key and certificate for use on Emulab nodes (this is
        analagous to the unencrypted SSH key we generate for users). The other mode
        of operation is used to generate an encrypted private key so that the user
        can drag a certificate to their home/desktop machine.
      
      * New webpage (gensslcert.php3) linked in from the My Emulab page that
        allows users to create a certificate. The user is prompted for a pass
        phrase to encrypt the private key, as well as the user's current Emulab
        login password. mkusercert is called to generate the certificate, and the
        result is stored in the user's ~/.ssl directory, and spit back to the user
        as a text file that can be downloaded and placed in the users homedir on
        their local machine.
      
      * The server needs to associate a certificate with a user so that it can
        flip to that user in the child after it forks. To do that, I have stored
        the uid of the user in the certificate. When a connection comes in, I grab
        the uid out of the certificate and check it against the DB. If there is a
        match (see below) the child does the usual setgid,setgroups,setuid to the
        user, instantiates the Emulab server class, and dispatches the method. At
        the moment, only one request per connection is dispatched. I'm not sure
        how to do a persistant connection on the SSL path, but probably not a big
        deal right now.
      
      * New DB table user_sslcerts that stores the PEM formatted certificates and
        private keys, as well as the serial number of the certificate, for each
        user. I also mark if the private key is encrypted or not, although not
        making any use of this data. At the moment, each user is allowed to get
        one unencrypted cert/key pair and one encrypted cert/key pair. No real
        reason except that I do not want to spend too much time on this until we
        see how/if it gets used. Anyway, the serial number is used as a crude form
        of certificate revocation. When the connection is made, I suck the serial
        number and uid out of the certificate, and look for a match in the table.
        If cert serial number does not match, the connection is rejected. In other
        words, revoking a certificate just means removing its entry from the DB
        for that user. I could also compare the certificate itself, but I am not
        sure what purpose that would serve since that is what the SSL handshake is
        supposed to take of, right?
      
      * Updated the documentation for the XMLRPC server to mention the existence
        of the SSL server and client, with a pointer into the downloads directory
        where users can pick up the client.
      a9c1045e
    • Leigh B. Stoller's avatar
      Turn off nologins check in state and statewait methods so that we can · 60c2a7ab
      Leigh B. Stoller authored
      use/test the event system while logins are turned off.
      60c2a7ab
  7. 27 Aug, 2004 1 commit
    • Leigh B. Stoller's avatar
      Guts of the new ssl server implemented. The server operates more or less · 5a025f36
      Leigh B. Stoller authored
      like this:
      
      * Listen for connections on port 3069. The server requires client
        authentication, and will fail if a certificate is not provided by
        the client.
      
      * Once the certificate is accepted, the server forks a new child.
      
      * The child looks inside the certificate to get the CN field of the
        Distinguished Name (subject). The CN field must hold the uid of the
        user, which is checked against the DB for a matching user. We get
        the groupslist from the DB, and do a setgid,setgroups,setuid to flip
        to the user in the child.
      
      * A instance of the emulabserver class is created, and the request is
        dispatched.
      
      I added an sslxmlrpc_client.py script that mirrors the ssh version of
      the client script. I could probably roll these into one, but decided
      not to to avoid confusing people who might download it.
      5a025f36
  8. 26 Aug, 2004 1 commit
  9. 25 Aug, 2004 2 commits
  10. 23 Aug, 2004 1 commit
  11. 19 Aug, 2004 2 commits
  12. 18 Aug, 2004 1 commit
    • Leigh B. Stoller's avatar
      Some rather crude privledge level hacks to allow admin people (real · c2a4acd4
      Leigh B. Stoller authored
      shells on boss) to use the rpc server without an agent running.
      Using the no-passphrase key, these changes allow us to use the server
      from ops in a very restricted manner. This change is temporary, until
      I have something better in place. In the meantime, admin people change
      their auth keys files on *boss* as such:
      
      command="/usr/testbed/sbin/sshxmlrpc_server.py -ro",from="ops.emulab.net" ... rest of emulab generated key ...
      
      Note the -ro argument; very important!
      c2a4acd4
  13. 13 Aug, 2004 2 commits
    • Leigh B. Stoller's avatar
      Add three new methods to return all the event system goo to the event · 7d91c544
      Leigh B. Stoller authored
      scheduler running on ops.
      
      Okay, so this is probably not an ideal mechanism for wrapping up what are
      essentially DB queries, since there are no xmlrpc-over-ssh bindings for
      C. So we have to run an external client (sshxmlrpc_client) to talk to the
      server. This is not so bad in itself, but returning the eventlist as a
      single response seems like an unnatural approach for returning what is
      essentialy a stream of data. Besides, unless we expect people might want to
      write their own schedulers (highly unlikely), exporting this stuff via the
      XMLRPC server is a silly thing to do.
      
      Instead, some kind of protected DB access proxy might be more appropriate.
      I have no idea how this might operate, but its something to think about.
      7d91c544
    • Leigh B. Stoller's avatar
  14. 10 Aug, 2004 1 commit
  15. 09 Aug, 2004 1 commit
    • Leigh B. Stoller's avatar
      Major rework of the script interface to Emulab. Up to now we have been · 5ef8f70a
      Leigh B. Stoller authored
      supporting both a shell script driven interface, plus the newer XMLRPC
      interface. This change removes the script driven interface from boss,
      replacing it with just the XMLRPC interface. Since we like to maintain
      backwards compatability with interfaces we have advertised to users (and
      which we know are being used), I have implemented a script wrapper that
      exports the same interface, but which converts the operations into XMLRPC
      requests to the server. This wrapper is written in python and uses our
      locally grown xmlrpc-over-ssh library. Like the current "demonstation"
      client, you can take this wrapper to your machine that has python and ssh
      installed, and use it there; you do not need to use these services from
      just users.emulab.net. Other things to note:
      
      * The wrapper is a single python script that has a "class" for each wrapped
        script. Running the wrapper without any arguments will list all of the
        operations it supports. You can invoke the wrapper with the operation as
        its argument:
      
          {987} stoller$ script_wrapper.py swapexp --help
          swapexp -e pid,eid in|out
          swapexp pid eid in|out
          where:
               -w   - Wait for experiment to finish swapping
               -e   - Project and Experiment ID
               in   - Swap experiment in  (must currently be swapped out)
              out   - Swap experiment out (must currently be swapped in)
      
          Wrapper Options:
              --help      Display this help message
              --server    Set the server hostname
              --login     Set the login id (defaults to $USER)
              --debug     Turn on semi-useful debugging
      
         But more convenient is to create a set of symlinks so that you can just
         invoke the operation by its familiar scriptname. This is what I have
         done on users.emulab.net.
      
          {987} stoller$ /usr/tesbed/bin/swapexp --help
          swapexp -e pid,eid in|out
          swapexp pid eid in|out
      
      
      * For those of you talking directly to the RPC server from python, I have
        added a wrapper class so that you can issue requests to any of the
        modules from a single connection. Instead using /xmlrpc/modulename, you
        can use just /xmlrpc, and use method names of the form experiment.swapexp,
        node.reboot, etc.
      
        Tim this should be useful for the netlab client which I think opens up
        multiple ssh connections?
      
      * I have replaced the paperbag shell with a stripped down xmlrpcbag shell
        that is quite a bit simpler since we no longer allow access to anything
        but the RPC server. No interactive mode, no argument processing, no
        directory changing, etc. My main reason for reworking the bag is to make
        it easier to understand, maintain, and verify that it is secure. The new
        bag also logs all connections to syslog (something we should have done in
        the orginal). I also added some setrlimit calls (core, maxcpu). I also
        thought about niceing the server down, but that would put RPC users at a
        disadvantage relative to web interface users. When we switch the web
        interface to use the XMLRPC backend, we can add this (reniceing from the
        web server would be a pain cause of its scattered implementation).
      5ef8f70a
  16. 03 Aug, 2004 2 commits
    • Leigh B. Stoller's avatar
      Add "server" debug wrapper so admin folks can run the new client · 79b1bcc5
      Leigh B. Stoller authored
      interface. This stuff is going to dissappear pretty soon, once we
      shift from the paperbag to the xmlrpcbag.
      79b1bcc5
    • Leigh B. Stoller's avatar
      A couple more minor changes before I turn the new stuff loose. · 8fddf3ce
      Leigh B. Stoller authored
      * Added a wrapper class so that you can invoke methods as
        experiment.swapexp or node.reboot. So instead of invoking as
        /XMLRPC/experiment can calling swapexp, you can call the server as
        /XMLRPC and call experiment.swapexp. This allows you to use a single
        connection to talk to different parts of the API. Note this is standard
        (or is it defacto) syntax in XMLRPC.
      
      * Changed the demonstration client to talk the server this way.
      
      * Changed paperbag to allow this as well; the xmlrpc server is invoked with
        no args, which tells it to export the wrapper interface instead of a
        specific module interface.
      
      * A few more cleanups in the server, more permission checks, etc.
      8fddf3ce
  17. 31 Jul, 2004 1 commit
  18. 30 Jul, 2004 1 commit
    • Leigh B. Stoller's avatar
      Preparation for replacing script interface on ops with XMLRPC only · c0cb83e4
      Leigh B. Stoller authored
      interface (which will have its own frontend to emulate the original
      script interface!)
      
      * Refactor the experiment permission checks into a subroutine.
      
      * Cleanup some parameter names that did not make sense.
      
      * Add portstats, readycount, node_admin, eventsys_control, savelogs,
      
      * A couple of little fixes.
      c0cb83e4
  19. 26 Jul, 2004 1 commit
  20. 19 Jul, 2004 1 commit
  21. 15 Jul, 2004 1 commit
    • Timothy Stack's avatar
      · e9b599df
      Timothy Stack authored
      Fix experiment.info() to conditionally add the "idle" field to the
      mapping aspect.
      e9b599df
  22. 12 Jul, 2004 2 commits
  23. 21 Jun, 2004 1 commit
  24. 04 Jun, 2004 1 commit
    • Leigh B. Stoller's avatar
      Back out previous revision that changed the XML output format to RPC · f8ed2e3b
      Leigh B. Stoller authored
      format instead of plain old well formed XML. Note this change does not
      affect the parser in any way; the parser continues to spit out its own
      brand of XML, and xmlconvert treats that special when it reads it.
      
      The XML I spit out now is better formed and makes more sense when you
      look at it. It also helps that its much faster to generate, and the
      resulting output is much smaller. For example:
      
      	time /usr/testbed/libexec/xmlconvert testbed jail-2440
      	real    2m54.229s
      	user    1m7.492s
      	sys     1m15.927s
      
      	time /usr/testbed/devel/stoller/libexec/xmlconvert testbed jail-2440
      	real    0m14.738s
      	user    0m5.755s
      	sys     0m5.156s
      
      	/tmp/bar*.xml
      	-rw-rw-r--  1 stoller  wheel  15595161 Jun  4 11:16 /tmp/rpc.xml
      	-rw-rw-r--  1 stoller  wheel   5285690 Jun  4 11:17 /tmp/plain.xml
      
      Thats for a big experiment. For a small experiment:
      
      	time /usr/testbed/libexec/xmlconvert testbed jail-416
      	real    0m9.346s
      	user    0m5.083s
      	sys     0m2.675s
      
      	time /usr/testbed/devel/stoller/libexec/xmlconvert testbed jail-416
      	real    0m2.751s
      	user    0m1.377s
      	sys     0m1.183s
      
      	ll /tmp/bar*.xml
      	-rw-rw-r--  1 stoller  wheel  3135518 Jun  4 11:19 /tmp/rpc.xml
      	-rw-rw-r--  1 stoller  wheel  1015745 Jun  4 11:19 /tmp/plain.xml
      
      As you can see, the RPC code is rather non-linear in its performance!
      
      Since this affects Tim and netlab-client, I have added an alternate
      XMLRPC routine called virtual_topology_xml() that returns the new format,
      since Tim will need to transition to this new format (parsing it into a
      Java data structure). The old format is left in the RPC server until that
      is done.
      f8ed2e3b
  25. 17 May, 2004 2 commits
    • Leigh B. Stoller's avatar
      Oops, typo'ed the name of the table. · d7780943
      Leigh B. Stoller authored
      d7780943
    • Leigh B. Stoller's avatar
      Redo virtual_topology call to be faster: · 6fb0009d
      Leigh B. Stoller authored
      For tbres/resweb (75 nodes):
      
      	Old time: 42 seconds.
      	New time: 19 seconds.
      
      For testbed/BigLan (600 nodes):
      
      	Old time: 120 seconds.
      	New time: 40  seconds.
      
      So, still needs some work, but for the average case its not so bad
      anymore.
      
      I also turn on compression to level 5 by default in our sshxmlrpc.py
      library, thereby overriding the user's config file for that option.
      6fb0009d
  26. 13 May, 2004 1 commit
  27. 12 May, 2004 2 commits
  28. 11 May, 2004 2 commits