1. 07 Jul, 2004 1 commit
  2. 02 Jul, 2004 1 commit
  3. 29 Jun, 2004 1 commit
  4. 25 Jun, 2004 1 commit
  5. 23 Jun, 2004 1 commit
    • Leigh B. Stoller's avatar
      Switch to "dynamic" creation of certain virtual nodes (jail/pcvm and · 4d5620c2
      Leigh B. Stoller authored
      vserver/pcplab). Rather then 10000s of fixed node entries in the DB,
      create node entries on the fly as needed (assign_wrapper), and then
      delete them when they are no longer used (nfree). When new nodes check
      in, no longer create all those nodes table entries (utils/newnode.in
      and tbsetup/plab/libplab.py.in).
      
      I've added a new library: db/Node.pm which is something I started a
      while back, and decided to commit, along with the support for creating
      and deleting virtual nodes. CreateVnodes() creates a new set of nodes,
      choosing non-conflicting names in the DB, and then immediately
      reserves them to the pid/eid specified. DeleteVnodes takes a list of
      vnodes and deletes them from nodes,reserved,etc. This library does a
      few other things which I am going to be playing with, so you might
      want to go read the comment at the top of the file. Feel free to speak
      up. CreateVnodes() is from called assign_wrapper when a node type has
      the "isdynamic" property. Otherwise does the existing avail/nalloc
      stuff. DeleteVnodes() is called from nfree when the node type has the
      isdynamic property.
      
      I've added a script (sql/delvnodes.pl) to run after updating the DB
      and software. All free pcvm and pcplab virtual nodes are deleted from
      the DB; reserved ones will get deleted whenever their experiment ends.
      I've noted all of this in doc/UPDATING, including setting the
      isdynamic property on pcvm in the node_types table.
      
      I've left tbsetup/plab/libplab.py.in to create a single pcplab node
      for the management sliver (still called -20). We can worry about this
      later.
      
      All this for modelnet?
      4d5620c2
  6. 19 Jun, 2004 1 commit
    • Leigh B. Stoller's avatar
      A number of changes: · 931930ae
      Leigh B. Stoller authored
      * Put in debugging timestamps to figure out why so slow.
      
      * Cleanup, fixing bad identing, bad style, bad comments, etc.
      
      * Restructure so that instead of locking three tables (nodes,
        reserved, and node_activity), we just need to lock the reserved
        table, by moving the node existence check to before the main loop,
        and the node_activity update to after we unlock tables. Not locking
        the nodes table results in less serialization and less waiting for
        the lock to be granted.
      
      * XXX: Check each node for an entry in tiplines table; Do not call
        into console_setup.in if none of the nodes have tiplines. This is
        not a very clean thing to do of course, but it saves up to half the
        time it takes to do an nalloc cause of the min startup time for our
        perl scripts (almost 1/2 a second). When allocating virtnodes on a
        physnode, no reason to even call console_setup.in.
      
      Bottom line? Well, not as good as one would like ... time to allocate
      10 vnodes on a pnode goes from .8 seconds to .4 seconds ...
      931930ae
  7. 08 Jun, 2004 1 commit
  8. 04 Jun, 2004 1 commit
    • Leigh B. Stoller's avatar
      Back out previous revision that changed the XML output format to RPC · f8ed2e3b
      Leigh B. Stoller authored
      format instead of plain old well formed XML. Note this change does not
      affect the parser in any way; the parser continues to spit out its own
      brand of XML, and xmlconvert treats that special when it reads it.
      
      The XML I spit out now is better formed and makes more sense when you
      look at it. It also helps that its much faster to generate, and the
      resulting output is much smaller. For example:
      
      	time /usr/testbed/libexec/xmlconvert testbed jail-2440
      	real    2m54.229s
      	user    1m7.492s
      	sys     1m15.927s
      
      	time /usr/testbed/devel/stoller/libexec/xmlconvert testbed jail-2440
      	real    0m14.738s
      	user    0m5.755s
      	sys     0m5.156s
      
      	/tmp/bar*.xml
      	-rw-rw-r--  1 stoller  wheel  15595161 Jun  4 11:16 /tmp/rpc.xml
      	-rw-rw-r--  1 stoller  wheel   5285690 Jun  4 11:17 /tmp/plain.xml
      
      Thats for a big experiment. For a small experiment:
      
      	time /usr/testbed/libexec/xmlconvert testbed jail-416
      	real    0m9.346s
      	user    0m5.083s
      	sys     0m2.675s
      
      	time /usr/testbed/devel/stoller/libexec/xmlconvert testbed jail-416
      	real    0m2.751s
      	user    0m1.377s
      	sys     0m1.183s
      
      	ll /tmp/bar*.xml
      	-rw-rw-r--  1 stoller  wheel  3135518 Jun  4 11:19 /tmp/rpc.xml
      	-rw-rw-r--  1 stoller  wheel  1015745 Jun  4 11:19 /tmp/plain.xml
      
      As you can see, the RPC code is rather non-linear in its performance!
      
      Since this affects Tim and netlab-client, I have added an alternate
      XMLRPC routine called virtual_topology_xml() that returns the new format,
      since Tim will need to transition to this new format (parsing it into a
      Java data structure). The old format is left in the RPC server until that
      is done.
      f8ed2e3b
  9. 20 May, 2004 1 commit
    • Leigh B. Stoller's avatar
      Add EventFork() to event.pm (perl interface to event system) and to · 116539b6
      Leigh B. Stoller authored
      the tail file of course. Called from TBdbfork() in libdb, EventFork
      resets the event handle so that the child does a reconnect. Note that
      I do not disconnect in the child since I have no idea what that is
      going to do to the parents connection to the elvind, as Elvin makes no
      mention of what to do in the presence of a process that forks after
      connecting to the event server. At the least, this avoids a bunch of
      warnings and errors from vnodesetup!
      116539b6
  10. 05 May, 2004 1 commit
    • Leigh B. Stoller's avatar
      Add new script to change the uid of a user, since this seems to be · 9f73d582
      Leigh B. Stoller authored
      something we are continually being asked to do! This script will only
      operate in UNAPPROVED users; once a user is approved it becomes too
      difficult to deal with cause of existing directories which might be
      exported to active experiments. We can probably deal with this, but we
      rarely do this to an approved user, so its not a big deal.
      
      Obvious: This script can get out of sync with the DB, but of course.
      9f73d582
  11. 29 Apr, 2004 1 commit
    • Leigh B. Stoller's avatar
      Fix for the problem of not setting default values when inserting · 6ab03d5f
      Leigh B. Stoller authored
      experiment table rows after a parse. It used to be that slots that
      were not explicitly given values by the parser would be left as
      whatever they were in the DB. Now I get the default values by doing
      a "describe experiments" query, inserting those values for any slot
      that is not set by the parser.
      6ab03d5f
  12. 23 Apr, 2004 1 commit
    • Leigh B. Stoller's avatar
      I have changed xmlconvert to do the following (note that I have not changed · a3ad59cd
      Leigh B. Stoller authored
      the parser; it still spits out the simplistic xml stuff and xmlconvert
      still reads that):
      
      * When generating an XML representation of an experiment, I read the
        virtual tables and build a *flat* data structure.
      
      	{"settings" => {pid:testbed, eid:foobar ....}.
      	 "nodes"    => [ {vname:nodeA, osid:RHL-STD ... },  ],
      	 "lans"     => [ {vname:link0, member:nodeA ... },  ],
               ...
      
        So, nodes is a list of node hash tables, lans is a list of hash tables,
        etc. All of the other virt_tables are represented as well.  You can find
        everything you want as long as you the names of the fields!
      
      * Once I have the above data structure, I use the XML-RPC lib to turn it
        into an XML document, and spit out the resulting XML string (a very long
        string!) Note that XML-RPC has its own format that allows you to convert
        data structures into XML and back; thats how you issue an RPC from your
        python client to a perl server.
      
      * Our python XMLRPC server basically uses xmlconvert as a backend so it can
        ship the above data structure back to client. We could also allow the
        client to ship the data structure back, and have xmlconvert upload the
        changes back to the DB.
      
        The reason for doing it in xmlconvert as a backend is so that this XML
        stuff is in just one place; whoever reads/writes this stuff has to know a
        lot of internal detail about an experiment (like what things should not
        be exported and what things we should not allow to be imported). I prefer
        to keep that all in one place since its easy to pass XML strings around
        (as long as everyone is talking the same dialect).
      
      Future work is to pick an XML representation that is not tied to
      XML-RPC. Ideally, we want to take a data structure in Perl, spit out
      the XML tree (with a DTD) and be able to reconstitute that in another
      program, using the XML Parser library (based on expat, which is common
      to Perl, Python, C).
      a3ad59cd
  13. 19 Apr, 2004 1 commit
  14. 12 Apr, 2004 1 commit
  15. 09 Apr, 2004 1 commit
  16. 08 Apr, 2004 1 commit
  17. 31 Mar, 2004 3 commits
  18. 25 Mar, 2004 1 commit
  19. 23 Mar, 2004 3 commits
    • Kirk Webb's avatar
      Couple of items: · ed848fd5
      Kirk Webb authored
      * Small fix to DBQueryFatal in libdb.py: () is a valid return value,
        don't fail on it(insert/replace); do fail if DBQuery returns None though.
      
      * Fix up libplab.py to not choke on new plab_slices column.
      ed848fd5
    • Kirk Webb's avatar
      Snapshot. · fd6d8cc9
      Kirk Webb authored
      * incompatible option handling and use removed from gen purpose libs
      * Global PLC mutex implemented, but currently disabled
      * plabmonitord parallelization cut in half (for now)
      
      I'm still very frustrated with option handling/passing.  Needs more thought,
      but the primary issue is that there really isn't a global variable space in
      python (global to file, yes, but not global to interpreter invocation).
      
      I've learned that __builtin__ might work for this, but it seems hacky..
      fd6d8cc9
    • Leigh B. Stoller's avatar
      Fix my last revision to that DBQueryFatal() and DBQuery() really do · af79be8f
      Leigh B. Stoller authored
      what I want them to do and properly mirror the perl impl.
      af79be8f
  20. 19 Mar, 2004 3 commits
  21. 17 Mar, 2004 1 commit
    • Kirk Webb's avatar
      Snapshot. · 856c2509
      Kirk Webb authored
      * Changed the way options are parsed in the python scripts so that modules
        can easily add and use their own options independent of top-level scripts.
      
      * Added --noIS and --pollNodes module options.
      
      * Added batch option to vnode_setup (degree of parallelization)
        - defaults to 10
      
      * Major updates to plamonitord
        - batches testing, currently to 40
      856c2509
  22. 16 Mar, 2004 2 commits
  23. 08 Mar, 2004 1 commit
  24. 26 Feb, 2004 1 commit
  25. 25 Feb, 2004 1 commit
    • Kirk Webb's avatar
      Kirk takes the weed whacker to the plab code. This is the first pass result. · ae2eec76
      Kirk Webb authored
      I'll come along for a closer cut in the future.
      
      * Modularized the plab communications 'adaptor' interface and moved the
        dslice- and PLC-specific code into their own modules.
      
      * Wrote an API definition README
      
      * Separated out generic routines from libplab into their own library modules
        (libtestbed.py and libdb.py)
      
      Functionally, not much has changed - this was just a massive re-org with some
      other cleanup.  Should be much easier to code up new PLAB interfaces as the
      plab folks flail around in their attempt to standardize on something.
      
      XXX: may want to re-think where the generic library modules should go.  If
      more python code enters Elab, we'll probably want to move 'em to more standard
      locations.
      
      This isn't the end of the cleanup - I would eventually like to go back and
      rethink the class structures, beef up the comments, and extend the API.
      ae2eec76
  26. 23 Feb, 2004 1 commit
  27. 16 Feb, 2004 2 commits
  28. 11 Feb, 2004 3 commits
  29. 10 Feb, 2004 1 commit
  30. 09 Feb, 2004 1 commit