1. 15 Jul, 2004 1 commit
    • Leigh Stoller's avatar
      Overview: Add Event Groups: · ed964507
      Leigh Stoller authored
      	set g1 [new EventGroup $ns]
      	$g1 add  $link0 $link1
      	$ns at 60.0 "$g1 down"
      
      See the new advanced tutorial section on event groups for a better
      example.
      
      Changed tbreport to dump the event groups table when in summary mode.
      At the same time, I changed tbreport to use the recently added
      virt_lans:vnode and ip slots, decprecating virt_nodes:ips in one more
      place. I also changed the web interface to always dump the event and
      event group summaries.
      
      The parser gets a new file (event.tcl), and the "at" method deals with
      event group events by expanding them inline into individual events
      sent to each member. For some agents, this is unavoidable; traffic
      generators get the initial params in the event, so it is not possible
      to send a single event to all members of the group. Same goes for
      program objects, although program objects do default to the initial
      command now, at least on new images.
      
      Changed the event scheduler to load the event groups table. The
      current operation is that the scheduler expands events sent to a
      group, into a set of distinct events sent to each member of the
      group. At some point we proably want to optimize this by telling the
      agents (running on the nodes) what groups they are members of.
      
      Other News: Added a "mustdelay" slot to the virt_lans table so the
      parser can tell assign_wrapper that a link needs to be delayed, say if
      there are events or if the link is red/gred. Previously,
      assign_wrapper tried to figure this out by looking at the event list,
      etc. I have removed that code; see database-migrate for instructions
      on how to initialize this slot in existing experiments. assign_wrapper
      is free to ignore or insert delays anyway, but having the parser do
      this makes more sense.
      
      I also made some "rename" changes to the parser wrt queues and lans
      and links. Not really necessary, but I got sidetracked (for several
      hours!) trying to understand that rename stuff a little better, and
      now I do.
      ed964507
  2. 07 Jul, 2004 1 commit
  3. 04 Jun, 2004 1 commit
    • Leigh Stoller's avatar
      Back out previous revision that changed the XML output format to RPC · f8ed2e3b
      Leigh Stoller authored
      format instead of plain old well formed XML. Note this change does not
      affect the parser in any way; the parser continues to spit out its own
      brand of XML, and xmlconvert treats that special when it reads it.
      
      The XML I spit out now is better formed and makes more sense when you
      look at it. It also helps that its much faster to generate, and the
      resulting output is much smaller. For example:
      
      	time /usr/testbed/libexec/xmlconvert testbed jail-2440
      	real    2m54.229s
      	user    1m7.492s
      	sys     1m15.927s
      
      	time /usr/testbed/devel/stoller/libexec/xmlconvert testbed jail-2440
      	real    0m14.738s
      	user    0m5.755s
      	sys     0m5.156s
      
      	/tmp/bar*.xml
      	-rw-rw-r--  1 stoller  wheel  15595161 Jun  4 11:16 /tmp/rpc.xml
      	-rw-rw-r--  1 stoller  wheel   5285690 Jun  4 11:17 /tmp/plain.xml
      
      Thats for a big experiment. For a small experiment:
      
      	time /usr/testbed/libexec/xmlconvert testbed jail-416
      	real    0m9.346s
      	user    0m5.083s
      	sys     0m2.675s
      
      	time /usr/testbed/devel/stoller/libexec/xmlconvert testbed jail-416
      	real    0m2.751s
      	user    0m1.377s
      	sys     0m1.183s
      
      	ll /tmp/bar*.xml
      	-rw-rw-r--  1 stoller  wheel  3135518 Jun  4 11:19 /tmp/rpc.xml
      	-rw-rw-r--  1 stoller  wheel  1015745 Jun  4 11:19 /tmp/plain.xml
      
      As you can see, the RPC code is rather non-linear in its performance!
      
      Since this affects Tim and netlab-client, I have added an alternate
      XMLRPC routine called virtual_topology_xml() that returns the new format,
      since Tim will need to transition to this new format (parsing it into a
      Java data structure). The old format is left in the RPC server until that
      is done.
      f8ed2e3b
  4. 29 Apr, 2004 1 commit
    • Leigh Stoller's avatar
      Fix for the problem of not setting default values when inserting · 6ab03d5f
      Leigh Stoller authored
      experiment table rows after a parse. It used to be that slots that
      were not explicitly given values by the parser would be left as
      whatever they were in the DB. Now I get the default values by doing
      a "describe experiments" query, inserting those values for any slot
      that is not set by the parser.
      6ab03d5f
  5. 23 Apr, 2004 1 commit
    • Leigh Stoller's avatar
      I have changed xmlconvert to do the following (note that I have not changed · a3ad59cd
      Leigh Stoller authored
      the parser; it still spits out the simplistic xml stuff and xmlconvert
      still reads that):
      
      * When generating an XML representation of an experiment, I read the
        virtual tables and build a *flat* data structure.
      
      	{"settings" => {pid:testbed, eid:foobar ....}.
      	 "nodes"    => [ {vname:nodeA, osid:RHL-STD ... },  ],
      	 "lans"     => [ {vname:link0, member:nodeA ... },  ],
               ...
      
        So, nodes is a list of node hash tables, lans is a list of hash tables,
        etc. All of the other virt_tables are represented as well.  You can find
        everything you want as long as you the names of the fields!
      
      * Once I have the above data structure, I use the XML-RPC lib to turn it
        into an XML document, and spit out the resulting XML string (a very long
        string!) Note that XML-RPC has its own format that allows you to convert
        data structures into XML and back; thats how you issue an RPC from your
        python client to a perl server.
      
      * Our python XMLRPC server basically uses xmlconvert as a backend so it can
        ship the above data structure back to client. We could also allow the
        client to ship the data structure back, and have xmlconvert upload the
        changes back to the DB.
      
        The reason for doing it in xmlconvert as a backend is so that this XML
        stuff is in just one place; whoever reads/writes this stuff has to know a
        lot of internal detail about an experiment (like what things should not
        be exported and what things we should not allow to be imported). I prefer
        to keep that all in one place since its easy to pass XML strings around
        (as long as everyone is talking the same dialect).
      
      Future work is to pick an XML representation that is not tied to
      XML-RPC. Ideally, we want to take a data structure in Perl, spit out
      the XML tree (with a DTD) and be able to reconstitute that in another
      program, using the XML Parser library (based on expat, which is common
      to Perl, Python, C).
      a3ad59cd
  6. 12 Apr, 2004 1 commit
  7. 10 Feb, 2004 1 commit
  8. 02 Feb, 2004 1 commit
  9. 28 Jan, 2004 1 commit
  10. 09 Jan, 2004 1 commit
    • Shashi Guruprasad's avatar
      During a simparse, virt_agents and eventlist need to be cleaned up · 467cb150
      Shashi Guruprasad authored
      but not other virt_* tables. Thats coz the above two tables are not
      truly virtual since they contain the vnode to which the event should
      be sent to. My previous patch has been to use replace instead of
      insert. Unfortunately, the tables get messed up with the same agent
      having multiple entries where some of the entries were left behind
      from the previous swapin.
      467cb150
  11. 29 Dec, 2003 1 commit
    • Shashi Guruprasad's avatar
      If xmlconvert is called after an nseparse, we need to use replace coz · 08a40103
      Shashi Guruprasad authored
      some of the tables such as virt_agents and eventlist are not truly
      virtual tables. That is coz they contain the vnode field which is
      the same as the vname field in the reserved table. For simulated
      nodes, the mapping may change across swapins and the event may have
      to be delivered to a different simhost. Of course this problem
      isn't there for jail vnodes or pc vnodes.
      08a40103
  12. 27 Dec, 2003 1 commit
    • Shashi Guruprasad's avatar
      nseconfigs table is used both as a virtual table and a physical table. · 5f27af03
      Shashi Guruprasad authored
      When xmlconvert is called during nseparse, we need to clean up the
      physical part so that rows for the new mapping can be added. I have
      added this. While I say that there is a physical part, essentially
      the vname column has the same value as the vname column in the
      reserved table for the PC such as 'simhost-0', 'simhost-1' etc.
      5f27af03
  13. 18 Dec, 2003 1 commit
    • Leigh Stoller's avatar
      First try at solving the problem of validating user input for the · 8dbead16
      Leigh Stoller authored
      zillions of DB fields that we have to set. My solution was to add a
      meta table that describes what is a legal value for each table/slot
      for which we take from user input. The table looks like this right
      now, but is likely to adapt as we get more experience with this
      approach (or it might get tossed if it turns out to be a pain in the
      ass!).
      
      	CREATE TABLE table_regex (
      	  table_name varchar(64) NOT NULL default '',
      	  column_name varchar(64) NOT NULL default '',
      	  column_type enum('text','int','float') default NULL,
      	  check_type enum('regex','function','redirect') default NULL,
      	  check tinytext NOT NULL,
      	  min int(11) NOT NULL default '0',
      	  max int(11) NOT NULL default '0',
      	  comment tinytext,
      	  UNIQUE KEY table_name (table_name,column_name)
      	) TYPE=MyISAM;
      
      Entries in this table look like this:
      
      	('virt_nodes','vname','text','regex','^[-\\w]+$',1,32,NULL);
      
      Which says that the vname slot of the virt_nodes table (which we trust the
      user to give us in some form) is a text field to be checked with the given
      regex (perlre of course), and that the min/max length of the text field is
      1 and 32 chars respectively.
      
      Now, you wouldn't want to write the same regex over and over, and since we
      use the same fields in many tables (like pid, eid, vname, etc) there is an
      option to redirect to another entry (recursively). So, for "PID" I do this:
      
              ('eventlist','pid','text','redirect','projects:pid',0,0,NULL);
      
      which redirects to:
      
      	('projects','pid','text','regex','^[a-zA-Z][-\\w]+$',2,12,NULL);
      
      And, for many fields you just want to describe generically what could go
      into it. For that I have defined some default fields. For example, a user
      description:
      
              ('experiment,'usr_name','text','redirect','default:tinytext',0,0,NULL);
      
      which redirects to:
      
      	('default','tinytext','text','regex','^[\\040-\\176]*$',0,256,NULL);
      
      and this says that a tinytext (in our little corner of the database
      universe) field can have printable characters (but not a newline), and
      since its a tinytext field, its maxlen is 256 chars.
      
      You also have integer fields, but these are little more irksome in the
      details.
      
      	('default','tinyint,'int,'regex','^[\\d]+$',-128,127,NULL);
      
      and you would use this anyplace you do not care about the min/max values
      being something specific in the tinyint range. The range for a float is of
      course stated as an integer, and thats kinda bogus, but we do not have many
      floats, and they generally do not take on specific values anyway.
      
      A note about the min/max fields and redirecting. If the initial entry has
      non-zero min/max fields, those are the min mac fields used. Otherwise they
      come from the default. So for example, you can do this:
      
          ('experiments','mem_usage','int','redirect','default:tinyint',0,5,NULL);
      
      So, you can redirect to the standard "tinyint" regular expression, but you
      still get to define min/max for the specific field.
      
      Isn't this is really neat and really obtuse too? Sure, you can say it.
      
      Anyway, xmlconvert now sends all of its input through these checks (its
      all wrapped up in library calls), and if a slot does not have an entry, it
      throws an error so that we are forced to define entries for new slots as we
      add them.
      
      In the web page, I have changed all of the public pages (login, join
      project, new project, and a couple of others) to also use these checks.
      As with the perl code, its all wrapped up in a library. Lots more code
      needs to be changed of course, but this is a start.
      8dbead16
  14. 15 Dec, 2003 1 commit
    • Shashi Guruprasad's avatar
      Distributed NSE changes. In other words, simulation resources are · d266bd71
      Shashi Guruprasad authored
      now mapped to more than one PC if required. The simnode_capacity
      column in the node_types table determines how many sim nodes can
      be packed on one PC. The packing factor can also be controlled via
      tb-set-colocate-factor to be smaller than simnode_capacity.
      
      - No frontend code changes. To summarize:
        $ns make-simulated {
          ...
        }
        is still the easy way to put a whole bunch of Tcl code to be
        in simulation.
        One unrelated fix in the frontend code is to fix the
        xmlencode() function which prior to this would knock off
        newlines from columns in the XML output. This affected
        nseconfigs since it is one of the few columns with embedded
        newlines. Also changed the event type and event object type
        in traffic.tcl from TRAFGEN/MODIFY to NSE/NSEEVENT.
      
      - More Tcl code in a new directory tbsetup/nseparse
        -> Runs on ops similar to the main parser. This is invoked
           from assign_wrapper in the end if there are simnodes
        -> Partitions the Tcl code into multiple Tcl specifications
           and updates the nseconfigs table via xmlconvert
        -> Comes with a lot of caveats. Arbitrary Tcl code such as user
           specified objects or procedures will not be re-generated. For
           example, if a user wanted a procedure to be included in Tcl
           code for all partitions, there is no way for code in nseparse
           to do that. Besides that, it needs to be tested more thoroughly.
      
      - xmlconvert has a new option -s. When invoked with this option,
        the experiments table is not allowed to be modified. Also,
        virtual tables are just updated (as opposed to deleting
        all rows in the first invocation before inserting new rows)
      
      - nse.patch has all the IP address related changes committed in
        iversion 1.11 + 2 other changes. 1) MTU discovery support in
        the ICMP agent 2) "$ns rlink" mechanism for sim node to real
        node links
      
      - nseinput.tcl includes several client side changes to add IP
        routes in NSE and the kernel routing table for packets crossing
        pnodes. Also made the parsing of tmcc command output more robust
        to new changes. Other client side changes in libsetup.pm and other
        scripts to run nse, are also in this commit
      
      - Besides the expected changes in assign_wrapper for simulated nodes,
        the interfaces and veth_interfaces tables are updated with
        routing table identifiers (rtabid). The tmcd changes are already
        committed. This field is used only by sim hosts on the client side.
        Of course, they can be used by jails as well if desired.
      d266bd71
  15. 02 Dec, 2003 1 commit
  16. 12 Nov, 2003 1 commit
    • Leigh Stoller's avatar
      Changes/Fixes to allow Shashi to spit NS files into the DB from the · fb2a8122
      Leigh Stoller authored
      parser:
      
      * Change xmlconvert to use XML::Parser instead of the ad-hoc stuff I
        wrote. This was easier then trying to deal with multiline fields or
        all the special chars. XML::Parser does all that for me. Also make
        sure all fields are either properly quoted (with DBQuoteMeta) or
        regex'ed before we stick them into the DB. For the future, we need
        to add table/slot specific regular expressions.
      
      * Also add Shashi's fixes to the parser; properly escape XML special
        characters, as well as cleanup of my original spitxml code.
      fb2a8122
  17. 05 Nov, 2003 1 commit
    • Leigh Stoller's avatar
      Frontend and parser portion of two event system changes: · 091a0b62
      Leigh Stoller authored
      * Generate a shared secret key for the event system. This key is
        stored into the DB, and passed to the node via tmcd. It is also
        stashed into a file in the experiment directory (can be accessed
        only by the project/group members). The key is used to attach a
        HMAC (hashed message authentication) to each event, which is checked
        by the receivers to ensure that the event is not bogus. More details
        on this later when I commit the event library/client changes.
      
      * Added "virt_programs" table to store info about each program object
        defined by the user. The intent is to no longer send the command
        string in the event, but to fix it in the DB, and transfer it via
        tmcd. This removes our "remote execution facility" which was always
        a bad idea (we have ssh for that, and that is a lot more secure then
        the event system!).
      
        Note that for the time being we need to continue send the command in
        the event because of old images, but the new images will now ignore
        that part of the event.
      091a0b62
  18. 26 Sep, 2003 1 commit
  19. 11 Sep, 2003 1 commit
  20. 05 Aug, 2003 1 commit
    • Leigh Stoller's avatar
      The rest of the sync server additions: · 212cc781
      Leigh Stoller authored
      * Parser: Added new tb command to set the name of the sync server:
      
      	tb-set-sync-server <node>
      
        This initializes the sync_server slot of the experiment entry to the
        *vname* of the node that should run the sync server for that
        experiment. In other words, the sync server is per-experiment, runs
        on a node in the experiment, and the user gets to chose which node
        it runs on.
      
      * tmcd and client side setup. Added new syncserver command which
        returns the name of the syncserver and whether the requesting node
        is the lucky one to run the daemon:
      
          SYNCSERVER SERVER='nodeG.syncserver.testbed.emulab.net' ISSERVER=1
      
        The name of the syncserver is written to /var/emulab/boot/syncserver
        on the nodes so that clients can easily figure out where the server
        is.
      
        Aside: The ready bits are now ignored (no DB accesses are made) for
        virtual nodes; they are forced to use the new sync server.
      
      * New os/syncd directory containing the daemon and the client. The
        daemon is pretty simple. It waits for TCP (and UDP, although that
        path is not complete yet) connections, and reads in a little
        structure that gives the name of the "barrier" to wait for, and an
        optional count of clients in the group (this would be used by the
        "master" who initializes barriers for clients). The socket is saved
        (no reply is made, so the client is blocked) until the count reaches
        zero. Then all clients are released by writting back to the
        sockets, and the sockets are closed. Obviously, the number of
        clients is limited by the numbed of FDs (open sockets), hence the
        need for a UDP variant, but that will take more work.
      
        The client has a simple command line interface:
      
          usage: emulab-sync [options]
          -n <name>         Optional barrier name; must be less than 64 bytes long
          -d                Turn on debugging
          -s server         Specify a sync server to connect to
          -p portnum        Specify a port number to connect to
          -i count          Initialize named barrier to count waiters
          -u                Use UDP instead of TCP
      
          The client figures out the server by looking for the file created
          above by libsetup (/var/emulab/boot/syncserver). If you do not
          specify a barrier "name", it uses an internal default. Yes, the
          server can handle multiple barriers (differently named of course)
          at once (non-overlapping clients obviously).
      
          Clients can wait before a barrier in "initialized." The count on
          the barrier just goes negative until someone initializes the
          barrier using the -i option, which increments the count by the
          count. Therefore, the master does not have to arrange to get there
          "first." As an example, consider a master and one client:
      
      	nodeA> /usr/local/etc/emulab/emulab-sync -n mybarrier
      	nodeB> /usr/local/etc/emulab/emulab-sync -n mybarrier -i 1
      
          Node A waits until Node B initializes the barrier (gives it a
          count).  The count is the number of *waiters*, not including the
          master. The master is also blocked until all of the waiters have
          checked in.
      
          I have not made an provision for timeouts or crashed clients. Lets
          see how it goes.
      212cc781
  21. 15 Jul, 2003 1 commit
  22. 02 Jul, 2003 1 commit
    • Leigh Stoller's avatar
      Hack for dealing with user output from the parser (puts). Since I spit · 10673342
      Leigh Stoller authored
      the XML to stdout, user calls to puts confuse the converter, and get
      lost so the user never sees them (before, they would end up in the
      start/preload log). The converter now eats up lines till it sees a
      valid XML header string, printing those lines to stdout so they end up
      in the log like before.
      
      Need to deal with this in a better manner at some point. I did not
      want to write the XML to temporary file. Not sure why, but I thought
      that was the right approach, to keep the user from getting his grubby
      little hands on it. I'm thinking now thats silly, and I'll eventually
      fix things up (since it requires more TCL hacking, and I don't have
      the energy for that).
      10673342
  23. 30 Jun, 2003 1 commit
    • Leigh Stoller's avatar
      Make the new parser live on mini. New parser ssh'es over to ops to · 2202fc5a
      Leigh Stoller authored
      do the actual parse. The parser now spits out XML instead of DB
      queries, and the wrapper on boss converts that to DB insertions after
      verification. There are some makefile changes as well to install the
      new parser on ops via NFS, since otherwise the parser could
      intolerably out of date on ops!
      2202fc5a
  24. 26 Jun, 2003 1 commit
    • Leigh Stoller's avatar
      New script. Convert experiment to a "well formed" XML, and back again. · f7fc7ff3
      Leigh Stoller authored
      The conversion from XML to an experiment representation updates the
      DB, although the experiment has to exist, and only certain fields can
      be updated. For the rest of the virtual toplogy, the old toplogy is
      deleted, and the new one installed. Use with caution. This is intended
      to be hooked to the parser, and later to the vis tool.
      f7fc7ff3