1. 28 Mar, 2006 1 commit
  2. 19 Dec, 2005 1 commit
  3. 31 May, 2005 1 commit
  4. 16 May, 2005 1 commit
    • Leigh B. Stoller's avatar
      Add support for specifying the CVS tag to use when getting the source code · f1863cfd
      Leigh B. Stoller authored
      for the inner elab.
      
      	tb-set-elabinelab-cvstag dist-foo
      
      Will result in this branch getting checked out from the mirrored repository
      (updated nightly) on boss and sent back to the node, instead of the usual
      source tarball that we keep in /usr/testbed/src (still the default
      behaviour if no tag is specified. You can also do this if you like:
      
      	tb-set-elabinelab-cvstag HEAD
      
      which of course is a special tag to CVS.
      f1863cfd
  5. 12 May, 2005 1 commit
    • Leigh B. Stoller's avatar
      Checkpoint the rest of my changes to support swapmod of both ElabInElab and · 6eff9de6
      Leigh B. Stoller authored
      Firewalled experiments (see tbsetup/elabinelab.in for the other stuff).
      
      * To support firewalled experiments, needed to add a new virt_firewalls
        table to split the existing firewalls table up, which included both
        virtual and physical stuff. There are the usual frontend changes and a
        few other things scattered around, including tmcd.c.
      
      * The firewall code in tbswap got some beefing up to support adding and
        deleting nodes from the its special control net vlan. Note that I have
        not made any progress on containment of deleted nodes, just as we do not
        do anything now for teardown (unless its paniced, in which case the
        experiment cannot be modified anyway).
      
      * ptopgen and assign_wrapper got some interesting modifications: Unlike
        regular swapmod, we cannot just tear down all the vlans since that would
        interrupt everything inside the inner elab. Instead, leave the vlans as
        is. The problem is that when assign runs, it can just as easily pick
        different interfaces on the same nodes, which would be a royal pain in
        the ass to deal with! So, ptopgen got a new option (-u) that assign
        wrapper uses to tell ptopgen that it should prune out unused interfaces
        from nodes that are already allocated to the experiment. This is, at
        best, as pathetically gross hack, but it makes sure that all the
        interfaces stay the same across swapmods.
      
      * The unrelated revision of elabinelab has a bunch of new code for adding
        and deleting nodes from the inner elab. Mostly it deals with dhcpd (inner
        and outer, waiting for nodes to reboot, etc). It also deals with updating
        the vlans table in the DB, pruning out any nodes (ports) that are deleted
        but for which there are still interfaces in existing vlans. Said ports
        are them moved back to the default vlan with calls to snmpit. Also under
        another revision a a couple of weeks ago are the web interface changes to
        support the newnode MFS inside an inner Emulab.
      
      * swapexp and endexp got some more checks for firewalled and paniced
        experiments, which were missing.
      6eff9de6
  6. 04 Apr, 2005 1 commit
    • Timothy Stack's avatar
      · 463ee6b1
      Timothy Stack authored
      Mote and robot related stuff.  The main thing is the addition of relay
      capabilities to capture and related things.
      
      	* GNUmakefile.in: Add the capture and tip subdirectories to the
      	client and client-install targets.
      
      	* configure, configure.in, config.h.in: Detect srandomdev() for
      	capture and add "mote/newmote" script.
      
      	* capture/GNUmakefile.in, capture/capture.c: Add "relay"
      	capabilities to capture.
      
      	* capture/capquery.c: Query the capserver for the relay receiver's
      	port number.
      
      	* capture/capserver.c: Small hack to return the port number
              for a node.
      
      	* db/libdb.pm.in, db/xmlconvert.in: Add virt_tiptunnels table.
      
      	* event/program-agent/program-agent.c: Change log file names to
      	something a little more user-friendly.  Add a "MODIFY" event
      	handler that lets the user set agent attributes (command, tag,
      	timeout) without having to run a program.
      
      	* event/sched/GNUmakefile.in, event/sched/console-agent.cc,
      	event/sched/console-agent.h, event/sched/event-sched.c: Add
      	console agents that can be used to snapshot a section of the
      	capture log file.
      
      	* event/sched/node-agent.cc: Some minor cleanup.
      
      	* event/sched/simulator-agent.cc, event/sched/simulator-agent.h:
      	Add the config data to the report mail.  Add a "RESET" event
      	handler that runs "loghole clean".  Save the report mail in a file
      	so it gets archived with the rest of the logs.
      
      	* lib/libtb/tbdefs.h: Add CONSOLE object type.
      
      	* mote/GNUmakefile.in, mote/newmote: Add newmote script, just a
      	quick hack to add motes to the DB.
      
      	* mote/tbuisp.in: Add another backend for loading motes through
      	their relay capture server.
      
      	* robots/mtp/mtp_dump.c: Dump the min/max values for x and y,
      	handy for figuring out the bounds of the camera.
      
      	* sql/database-fill.sql: Change the RELOAD-MOTE/SHUTDOWN ->
      	ALWAYSUP/SHUTDOWN mode transition to ALWAYSUP/ISUP since stated
      	doesn't seem to run triggers after a state change by a mode
      	transition.
      
      	* tbsetup/tbreport.in: Change the ordering of the eventlist so it
      	displays event-sequences appropriately.
      
      	* tbsetup/ns2ir/GNUmakefile.in, tbsetup/ns2ir/console.tcl,
      	tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/parse.tcl.in,
      	tbsetup/ns2ir/sim.tcl.in: Add a "console" agent that represents
      	the serial console for a node.
      
      	* tbsetup/ns2ir/sequence.tcl: Add an "append" method so it is
      	easier to build sequences dynamically.
      
      	* tbsetup/ns2ir/topography.tcl: Make checkdest available to
      	regular users.
      
      	* tip/GNUmakefile.in, tip/tiptunnel.c: Add support for uploading a
      	file to a relay version of capture and exporting the end
      	connection as a pty.
      
      	* tmcd/decls.h, tmcd/common/libsetup.pm: Bump version number since
      	the dosubnodelist change is not backwards compatible.
      
      	* tmcd/tmcd.c: Make dosubnodelist and dosubconfig callable even
      	when a node isn't allocated.  Add dotiptunnels command that
      	returns which serial consoles are to be mounted on a node.  Add
      	mote version of subconfig that returns information needed to
      	startup the relay version of capture.
      
      	* tmcd/common/bootsubnodes: For motes, startup the relay version
      	of capture (XXX stargate specific).
      
      	* tmcd/common/libsetup.pm, tmcd/common/libtmcc.pm,
      	tmcd/common/config/rc.config, tmcd/common/config/rc.tiptunnels:
      	Client side changes for mounting another nodes serial line.
      
      	* tmcd/common/rc.bootsetup: Always boot the subnodes, even when
      	free.  This is used for motes since their capture needs to be up
      	for reloading at the time.
      
      	* tmcd/linux/ixpboot: Shuffle some code around so the script
      	doesn't fail if the ixp isn't allocated.
      
      	* utils/loghole.in: Add "digest.out" and "report.mail" as global
      	logs to be saved in archives and display the "report.mail" file
      	when showing a loghole archive.
      
      	* xmlrpc/emulabserver.py.in: Scrub more of the return values to
      	get rid of "None"s.
      463ee6b1
  7. 07 Mar, 2005 1 commit
    • Timothy Stack's avatar
      · 898cf9a2
      Timothy Stack authored
      Checkin some changes related to experiment automation and vnode feedback:
      
      	* configure, configure.in: Add sensors/canaryd/feedbacklogs
      	template.
      
      	* db/libdb.pm.in, db/xmlconvert.in: Add "virt_user_environment"
      	table that holds environment variable names and values.
      
      	* event/lib/event.c: Allocate memory of the right size for
      	event_notifications.
      
      	* event/program-agent/GNUmakefile.in: Add version.c file and
      	add install targets for the man page.
      
      	* event/program-agent/program-agent.8: Man page describing the
      	program-agent daemon.
      
      	* event/program-agent/program-agent.c: Add a bunch of convenience
      	features: let the user specify the working directory for commands;
      	save output to separate files on every invocation of an agent; let
      	the user specify a timeout for a command; make the set of
      	environment variables sane and add vars given in the NS file in
      	the opt array; a "status" file containing process information is
      	written out when children are collected.  Internal changes: child
      	processes are collected immediately, instead of waiting for the
      	next START event, so we can send back COMPLETE events; the daemon
      	now runs with a real-time priority, to increase the chances of
      	receiving events.
      
      	* event/proxy/evproxy.c: Made it bidirectional so the
      	program-agent's COMPLETE events make it back to the scheduler.
      
      	* event/sched/error-record.c: Change the default log directory.
      
      	* event/sched/event-sched.h, event/sched/event-sched.c: Setup an
      	environment similar to a program-agent to run the user's log
      	digester.
      
      	* event/sched/node-agent.cc: Add a handler for the SNAPSHOT event
      	that runs create_image for the node.
      
      	* event/sched/simulator-agent.h, event/sched/simulator-agent.cc:
      	Let the user specify a "DIGESTER" script that digests the log
      	files into a summary of the results.  Add event handler for
      	remapping a vnode experiment.
      
      	* event/sched/timeline-agent.c: Accept the RUN event as well as
      	the START event.
      
      	* os/GNUmakefile.in: Install the install-tarfile.1 man page.
      
      	* os/install-tarfile: Automatically chown/chgrp any files that do
      	not have valid user or group IDs, the new owner will be the user
      	that swapped in the experiment.  Include the install directory in
      	the DB file.  Add a "list" mode that just dumps what files have
      	been installed and where.  Add a "force" option so the user can
      	forcefully install the file, even though the DB says its already
      	there.
      
      	* os/install-tarfile.1: Man page describing the install-tarfile
      	tool.
      
      	* os/syncd/GNUmakefile.in: Install man pages on ops.
      
      	* sensors/canaryd/GNUmakefile.in: Link canaryd statically and
      	install "feedbacklogs" tool.
      
      	* sensors/canaryd/canaryd.c: Dump dummynet pipe data.
      
      	* sensors/canaryd/canarydEvents.c: Log errors.
      
      	* sensors/canaryd/feedbacklogs.in: Tool used to generate feedback
      	data from canaryd log files.
      
      	* sensors/slothd/GNUmakefile.in: Install digest-slothd on ops.
      
      	* sensors/slothd/digest-slothd: Fix some bugs and write out an
      	"alert" file with all the nodes/links that were overloaded.
      
      	* tbsetup/os_load.in, tbsetup/libosload.pm.in: Add "waitmode"
      	argument that lets you specify that you want to wait for the disk
      	to finish loading and/or wait for the node to come back up in the
      	new OS.
      
      	* tbsetup/power.in: Remove debugging printf.
      
      	* tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/program.tcl,
      	tbsetup/ns2ir/sequence.tcl, tbsetup/ns2ir/sim.tcl.in: Fix some
      	quoting problems with event-sequences.  Add -expected-exit-code
      	and -tag options to the "$program run" event.  Add -digester to
      	the "$ns report" event that lets the user specify a program to run
      	to digest the log files.
      
      	* tbsetup/ns2ir/tb_compat.tcl.in: Change the initial scaling
      	factor for feedback nodes to 1%, instead of 100%.
      
      	* tmcd/tmcd.c, tmcd/common/libtmcc.pm: Add "userenv" command that
      	returns the values in "virt_user_environment".  Return new program
      	agent fields: dir, timeout, and expected_exit_code.
      
      	* tmcd/common/GNUmakefile.in: Install rc.canaryd.
      
      	* tmcd/common/bootvnodes: Add hack to boost the program-agents to
      	a real-time priority, they can't do it from inside the jail.
      
      	* tmcd/common/rc.canaryd: Rc script for canaryd.
      
      	* tmcd/common/watchdog: Don't fail outright if there is a bad line
      	in the battery.log
      
      	* tmcd/common/rc.progagent: Append "userenv" data to the
      	program-agent config file.
      
      	* utils/GNUmakefile.in: Install loghole and its man page on ops.
      
      	* utils/loghole.1: Document "clean" command and the change in
      	loghole directories.
      
      	* utils/loghole.in: Add "clean" command and parallelization.
      
      	* xmlrpc/emulabserver.py.in: Add "virt_user_environment" table.
      	Order the eventlist by "idx" and time, needed for sequences.  And
      	removed unnecessary nologin checks.
      898cf9a2
  8. 02 Feb, 2005 1 commit
    • Leigh B. Stoller's avatar
      Add tb-set-delay-capacity NS directive: · 21044084
      Leigh B. Stoller authored
      	tb-set-delay-capacity 1
      
      Will override the default delay capacity as specfied in the node_types
      table for each node type, and set it for all types when generating the
      ptop file.
      
      This is a big stick, to be used with caution, as it will effectively
      double the number of nodes used for delay nodes (withing an experiment).
      21044084
  9. 17 Jan, 2005 1 commit
    • Timothy Stack's avatar
      · bf489797
      Timothy Stack authored
      More robot integration and some event system updates.
      
      	* configure, configure.in: Detect rsync for loghole and add
      	utils/loghole to the list of template files.
      
      	* db/libdb.pm.in, db/xmlconvert.in: Add virt_node_startloc to the
      	list of virtual tables.
      
      	* event/lib/event.h, event/lib/event.c, event/lib/tbevent.py.tail:
      	Add event_stop_main function to break out of the event_main()
      	loop.  Add timeline to the address tuple.
      
      	* event/sched/GNUmakefile.in, event/sched/error-record.h,
      	event/sched/error-record.c, event/sched/event-sched.8,
      	event/sched/event-sched.h, event/sched/event-sched.c,
      	event/sched/group-agent.h, event/sched/group-agent.c,
      	event/sched/listNode.h, event/sched/listNode.c,
      	event/sched/local-agent.h, event/sched/local-agent.c,
      	event/sched/node-agent.h, event/sched/node-agent.cc,
      	event/sched/queue.c, event/sched/rpc.h, event/sched/rpc.cc,
      	event/sched/simulator-agent.h, event/sched/simulator-agent.c,
      	event/sched/timeline-agent.h, event/sched/timeline-agent.c:
      	Updated event scheduler, not completely finished, but well enough
      	along for the robots.
      
      	* lib/libtb/GNUmakefile.in, lib/libtb/popenf.h,
      	lib/libtb/popenf.c, lib/libtb/systemf.h, lib/libtb/systemf.c: Add
      	some handy versions of system/popen that take format arguments.
      
      	* lib/libtb/tbdefs.h, lib/libtb/tbdefs.c: Add some more event and
      	object types.
      
      	* tbsetup/assign_wrapper.in: Add the virt_node_startloc building
      	to desires string for a node.
      
      	* tbsetup/ptopgen.in: Add a node's location to the feature list.
      
      	* tbsetup/tbreport.in: Display the timeline/sequence an event is a
      	part of.
      
      	* tbsetup/ns2ir/GNUmakefile.in: Add timeline, sequence, and
      	topography files.
      
      	* tbsetup/ns2ir/node.tcl: Add initial position for nodes and allow
      	them to be attached to "topographys".
      
      	* tbsetup/ns2ir/parse-ns.in: Make a hwtype_class array with a
      	node_type's class.  Make an 'areas' array that holds the
      	buildings where nodes are located.  Make an 'obstacles' table
      	with any obstacles in the building.
      
      	* tbsetup/ns2ir/parse.tcl.in: Move named-args function from
      	tb_compat.tcl to here.  Add reltime-to-secs function that converts
      	time given in a format like "10h2m1s" to a seconds value, used in
      	"$ns at" so its easier to write time values.  Add "K", "Kb", and
      	"Kbps" as possible units for bandwidth (only the lowercase
      	versions were available before).
      
      	* tbsetup/ns2ir/program.tcl: Add "dir" and "timeout" attributes,
      	although they don't go anywhere at the moment.
      
      	* tbsetup/ns2ir/sequence.tcl, tbsetup/ns2ir/timeline.tcl,
      	tbsetup/ns2ir/topography.tcl: Initial versions.
      
      	* tbsetup/ns2ir/sim.tcl.in: Add support for timelines and
      	sequences.  Add 'node-config' method to change the default
      	configuration for nodes produced by the Simulator object.  Send an
      	initial MODIFY event to any trafgen objects so their configuration
      	gets through, even when there are no start/stop events.  Move
      	event parsing to the 'make_event' method.
      
      	* utils/loghole.1, utils/loghole.in: Loghole utility, used for
      	retrieving logs from experimental nodes and creating archives of
      	the logs.
      
      	* xmlrpc/emulabclient.py.in: Escape any strange characters in the
      	output field.
      
      	* xmlrpc/emulabserver.py.in: Add virt_node_startloc to the list of
      	virtual_tables.  Add emulab.vision_config and
      	emulab.obstacle_config methods for getting information pertaining
      	to the robots.  Change the OSID listing to include more fields.
      	Add a "physical" aspect to experiment.info to get information
      	about the physical nodes.  Add parent field to the events in the
      	array returned by eventlist.  Add sshdescription to get extra
      	information needed to log into a vnode.  Add node.statewait so you
      	can wait for nodes to come up.
      bf489797
  10. 08 Dec, 2004 1 commit
  11. 06 Dec, 2004 1 commit
  12. 08 Oct, 2004 1 commit
  13. 17 Sep, 2004 1 commit
  14. 25 Aug, 2004 1 commit
  15. 11 Aug, 2004 1 commit
    • Leigh B. Stoller's avatar
      Add new per-lan table, which currently is just for Mike: · d09d9696
      Leigh B. Stoller authored
      1.269: Add new table to generate a per virt_lan index for use with
             veth vlan tags. This would be so much easier if the virt_lans
             table had been split into virt_lans and virt_lan_members.
             Anyway, this table might someday become the per-lan table, with a
             table of member settings. This would reduce the incredible amount of
             duplicate info in virt_lans!
      
      	CREATE TABLE virt_lan_lans (
      	  pid varchar(12) NOT NULL default '',
      	  eid varchar(32) NOT NULL default '',
      	  idx int(11) NOT NULL auto_increment,
      	  vname varchar(32) NOT NULL default '',
      	  PRIMARY KEY  (pid,eid,idx),
      	  UNIQUE KEY vname (pid,eid,vname)
      	) TYPE=MyISAM;
      
             This arrangement will provide a unique index per virt_lan, within
             each pid,eid. That is, it starts from 1 for each pid,eid. That is
             necessary since the limit is 16 bits, so a global index would
             quickly overflow. The above table is populated with:
      
      	insert into virt_lan_lans (pid, eid, vname)
                  select distinct pid,eid,vname from virt_lans;
      d09d9696
  16. 29 Jul, 2004 1 commit
  17. 15 Jul, 2004 1 commit
    • Leigh B. Stoller's avatar
      Overview: Add Event Groups: · ed964507
      Leigh B. Stoller authored
      	set g1 [new EventGroup $ns]
      	$g1 add  $link0 $link1
      	$ns at 60.0 "$g1 down"
      
      See the new advanced tutorial section on event groups for a better
      example.
      
      Changed tbreport to dump the event groups table when in summary mode.
      At the same time, I changed tbreport to use the recently added
      virt_lans:vnode and ip slots, decprecating virt_nodes:ips in one more
      place. I also changed the web interface to always dump the event and
      event group summaries.
      
      The parser gets a new file (event.tcl), and the "at" method deals with
      event group events by expanding them inline into individual events
      sent to each member. For some agents, this is unavoidable; traffic
      generators get the initial params in the event, so it is not possible
      to send a single event to all members of the group. Same goes for
      program objects, although program objects do default to the initial
      command now, at least on new images.
      
      Changed the event scheduler to load the event groups table. The
      current operation is that the scheduler expands events sent to a
      group, into a set of distinct events sent to each member of the
      group. At some point we proably want to optimize this by telling the
      agents (running on the nodes) what groups they are members of.
      
      Other News: Added a "mustdelay" slot to the virt_lans table so the
      parser can tell assign_wrapper that a link needs to be delayed, say if
      there are events or if the link is red/gred. Previously,
      assign_wrapper tried to figure this out by looking at the event list,
      etc. I have removed that code; see database-migrate for instructions
      on how to initialize this slot in existing experiments. assign_wrapper
      is free to ignore or insert delays anyway, but having the parser do
      this makes more sense.
      
      I also made some "rename" changes to the parser wrt queues and lans
      and links. Not really necessary, but I got sidetracked (for several
      hours!) trying to understand that rename stuff a little better, and
      now I do.
      ed964507
  18. 07 Jul, 2004 1 commit
  19. 04 Jun, 2004 1 commit
    • Leigh B. Stoller's avatar
      Back out previous revision that changed the XML output format to RPC · f8ed2e3b
      Leigh B. Stoller authored
      format instead of plain old well formed XML. Note this change does not
      affect the parser in any way; the parser continues to spit out its own
      brand of XML, and xmlconvert treats that special when it reads it.
      
      The XML I spit out now is better formed and makes more sense when you
      look at it. It also helps that its much faster to generate, and the
      resulting output is much smaller. For example:
      
      	time /usr/testbed/libexec/xmlconvert testbed jail-2440
      	real    2m54.229s
      	user    1m7.492s
      	sys     1m15.927s
      
      	time /usr/testbed/devel/stoller/libexec/xmlconvert testbed jail-2440
      	real    0m14.738s
      	user    0m5.755s
      	sys     0m5.156s
      
      	/tmp/bar*.xml
      	-rw-rw-r--  1 stoller  wheel  15595161 Jun  4 11:16 /tmp/rpc.xml
      	-rw-rw-r--  1 stoller  wheel   5285690 Jun  4 11:17 /tmp/plain.xml
      
      Thats for a big experiment. For a small experiment:
      
      	time /usr/testbed/libexec/xmlconvert testbed jail-416
      	real    0m9.346s
      	user    0m5.083s
      	sys     0m2.675s
      
      	time /usr/testbed/devel/stoller/libexec/xmlconvert testbed jail-416
      	real    0m2.751s
      	user    0m1.377s
      	sys     0m1.183s
      
      	ll /tmp/bar*.xml
      	-rw-rw-r--  1 stoller  wheel  3135518 Jun  4 11:19 /tmp/rpc.xml
      	-rw-rw-r--  1 stoller  wheel  1015745 Jun  4 11:19 /tmp/plain.xml
      
      As you can see, the RPC code is rather non-linear in its performance!
      
      Since this affects Tim and netlab-client, I have added an alternate
      XMLRPC routine called virtual_topology_xml() that returns the new format,
      since Tim will need to transition to this new format (parsing it into a
      Java data structure). The old format is left in the RPC server until that
      is done.
      f8ed2e3b
  20. 29 Apr, 2004 1 commit
    • Leigh B. Stoller's avatar
      Fix for the problem of not setting default values when inserting · 6ab03d5f
      Leigh B. Stoller authored
      experiment table rows after a parse. It used to be that slots that
      were not explicitly given values by the parser would be left as
      whatever they were in the DB. Now I get the default values by doing
      a "describe experiments" query, inserting those values for any slot
      that is not set by the parser.
      6ab03d5f
  21. 23 Apr, 2004 1 commit
    • Leigh B. Stoller's avatar
      I have changed xmlconvert to do the following (note that I have not changed · a3ad59cd
      Leigh B. Stoller authored
      the parser; it still spits out the simplistic xml stuff and xmlconvert
      still reads that):
      
      * When generating an XML representation of an experiment, I read the
        virtual tables and build a *flat* data structure.
      
      	{"settings" => {pid:testbed, eid:foobar ....}.
      	 "nodes"    => [ {vname:nodeA, osid:RHL-STD ... },  ],
      	 "lans"     => [ {vname:link0, member:nodeA ... },  ],
               ...
      
        So, nodes is a list of node hash tables, lans is a list of hash tables,
        etc. All of the other virt_tables are represented as well.  You can find
        everything you want as long as you the names of the fields!
      
      * Once I have the above data structure, I use the XML-RPC lib to turn it
        into an XML document, and spit out the resulting XML string (a very long
        string!) Note that XML-RPC has its own format that allows you to convert
        data structures into XML and back; thats how you issue an RPC from your
        python client to a perl server.
      
      * Our python XMLRPC server basically uses xmlconvert as a backend so it can
        ship the above data structure back to client. We could also allow the
        client to ship the data structure back, and have xmlconvert upload the
        changes back to the DB.
      
        The reason for doing it in xmlconvert as a backend is so that this XML
        stuff is in just one place; whoever reads/writes this stuff has to know a
        lot of internal detail about an experiment (like what things should not
        be exported and what things we should not allow to be imported). I prefer
        to keep that all in one place since its easy to pass XML strings around
        (as long as everyone is talking the same dialect).
      
      Future work is to pick an XML representation that is not tied to
      XML-RPC. Ideally, we want to take a data structure in Perl, spit out
      the XML tree (with a DTD) and be able to reconstitute that in another
      program, using the XML Parser library (based on expat, which is common
      to Perl, Python, C).
      a3ad59cd
  22. 12 Apr, 2004 1 commit
  23. 10 Feb, 2004 1 commit
  24. 02 Feb, 2004 1 commit
  25. 28 Jan, 2004 1 commit
  26. 09 Jan, 2004 1 commit
    • Shashi Guruprasad's avatar
      During a simparse, virt_agents and eventlist need to be cleaned up · 467cb150
      Shashi Guruprasad authored
      but not other virt_* tables. Thats coz the above two tables are not
      truly virtual since they contain the vnode to which the event should
      be sent to. My previous patch has been to use replace instead of
      insert. Unfortunately, the tables get messed up with the same agent
      having multiple entries where some of the entries were left behind
      from the previous swapin.
      467cb150
  27. 29 Dec, 2003 1 commit
    • Shashi Guruprasad's avatar
      If xmlconvert is called after an nseparse, we need to use replace coz · 08a40103
      Shashi Guruprasad authored
      some of the tables such as virt_agents and eventlist are not truly
      virtual tables. That is coz they contain the vnode field which is
      the same as the vname field in the reserved table. For simulated
      nodes, the mapping may change across swapins and the event may have
      to be delivered to a different simhost. Of course this problem
      isn't there for jail vnodes or pc vnodes.
      08a40103
  28. 27 Dec, 2003 1 commit
    • Shashi Guruprasad's avatar
      nseconfigs table is used both as a virtual table and a physical table. · 5f27af03
      Shashi Guruprasad authored
      When xmlconvert is called during nseparse, we need to clean up the
      physical part so that rows for the new mapping can be added. I have
      added this. While I say that there is a physical part, essentially
      the vname column has the same value as the vname column in the
      reserved table for the PC such as 'simhost-0', 'simhost-1' etc.
      5f27af03
  29. 18 Dec, 2003 1 commit
    • Leigh B. Stoller's avatar
      First try at solving the problem of validating user input for the · 8dbead16
      Leigh B. Stoller authored
      zillions of DB fields that we have to set. My solution was to add a
      meta table that describes what is a legal value for each table/slot
      for which we take from user input. The table looks like this right
      now, but is likely to adapt as we get more experience with this
      approach (or it might get tossed if it turns out to be a pain in the
      ass!).
      
      	CREATE TABLE table_regex (
      	  table_name varchar(64) NOT NULL default '',
      	  column_name varchar(64) NOT NULL default '',
      	  column_type enum('text','int','float') default NULL,
      	  check_type enum('regex','function','redirect') default NULL,
      	  check tinytext NOT NULL,
      	  min int(11) NOT NULL default '0',
      	  max int(11) NOT NULL default '0',
      	  comment tinytext,
      	  UNIQUE KEY table_name (table_name,column_name)
      	) TYPE=MyISAM;
      
      Entries in this table look like this:
      
      	('virt_nodes','vname','text','regex','^[-\\w]+$',1,32,NULL);
      
      Which says that the vname slot of the virt_nodes table (which we trust the
      user to give us in some form) is a text field to be checked with the given
      regex (perlre of course), and that the min/max length of the text field is
      1 and 32 chars respectively.
      
      Now, you wouldn't want to write the same regex over and over, and since we
      use the same fields in many tables (like pid, eid, vname, etc) there is an
      option to redirect to another entry (recursively). So, for "PID" I do this:
      
              ('eventlist','pid','text','redirect','projects:pid',0,0,NULL);
      
      which redirects to:
      
      	('projects','pid','text','regex','^[a-zA-Z][-\\w]+$',2,12,NULL);
      
      And, for many fields you just want to describe generically what could go
      into it. For that I have defined some default fields. For example, a user
      description:
      
              ('experiment,'usr_name','text','redirect','default:tinytext',0,0,NULL);
      
      which redirects to:
      
      	('default','tinytext','text','regex','^[\\040-\\176]*$',0,256,NULL);
      
      and this says that a tinytext (in our little corner of the database
      universe) field can have printable characters (but not a newline), and
      since its a tinytext field, its maxlen is 256 chars.
      
      You also have integer fields, but these are little more irksome in the
      details.
      
      	('default','tinyint,'int,'regex','^[\\d]+$',-128,127,NULL);
      
      and you would use this anyplace you do not care about the min/max values
      being something specific in the tinyint range. The range for a float is of
      course stated as an integer, and thats kinda bogus, but we do not have many
      floats, and they generally do not take on specific values anyway.
      
      A note about the min/max fields and redirecting. If the initial entry has
      non-zero min/max fields, those are the min mac fields used. Otherwise they
      come from the default. So for example, you can do this:
      
          ('experiments','mem_usage','int','redirect','default:tinyint',0,5,NULL);
      
      So, you can redirect to the standard "tinyint" regular expression, but you
      still get to define min/max for the specific field.
      
      Isn't this is really neat and really obtuse too? Sure, you can say it.
      
      Anyway, xmlconvert now sends all of its input through these checks (its
      all wrapped up in library calls), and if a slot does not have an entry, it
      throws an error so that we are forced to define entries for new slots as we
      add them.
      
      In the web page, I have changed all of the public pages (login, join
      project, new project, and a couple of others) to also use these checks.
      As with the perl code, its all wrapped up in a library. Lots more code
      needs to be changed of course, but this is a start.
      8dbead16
  30. 15 Dec, 2003 1 commit
    • Shashi Guruprasad's avatar
      Distributed NSE changes. In other words, simulation resources are · d266bd71
      Shashi Guruprasad authored
      now mapped to more than one PC if required. The simnode_capacity
      column in the node_types table determines how many sim nodes can
      be packed on one PC. The packing factor can also be controlled via
      tb-set-colocate-factor to be smaller than simnode_capacity.
      
      - No frontend code changes. To summarize:
        $ns make-simulated {
          ...
        }
        is still the easy way to put a whole bunch of Tcl code to be
        in simulation.
        One unrelated fix in the frontend code is to fix the
        xmlencode() function which prior to this would knock off
        newlines from columns in the XML output. This affected
        nseconfigs since it is one of the few columns with embedded
        newlines. Also changed the event type and event object type
        in traffic.tcl from TRAFGEN/MODIFY to NSE/NSEEVENT.
      
      - More Tcl code in a new directory tbsetup/nseparse
        -> Runs on ops similar to the main parser. This is invoked
           from assign_wrapper in the end if there are simnodes
        -> Partitions the Tcl code into multiple Tcl specifications
           and updates the nseconfigs table via xmlconvert
        -> Comes with a lot of caveats. Arbitrary Tcl code such as user
           specified objects or procedures will not be re-generated. For
           example, if a user wanted a procedure to be included in Tcl
           code for all partitions, there is no way for code in nseparse
           to do that. Besides that, it needs to be tested more thoroughly.
      
      - xmlconvert has a new option -s. When invoked with this option,
        the experiments table is not allowed to be modified. Also,
        virtual tables are just updated (as opposed to deleting
        all rows in the first invocation before inserting new rows)
      
      - nse.patch has all the IP address related changes committed in
        iversion 1.11 + 2 other changes. 1) MTU discovery support in
        the ICMP agent 2) "$ns rlink" mechanism for sim node to real
        node links
      
      - nseinput.tcl includes several client side changes to add IP
        routes in NSE and the kernel routing table for packets crossing
        pnodes. Also made the parsing of tmcc command output more robust
        to new changes. Other client side changes in libsetup.pm and other
        scripts to run nse, are also in this commit
      
      - Besides the expected changes in assign_wrapper for simulated nodes,
        the interfaces and veth_interfaces tables are updated with
        routing table identifiers (rtabid). The tmcd changes are already
        committed. This field is used only by sim hosts on the client side.
        Of course, they can be used by jails as well if desired.
      d266bd71
  31. 02 Dec, 2003 1 commit
  32. 12 Nov, 2003 1 commit
    • Leigh B. Stoller's avatar
      Changes/Fixes to allow Shashi to spit NS files into the DB from the · fb2a8122
      Leigh B. Stoller authored
      parser:
      
      * Change xmlconvert to use XML::Parser instead of the ad-hoc stuff I
        wrote. This was easier then trying to deal with multiline fields or
        all the special chars. XML::Parser does all that for me. Also make
        sure all fields are either properly quoted (with DBQuoteMeta) or
        regex'ed before we stick them into the DB. For the future, we need
        to add table/slot specific regular expressions.
      
      * Also add Shashi's fixes to the parser; properly escape XML special
        characters, as well as cleanup of my original spitxml code.
      fb2a8122
  33. 05 Nov, 2003 1 commit
    • Leigh B. Stoller's avatar
      Frontend and parser portion of two event system changes: · 091a0b62
      Leigh B. Stoller authored
      * Generate a shared secret key for the event system. This key is
        stored into the DB, and passed to the node via tmcd. It is also
        stashed into a file in the experiment directory (can be accessed
        only by the project/group members). The key is used to attach a
        HMAC (hashed message authentication) to each event, which is checked
        by the receivers to ensure that the event is not bogus. More details
        on this later when I commit the event library/client changes.
      
      * Added "virt_programs" table to store info about each program object
        defined by the user. The intent is to no longer send the command
        string in the event, but to fix it in the DB, and transfer it via
        tmcd. This removes our "remote execution facility" which was always
        a bad idea (we have ssh for that, and that is a lot more secure then
        the event system!).
      
        Note that for the time being we need to continue send the command in
        the event because of old images, but the new images will now ignore
        that part of the event.
      091a0b62
  34. 26 Sep, 2003 1 commit
  35. 11 Sep, 2003 1 commit
  36. 05 Aug, 2003 1 commit
    • Leigh B. Stoller's avatar
      The rest of the sync server additions: · 212cc781
      Leigh B. Stoller authored
      * Parser: Added new tb command to set the name of the sync server:
      
      	tb-set-sync-server <node>
      
        This initializes the sync_server slot of the experiment entry to the
        *vname* of the node that should run the sync server for that
        experiment. In other words, the sync server is per-experiment, runs
        on a node in the experiment, and the user gets to chose which node
        it runs on.
      
      * tmcd and client side setup. Added new syncserver command which
        returns the name of the syncserver and whether the requesting node
        is the lucky one to run the daemon:
      
          SYNCSERVER SERVER='nodeG.syncserver.testbed.emulab.net' ISSERVER=1
      
        The name of the syncserver is written to /var/emulab/boot/syncserver
        on the nodes so that clients can easily figure out where the server
        is.
      
        Aside: The ready bits are now ignored (no DB accesses are made) for
        virtual nodes; they are forced to use the new sync server.
      
      * New os/syncd directory containing the daemon and the client. The
        daemon is pretty simple. It waits for TCP (and UDP, although that
        path is not complete yet) connections, and reads in a little
        structure that gives the name of the "barrier" to wait for, and an
        optional count of clients in the group (this would be used by the
        "master" who initializes barriers for clients). The socket is saved
        (no reply is made, so the client is blocked) until the count reaches
        zero. Then all clients are released by writting back to the
        sockets, and the sockets are closed. Obviously, the number of
        clients is limited by the numbed of FDs (open sockets), hence the
        need for a UDP variant, but that will take more work.
      
        The client has a simple command line interface:
      
          usage: emulab-sync [options]
          -n <name>         Optional barrier name; must be less than 64 bytes long
          -d                Turn on debugging
          -s server         Specify a sync server to connect to
          -p portnum        Specify a port number to connect to
          -i count          Initialize named barrier to count waiters
          -u                Use UDP instead of TCP
      
          The client figures out the server by looking for the file created
          above by libsetup (/var/emulab/boot/syncserver). If you do not
          specify a barrier "name", it uses an internal default. Yes, the
          server can handle multiple barriers (differently named of course)
          at once (non-overlapping clients obviously).
      
          Clients can wait before a barrier in "initialized." The count on
          the barrier just goes negative until someone initializes the
          barrier using the -i option, which increments the count by the
          count. Therefore, the master does not have to arrange to get there
          "first." As an example, consider a master and one client:
      
      	nodeA> /usr/local/etc/emulab/emulab-sync -n mybarrier
      	nodeB> /usr/local/etc/emulab/emulab-sync -n mybarrier -i 1
      
          Node A waits until Node B initializes the barrier (gives it a
          count).  The count is the number of *waiters*, not including the
          master. The master is also blocked until all of the waiters have
          checked in.
      
          I have not made an provision for timeouts or crashed clients. Lets
          see how it goes.
      212cc781
  37. 15 Jul, 2003 1 commit
  38. 02 Jul, 2003 1 commit
    • Leigh B. Stoller's avatar
      Hack for dealing with user output from the parser (puts). Since I spit · 10673342
      Leigh B. Stoller authored
      the XML to stdout, user calls to puts confuse the converter, and get
      lost so the user never sees them (before, they would end up in the
      start/preload log). The converter now eats up lines till it sees a
      valid XML header string, printing those lines to stdout so they end up
      in the log like before.
      
      Need to deal with this in a better manner at some point. I did not
      want to write the XML to temporary file. Not sure why, but I thought
      that was the right approach, to keep the user from getting his grubby
      little hands on it. I'm thinking now thats silly, and I'll eventually
      fix things up (since it requires more TCL hacking, and I don't have
      the energy for that).
      10673342
  39. 30 Jun, 2003 1 commit
    • Leigh B. Stoller's avatar
      Make the new parser live on mini. New parser ssh'es over to ops to · 2202fc5a
      Leigh B. Stoller authored
      do the actual parse. The parser now spits out XML instead of DB
      queries, and the wrapper on boss converts that to DB insertions after
      verification. There are some makefile changes as well to install the
      new parser on ops via NFS, since otherwise the parser could
      intolerably out of date on ops!
      2202fc5a
  40. 26 Jun, 2003 1 commit
    • Leigh B. Stoller's avatar
      New script. Convert experiment to a "well formed" XML, and back again. · f7fc7ff3
      Leigh B. Stoller authored
      The conversion from XML to an experiment representation updates the
      DB, although the experiment has to exist, and only certain fields can
      be updated. For the rest of the virtual toplogy, the old toplogy is
      deleted, and the new one installed. Use with caution. This is intended
      to be hooked to the parser, and later to the vis tool.
      f7fc7ff3