1. 14 Aug, 2006 1 commit
    • Leigh Stoller's avatar
      Checkpoint my dynamic event stuff, crude as it is. The idea for this first · 9d021a07
      Leigh Stoller authored
      draft is that the user will at the end of an experiment run, log into one
      of his nodes and perform some analysis which is intended to be repeated at
      the end of the next run, and in future instantiations of the template.
      
      A new table called experiment_template_events holds the dynamic events for
      the template. Right now I am supporting just program events, but it will be
      easy to support arbitrary events later. As an absurd example:
      
      	node6> /usr/local/bin/template_analyze ~/data_analyze arg arg ...
      
      The user is currently responsible for making sure the output goes into a
      file in the archive. I plan to make the template_analyze wrapper handle
      that automatically later, but for now what you really want is to invoke a
      script that encapsulates that, redirecting output to $ARCHIVE (this
      variable is installed in the environment template_analyze.
      
      The wrapper script will save the current time, and then run the program.
      If the program terminates with a zero exit status, it will ssh over to ops
      and invoke an xmlrpc routine to tell boss to add a program event to both
      the eventlist for the current instance, and to the template_eventlist for
      future instances. The time of the event is the relative start time that was
      saved above (remember, each experiment run replays the event stream from
      time zero).
      
      For the future, we want to allow this to be done on ops as well, but
      that will take more infrastructure, to run "program agents" on ops.
      
      It would be nice to install the ssl xmlrpc client side on our images so
      that we do not have to ssh to ops to invoke the client.
      9d021a07
  2. 09 Aug, 2006 1 commit
  3. 03 Aug, 2006 1 commit
    • Leigh Stoller's avatar
      Support for capturing the trace data that is stored in the pcal files · 4ce9c421
      Leigh Stoller authored
      into per-experiment databases on ops. Additional support for reconsituting
      those databases back into temporary databases on ops, for post processing.
      
      * This revision relies on the "snort" port (/usr/ports/security/snort)
        to read the pcap files and load them into a database. The schema is
        probably not ideal, but its better then nothing. See the file
        ops:/usr/local/share/examples/snort/create_mysql for the schema.
      
      * For simplicity, I have hooked into loghole, which already had all
        the code for downloading the trace data. I added some new methods to
        the XMLRPC server for loghole to use, to get the users DB password
        and the name of the per-experiment database. There is a new slot in
        the traces table that indicates that the trace should be snorted to
        its DB. In case you forgot, at the end of a run or when the instance
        is swapped out, loghole is run to download the trace data.
      
      * For reconsituting, there are lots of additions to opsdb_control and
        opsdb_control.proxy to create "temporary" databases and load them
        from a dump file that is stored in the archive. I've added a button
        to the Template Record page, inappropriately called "Analyze" since
        right now all it does is reconsitute the trace data into a DB on
        ops.
      
        Currently, the only indication of what has been done (the name of
        the DBs created on ops) is the log email that the user gets. A
        future project is tell the user this info in the web interface.
      
      * To turn on database capturing of trace data, do this in your NS
        file:
      
      	set link0 ...
      	$link0 trace
      	$link0 trace_snaplen 128
      	$link0 trace_db 1
      
         the increase in snaplen is optional, but a good idea if you want
         snort to undertand more then just ip headers.
      
      * Also some changes to the parser to allow plain experiments to take
        advantage of all this stuff. To simple get yourself a per-experiment
        DB, put this in your NS file:
      
      	tb-set-dpdb 1
      
        however, anytime you turn trace_db on for a link or lan, you
        automatically get a per-experiment DB.
      
      * To capture the trace data to the DB, you can run loghole by hand:
      
      	loghole sync -s
      
        the -s option turns on the "post-process" phase of loghole.
      4ce9c421
  4. 03 Jul, 2006 1 commit
    • Mike Hibler's avatar
      Framework for supporting 802.1q tagged VLANs as a form of multiplexed link. · 3f1c15e2
      Mike Hibler authored
      Actually, most of the changes here were just to generalize the "virtual
      interface" state in the DB.  Other than the client-side scripts, there
      is very little specific here specific to tagged VLANs.
      
      In fact, you cannot specify "vlan" as a type yet as we haven't done the
      snmpit support for setting up the switches.
      
      For more info see bas:~mike/flux/doc/testbed-virtinterfaces.txt, which I
      will integrate into the knowledge base and the Emulab doc at some point.
      3f1c15e2
  5. 27 Jun, 2006 1 commit
  6. 07 Jun, 2006 1 commit
  7. 04 May, 2006 1 commit
  8. 02 May, 2006 2 commits
  9. 23 Apr, 2006 1 commit
  10. 15 Mar, 2006 2 commits
  11. 16 Feb, 2006 2 commits
    • Kevin Atkinson's avatar
      · 9cef1d8f
      Kevin Atkinson authored
      In libxmlrpc.pm the function CallMethod throws away some information that
      I needed.  Thus I created a new function CallMethod0 which doesn't and
      have CallMethod use CallMethod0.
      9cef1d8f
    • Leigh Stoller's avatar
      Minor fixes to make Python 2.4 happy. · b4215f71
      Leigh Stoller authored
      b4215f71
  12. 09 Feb, 2006 2 commits
  13. 26 Jan, 2006 1 commit
    • Timothy Stack's avatar
      · 4b938465
      Timothy Stack authored
      Some pelab/plab event system hacks.
      
      	* event/lib/event.h, event/lib/event.c: Add event_subscribe_auth
      	which lets you specify whether any authentication should be done
      	for events received through this subscription.
      
      	* event/sched/event-sched.c: Handle EVPROXY objects.  Add a
      	separate subscription for EVPROXY UPDATE events for each plab
      	pnode (which might be too many...).  Also, need to update the EXPT
      	field for events received through a noauth subscription so the
      	proxies can figure out which experiments are active.
      
      	* lib/libtb/tbdefs.h, lib/libtb/tbdefs.c: Add UPDATE event defs.
      
      	* xmlrpc/emulabserver.py.in: Inject __plab_setup and
      	__plab_teardown timelines into the eventlist when an experiment
      	has plab nodes.  The __plab_setup timeline sends EVPROXY UPDATE
      	events to each physical node while the __plab_teardown sends
      	EVPROXY CLEAR events.  The __plab_setup timeline is run when the
      	scheduler starts up, the __plab_teardown isn't run automatically
      	yet.
      4b938465
  14. 02 Jan, 2006 1 commit
  15. 29 Dec, 2005 1 commit
  16. 22 Dec, 2005 1 commit
  17. 13 Dec, 2005 1 commit
  18. 01 Dec, 2005 1 commit
  19. 28 Nov, 2005 1 commit
    • Timothy Stack's avatar
      · 19dbd0fd
      Timothy Stack authored
      Make the netlab client applet available to locals.
      
      	* configure, configure.in: Add xmlrpcpipe.php3.
      
      	* xmlrpc/emulabserver.py.in: Add missing virtual_tables.  Add
      	getareas call to get the list of robot areas.  Add node.getlist
      	and node.typeinfo methods for getting information about the nodes.
      	Add a "nic" argument to node.available to get the count of
      	wireless nodes.  Add "exclude" argument to
      	experiment.virtual_topology so we don't have to download the
      	massive route table, also delete the 'pid'/'eid' fields for the
      	same reason.  Don't return string output of virtual_topology, it's
      	huge.  Return some more info in experiment.getlist().
      
      	* www/GNUmakefile.in: Add xmlrpcpipe.php3.
      
      	* www/beginexp_form.php3, www/modifyexp.php3: Add links to the
      	client gui.
      
      	* www/netlab-client.jar: The client binary.
      19dbd0fd
  20. 04 Nov, 2005 1 commit
  21. 08 Sep, 2005 1 commit
  22. 30 Aug, 2005 2 commits
  23. 26 Jul, 2005 1 commit
  24. 19 Jul, 2005 1 commit
  25. 24 Jun, 2005 2 commits
    • Timothy Stack's avatar
      · 71901cf5
      Timothy Stack authored
      Add a minor feature to loghole to support network link trace files.
      
      	* utils/loghole.in, utils/loghole.1: Create a directory for traced
      	network links that contains symbolic links to the pcap files.
      
      	* www/tutorial/advanced.html: Fix the link to the loghole doc.
      
      	* xmlrpc/emulabserver.py.in: Add a 'traces' aspect to the
      	experiment.info method (used by loghole).
      71901cf5
    • Timothy Stack's avatar
      0d68d00f
  26. 21 Jun, 2005 1 commit
  27. 17 Jun, 2005 1 commit
    • Timothy Stack's avatar
      · 41d4c6d4
      Timothy Stack authored
      Event system documentation and tweaks:
      
      	* event/sched/event-sched.c: The snapshot event got left out of
      	the list of events that send back a complete event.  If a top
      	level sequence stops with an error, automatically send a report.
      
      	* event/sched/group-agent.c, event/sched/timeline-agent.c: When
      	handling a complete event, return true if the event was consumed.
      
      	* event/sched/node-agent.cc: Clean the "/local/logs" loghole on
      	the node before taking a snapshot of the disk image.
      
      	* event/sched/simulator-agent.cc, event/sched/simulator-agent.h:
      	Made send_report non-static so it can be called when a sequence
      	errors out.  Actually check the clear flag when dumping report
      	data.
      
      	* lib/libtb/tbdefs.c: Add a comment about updating the event
      	scheduler when a new object type is added.
      
      	* tbsetup/ns2ir/sim.tcl.in: Check the OSIDs passed to the "reload"
      	and "snapshot-to" events.
      
      	* utils/loghole.1: Mention early and often the directories that
      	gets sync'd.  Document the 'change' action and new options for
      	'clean'.
      
      	* utils/loghole.in: Use rsync's '--include-from' option so we
      	don't have to run rsync separately for each directory.  Add a
      	'change' action that lets people mess with the metadata of an
      	existing archive.  Make the 'clean' action also remove log files
      	on the nodes.  Only rsync node's whose OS's support ssh.  Grab
      	logs from delay nodes.
      
      	* www/tutorial/advanced.html: Update the program agent examples to
      	use the new form.
      
      	* www/tutorial/eventsystem.html: First cut at an event system
      	reference manual that includes sequences, timelines, and other
      	new features.
      
      	* www/tutorial/loghole.html: Just a pointer to the man page on ops
      	for now.
      
      	* www/tutorial/nscommands.html: Add a section about capturing
      	parameters from the 'opt' array.
      
      	* xmlrpc/emulabserver.py.in: Add an osid.info() method that
      	returns information about a single OSID.
      41d4c6d4
  28. 25 May, 2005 1 commit
  29. 24 May, 2005 1 commit
  30. 16 May, 2005 1 commit
  31. 27 Apr, 2005 1 commit
    • Leigh Stoller's avatar
      A set of changes to allow the newnode path (MFS and adding nodes) to · 94d97303
      Leigh Stoller authored
      work inside ElabInElab.
      
      The crux of this is to emulate what switchmac does when incorporating nodes
      into the testbed. Rather then using snmpit to ask the switches for MAC info,
      we can just ask outer emulab via the proxy to get that same info from the
      DB. Note, there are no changes to the newnode MFS; it will boot and happily
      report it after launching teachswitch, but those packets are harmless to the situation.
      
      Another wrinkle is that we need the outer emulab to tell us which
      interfaces are control and which are experimental, since inside there
      is no real way to determine that. I did this by overloading the final
      field in the return from switchmac, and using that to override the
      "role" setting that utils/newnode would normally choose on its own.
      
      There is another new routine in the XMLRPC server that is intended to
      be used for bypassing the newnode path. This is not hooked into
      anything yet, but the intent is that rather then using the web
      interface to "add" nodes, we just return everything needed to seed the
      new_nodes and new_interfaces table, and then run newnodes directly.
      94d97303
  32. 04 Apr, 2005 1 commit
    • Timothy Stack's avatar
      · 463ee6b1
      Timothy Stack authored
      Mote and robot related stuff.  The main thing is the addition of relay
      capabilities to capture and related things.
      
      	* GNUmakefile.in: Add the capture and tip subdirectories to the
      	client and client-install targets.
      
      	* configure, configure.in, config.h.in: Detect srandomdev() for
      	capture and add "mote/newmote" script.
      
      	* capture/GNUmakefile.in, capture/capture.c: Add "relay"
      	capabilities to capture.
      
      	* capture/capquery.c: Query the capserver for the relay receiver's
      	port number.
      
      	* capture/capserver.c: Small hack to return the port number
              for a node.
      
      	* db/libdb.pm.in, db/xmlconvert.in: Add virt_tiptunnels table.
      
      	* event/program-agent/program-agent.c: Change log file names to
      	something a little more user-friendly.  Add a "MODIFY" event
      	handler that lets the user set agent attributes (command, tag,
      	timeout) without having to run a program.
      
      	* event/sched/GNUmakefile.in, event/sched/console-agent.cc,
      	event/sched/console-agent.h, event/sched/event-sched.c: Add
      	console agents that can be used to snapshot a section of the
      	capture log file.
      
      	* event/sched/node-agent.cc: Some minor cleanup.
      
      	* event/sched/simulator-agent.cc, event/sched/simulator-agent.h:
      	Add the config data to the report mail.  Add a "RESET" event
      	handler that runs "loghole clean".  Save the report mail in a file
      	so it gets archived with the rest of the logs.
      
      	* lib/libtb/tbdefs.h: Add CONSOLE object type.
      
      	* mote/GNUmakefile.in, mote/newmote: Add newmote script, just a
      	quick hack to add motes to the DB.
      
      	* mote/tbuisp.in: Add another backend for loading motes through
      	their relay capture server.
      
      	* robots/mtp/mtp_dump.c: Dump the min/max values for x and y,
      	handy for figuring out the bounds of the camera.
      
      	* sql/database-fill.sql: Change the RELOAD-MOTE/SHUTDOWN ->
      	ALWAYSUP/SHUTDOWN mode transition to ALWAYSUP/ISUP since stated
      	doesn't seem to run triggers after a state change by a mode
      	transition.
      
      	* tbsetup/tbreport.in: Change the ordering of the eventlist so it
      	displays event-sequences appropriately.
      
      	* tbsetup/ns2ir/GNUmakefile.in, tbsetup/ns2ir/console.tcl,
      	tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/parse.tcl.in,
      	tbsetup/ns2ir/sim.tcl.in: Add a "console" agent that represents
      	the serial console for a node.
      
      	* tbsetup/ns2ir/sequence.tcl: Add an "append" method so it is
      	easier to build sequences dynamically.
      
      	* tbsetup/ns2ir/topography.tcl: Make checkdest available to
      	regular users.
      
      	* tip/GNUmakefile.in, tip/tiptunnel.c: Add support for uploading a
      	file to a relay version of capture and exporting the end
      	connection as a pty.
      
      	* tmcd/decls.h, tmcd/common/libsetup.pm: Bump version number since
      	the dosubnodelist change is not backwards compatible.
      
      	* tmcd/tmcd.c: Make dosubnodelist and dosubconfig callable even
      	when a node isn't allocated.  Add dotiptunnels command that
      	returns which serial consoles are to be mounted on a node.  Add
      	mote version of subconfig that returns information needed to
      	startup the relay version of capture.
      
      	* tmcd/common/bootsubnodes: For motes, startup the relay version
      	of capture (XXX stargate specific).
      
      	* tmcd/common/libsetup.pm, tmcd/common/libtmcc.pm,
      	tmcd/common/config/rc.config, tmcd/common/config/rc.tiptunnels:
      	Client side changes for mounting another nodes serial line.
      
      	* tmcd/common/rc.bootsetup: Always boot the subnodes, even when
      	free.  This is used for motes since their capture needs to be up
      	for reloading at the time.
      
      	* tmcd/linux/ixpboot: Shuffle some code around so the script
      	doesn't fail if the ixp isn't allocated.
      
      	* utils/loghole.in: Add "digest.out" and "report.mail" as global
      	logs to be saved in archives and display the "report.mail" file
      	when showing a loghole archive.
      
      	* xmlrpc/emulabserver.py.in: Scrub more of the return values to
      	get rid of "None"s.
      463ee6b1
  33. 22 Mar, 2005 1 commit
  34. 08 Mar, 2005 1 commit
    • Timothy Stack's avatar
      · b3c23e5d
      Timothy Stack authored
      Fix a couple of things I missed in my last checkin.
      
      	* event/program-agent/program-agent.c: Setup of the user
      	environment variables wasn't ordered correctly.
      
      	* event/sched/event-sched.c, event/sched/rpc.h,
      	event/sched/rpc.cc: Setup the expected user environment variables,
      	just like in program-agent.
      
      	* event/sched/simulator-agent.cc: Rename the old feedback data so
      	we don't lose it in case the vnode remapping modify fails.
      
      	* xmlrpc/emulabserver.py.in: Add virt_user_environment table data
      	to the result of the experiment.metadata method, so the
      	event-scheduler can get it.
      b3c23e5d