1. 26 Sep, 2006 1 commit
  2. 20 Sep, 2006 1 commit
    • Leigh B. Stoller's avatar
      By popular demand, you can now force a swap modify to be done when · b9161642
      Leigh B. Stoller authored
      doing a Start Run. On the web page, there is a new checkbox, and
      on ops, template_startrun takes a new -m option.
      
      Caveat: You cannot specify a new NS file, yet. The original file is
      reparsed, and the idea is that a change in the template parameters
      will result in a change to the topology. I will add the ability to
      specify a new NS file in the next revision of this change.
      
      If you really really want to change the NS file, go to
      /proj/$pid/exp/$eid/archive/nsdata and edit nsfile.ns ...
      
      In addtion, DATASTORE is now defined while parsing the NS file. This
      turned to be quite the headache!
      b9161642
  3. 15 Sep, 2006 1 commit
  4. 13 Sep, 2006 1 commit
    • Leigh B. Stoller's avatar
      Changes to allow waiting for a specific completion event, which is needed · 5c564bc8
      Leigh B. Stoller authored
      to make stoprun waiting work correctly.
      
      When tevc is invoked with the -w (wait for completion) option, tevc
      generates a token to put into the notification. The event scheduler will
      not generate a new token if there is already on in the notification, but
      instead pass it on.
      
      For the specific case of stoprun, the simulator agent has to pass that
      token along to boss and template_exprun, which generates the completion
      event (for reasons discussed in prior commit message).
      5c564bc8
  5. 12 Sep, 2006 1 commit
    • Leigh B. Stoller's avatar
      This started out as a simple little hack to add a StopRun "ns" event, but · cbdc4178
      Leigh B. Stoller authored
      it got more complicated as it progressed.
      
      The bulk of the change was changing template_exprun so that it can take a
      pid/eid as an alternative to eid/guid. This is a big convenience since its
      easy to find the template from a running experiment, and it makes it
      possible to invoke from the event scheduler, which has never heard of a
      template before (and its not something I wanted to teach it about).  Its
      also easier on users.
      
      Anyway, back to the stoprun event. You can now do this:
      
      	$ns at 100 "$ns stoprun"
      or
      	tevc -e pid/eid now ns stoprun
      
      You can add the -w option to wait for the completion event that is sent,
      but this brings me to the glaring problems with this whole thing.
      
      * First, the scheduler has to fire off the stoprun in the background,
        since if it waits, we get deadlock. Why? Cause the implementation of
        stoprun uses the event system (SNAPSHOT event, other things), and if
        the scheduler is sitting and waiting, nothing happens.
      
        Okay, the solution to this was to generate a COMPLETION event from
        template_exprun once the stop operation is complete. This brings me
        to the second problem ...
      
      * Worse, is that the "ns" events that are sent to implement stoprun (like
        snapshot) send their own completion events, and that confuses anyone
        waiting on the original stoprun event (it returns early).
      
        So what to do about this? There is a "token" field in the completion
        event structure, which I presume is to allow you to match things up.  But
        there is no way to set this token using tevc (and then wait for it), and
        besides, the event scheduler makes them up anyway and sticks them into
        the event. So, the seed of a fix are already germinating in my mind, but
        I wanted to get this commit in so that Mike would have fun reading this
        commit log.
      cbdc4178
  6. 10 Sep, 2006 1 commit
    • Leigh B. Stoller's avatar
      The bulk of this commit adds the ability to run the program agent on ops · e8bb6bca
      Leigh B. Stoller authored
      so that users can schedule program events to run there. For example:
      
      	set myprog [new Program $ns]
      	$myprog set node "ops"
      	$myprog set command "/usr/bin/env >& /tmp/foo"
      
      	$ns at 10 "$myprog start"
      or
      	tevc -e pid/eid now myprog start
      
      Since the program agent cannot talk to tmcd from ops, there are new
      routines to create the config files that the program agent uses, in
      the expertment tbdata directory.
      
      I also rewrote the eventsys.proxy script that starts the event
      scheduler on ops; I rolled the startup of the program agent into this
      script, via new -a option which is passed over from boss when an ops
      program agent is detected in the virt topology. This keep the number
      of new processes on ops to a small number.
      
      Also part of the above rewrite is that we now catch when event
      scheduler (or the program agent) exits abnormally, sending email to
      tbops and the swapper of the experiment. We have been seeing abnormal
      exits of the scheduler and it would good to detect and see if we can
      figure out what is going wrong.
      
      Other small bug fixes in experiment run.
      e8bb6bca
  7. 05 Sep, 2006 1 commit
    • Leigh B. Stoller's avatar
      A bunch of template changes resulting from meetings last week. · 087dbfff
      Leigh B. Stoller authored
      * Add XMLRPC interface for template swapin,stoprun,startrun,swapout and
        add the appropriate wrappers to the script_wrapper on ops.
      
      * Allow parameter descriptions in NS files. This is probably not in its
        final form since its a bit confusing as to what has priority; something
        in the NS file or a metadata item. Anyway, you can do this in your NS
        file:
      
      	$ns define-template-parameter GUID "0/0" "The GUID to be analyzed"
      
        The rules are currently that the NS file description has priority and
        is copied to child templates, unless the user has modified a description
        via the web interface, in which case the NS file description is ignored.
        I know, sounds awful, but for the most part people are going to use the
        NS file anyway.
      
      * Add "clear" option when starting a new experiment run; the per
        experiment DB at the logholes are cleared. Note that this is *not* the
        default behaviour; you have to either check the checkbox on the web form
        or use the -c option to the script wrapper, or clear=yes if talking
        directly to the XMLRPC server.
      
      * Fix up how email is generated for template_swapin and template_create,
        so that Kevin can debug tblog/tbreport stuff, but also so that we maintain
        mail logs as before. I have made some improvements to libaudit so as to
        centralize the mail goo, and avoid duplicating all that stuff.
      
      * Minor fixes to the program agent so that the new environment strings are
        sent before the program agent exits and reloads them!
      
      * Other minor little things.
      087dbfff
  8. 31 Aug, 2006 1 commit
    • Leigh B. Stoller's avatar
      * Finish up the Commit From Template support. · 3327ba01
      Leigh B. Stoller authored
      * Export the above via the XMLRPC interface and add a wrapper function
        to the script_wrapper. This allows you do to this on ops:
      
      	cd /proj/testbed/templates/10023/1
              Edit some files
              template_commit
      
        Which creates a new template, using the current directory to infer
        the template. Otherwise, provide the template GUID on the command line.
        Hmm, maybe this should be called template_modify? Either way, the
        name does not quite match
      
      * Export template_export via the XMLRPC wrapper. This allows you to
        export a template (instance) record from the command line on ops.
      
      
      	cd /proj/testbed/templates/10023/1
              template_export -i 12
              Exported to /proj/testbed/export/10000/3/12
      
        Which exports the template record for instance number 12. Again, the
        GUID is infered, but you can specify one on the command line. The export
        directory is printed so you know where it went. Note that export does
        *not* populate a DB on ops with the old DB data.
      3327ba01
  9. 14 Aug, 2006 1 commit
    • Leigh B. Stoller's avatar
      Checkpoint my dynamic event stuff, crude as it is. The idea for this first · 9d021a07
      Leigh B. Stoller authored
      draft is that the user will at the end of an experiment run, log into one
      of his nodes and perform some analysis which is intended to be repeated at
      the end of the next run, and in future instantiations of the template.
      
      A new table called experiment_template_events holds the dynamic events for
      the template. Right now I am supporting just program events, but it will be
      easy to support arbitrary events later. As an absurd example:
      
      	node6> /usr/local/bin/template_analyze ~/data_analyze arg arg ...
      
      The user is currently responsible for making sure the output goes into a
      file in the archive. I plan to make the template_analyze wrapper handle
      that automatically later, but for now what you really want is to invoke a
      script that encapsulates that, redirecting output to $ARCHIVE (this
      variable is installed in the environment template_analyze.
      
      The wrapper script will save the current time, and then run the program.
      If the program terminates with a zero exit status, it will ssh over to ops
      and invoke an xmlrpc routine to tell boss to add a program event to both
      the eventlist for the current instance, and to the template_eventlist for
      future instances. The time of the event is the relative start time that was
      saved above (remember, each experiment run replays the event stream from
      time zero).
      
      For the future, we want to allow this to be done on ops as well, but
      that will take more infrastructure, to run "program agents" on ops.
      
      It would be nice to install the ssl xmlrpc client side on our images so
      that we do not have to ssh to ops to invoke the client.
      9d021a07
  10. 09 Aug, 2006 1 commit
  11. 03 Aug, 2006 1 commit
    • Leigh B. Stoller's avatar
      Support for capturing the trace data that is stored in the pcal files · 4ce9c421
      Leigh B. Stoller authored
      into per-experiment databases on ops. Additional support for reconsituting
      those databases back into temporary databases on ops, for post processing.
      
      * This revision relies on the "snort" port (/usr/ports/security/snort)
        to read the pcap files and load them into a database. The schema is
        probably not ideal, but its better then nothing. See the file
        ops:/usr/local/share/examples/snort/create_mysql for the schema.
      
      * For simplicity, I have hooked into loghole, which already had all
        the code for downloading the trace data. I added some new methods to
        the XMLRPC server for loghole to use, to get the users DB password
        and the name of the per-experiment database. There is a new slot in
        the traces table that indicates that the trace should be snorted to
        its DB. In case you forgot, at the end of a run or when the instance
        is swapped out, loghole is run to download the trace data.
      
      * For reconsituting, there are lots of additions to opsdb_control and
        opsdb_control.proxy to create "temporary" databases and load them
        from a dump file that is stored in the archive. I've added a button
        to the Template Record page, inappropriately called "Analyze" since
        right now all it does is reconsitute the trace data into a DB on
        ops.
      
        Currently, the only indication of what has been done (the name of
        the DBs created on ops) is the log email that the user gets. A
        future project is tell the user this info in the web interface.
      
      * To turn on database capturing of trace data, do this in your NS
        file:
      
      	set link0 ...
      	$link0 trace
      	$link0 trace_snaplen 128
      	$link0 trace_db 1
      
         the increase in snaplen is optional, but a good idea if you want
         snort to undertand more then just ip headers.
      
      * Also some changes to the parser to allow plain experiments to take
        advantage of all this stuff. To simple get yourself a per-experiment
        DB, put this in your NS file:
      
      	tb-set-dpdb 1
      
        however, anytime you turn trace_db on for a link or lan, you
        automatically get a per-experiment DB.
      
      * To capture the trace data to the DB, you can run loghole by hand:
      
      	loghole sync -s
      
        the -s option turns on the "post-process" phase of loghole.
      4ce9c421
  12. 03 Jul, 2006 1 commit
    • Mike Hibler's avatar
      Framework for supporting 802.1q tagged VLANs as a form of multiplexed link. · 3f1c15e2
      Mike Hibler authored
      Actually, most of the changes here were just to generalize the "virtual
      interface" state in the DB.  Other than the client-side scripts, there
      is very little specific here specific to tagged VLANs.
      
      In fact, you cannot specify "vlan" as a type yet as we haven't done the
      snmpit support for setting up the switches.
      
      For more info see bas:~mike/flux/doc/testbed-virtinterfaces.txt, which I
      will integrate into the knowledge base and the Emulab doc at some point.
      3f1c15e2
  13. 27 Jun, 2006 1 commit
  14. 07 Jun, 2006 1 commit
  15. 04 May, 2006 1 commit
  16. 02 May, 2006 2 commits
  17. 15 Mar, 2006 2 commits
  18. 09 Feb, 2006 1 commit
  19. 26 Jan, 2006 1 commit
    • Timothy Stack's avatar
      · 4b938465
      Timothy Stack authored
      Some pelab/plab event system hacks.
      
      	* event/lib/event.h, event/lib/event.c: Add event_subscribe_auth
      	which lets you specify whether any authentication should be done
      	for events received through this subscription.
      
      	* event/sched/event-sched.c: Handle EVPROXY objects.  Add a
      	separate subscription for EVPROXY UPDATE events for each plab
      	pnode (which might be too many...).  Also, need to update the EXPT
      	field for events received through a noauth subscription so the
      	proxies can figure out which experiments are active.
      
      	* lib/libtb/tbdefs.h, lib/libtb/tbdefs.c: Add UPDATE event defs.
      
      	* xmlrpc/emulabserver.py.in: Inject __plab_setup and
      	__plab_teardown timelines into the eventlist when an experiment
      	has plab nodes.  The __plab_setup timeline sends EVPROXY UPDATE
      	events to each physical node while the __plab_teardown sends
      	EVPROXY CLEAR events.  The __plab_setup timeline is run when the
      	scheduler starts up, the __plab_teardown isn't run automatically
      	yet.
      4b938465
  20. 02 Jan, 2006 1 commit
  21. 29 Dec, 2005 1 commit
  22. 22 Dec, 2005 1 commit
  23. 13 Dec, 2005 1 commit
  24. 01 Dec, 2005 1 commit
  25. 28 Nov, 2005 1 commit
    • Timothy Stack's avatar
      · 19dbd0fd
      Timothy Stack authored
      Make the netlab client applet available to locals.
      
      	* configure, configure.in: Add xmlrpcpipe.php3.
      
      	* xmlrpc/emulabserver.py.in: Add missing virtual_tables.  Add
      	getareas call to get the list of robot areas.  Add node.getlist
      	and node.typeinfo methods for getting information about the nodes.
      	Add a "nic" argument to node.available to get the count of
      	wireless nodes.  Add "exclude" argument to
      	experiment.virtual_topology so we don't have to download the
      	massive route table, also delete the 'pid'/'eid' fields for the
      	same reason.  Don't return string output of virtual_topology, it's
      	huge.  Return some more info in experiment.getlist().
      
      	* www/GNUmakefile.in: Add xmlrpcpipe.php3.
      
      	* www/beginexp_form.php3, www/modifyexp.php3: Add links to the
      	client gui.
      
      	* www/netlab-client.jar: The client binary.
      19dbd0fd
  26. 08 Sep, 2005 1 commit
  27. 30 Aug, 2005 2 commits
  28. 26 Jul, 2005 1 commit
  29. 19 Jul, 2005 1 commit
  30. 24 Jun, 2005 1 commit
    • Timothy Stack's avatar
      · 71901cf5
      Timothy Stack authored
      Add a minor feature to loghole to support network link trace files.
      
      	* utils/loghole.in, utils/loghole.1: Create a directory for traced
      	network links that contains symbolic links to the pcap files.
      
      	* www/tutorial/advanced.html: Fix the link to the loghole doc.
      
      	* xmlrpc/emulabserver.py.in: Add a 'traces' aspect to the
      	experiment.info method (used by loghole).
      71901cf5
  31. 21 Jun, 2005 1 commit
  32. 17 Jun, 2005 1 commit
    • Timothy Stack's avatar
      · 41d4c6d4
      Timothy Stack authored
      Event system documentation and tweaks:
      
      	* event/sched/event-sched.c: The snapshot event got left out of
      	the list of events that send back a complete event.  If a top
      	level sequence stops with an error, automatically send a report.
      
      	* event/sched/group-agent.c, event/sched/timeline-agent.c: When
      	handling a complete event, return true if the event was consumed.
      
      	* event/sched/node-agent.cc: Clean the "/local/logs" loghole on
      	the node before taking a snapshot of the disk image.
      
      	* event/sched/simulator-agent.cc, event/sched/simulator-agent.h:
      	Made send_report non-static so it can be called when a sequence
      	errors out.  Actually check the clear flag when dumping report
      	data.
      
      	* lib/libtb/tbdefs.c: Add a comment about updating the event
      	scheduler when a new object type is added.
      
      	* tbsetup/ns2ir/sim.tcl.in: Check the OSIDs passed to the "reload"
      	and "snapshot-to" events.
      
      	* utils/loghole.1: Mention early and often the directories that
      	gets sync'd.  Document the 'change' action and new options for
      	'clean'.
      
      	* utils/loghole.in: Use rsync's '--include-from' option so we
      	don't have to run rsync separately for each directory.  Add a
      	'change' action that lets people mess with the metadata of an
      	existing archive.  Make the 'clean' action also remove log files
      	on the nodes.  Only rsync node's whose OS's support ssh.  Grab
      	logs from delay nodes.
      
      	* www/tutorial/advanced.html: Update the program agent examples to
      	use the new form.
      
      	* www/tutorial/eventsystem.html: First cut at an event system
      	reference manual that includes sequences, timelines, and other
      	new features.
      
      	* www/tutorial/loghole.html: Just a pointer to the man page on ops
      	for now.
      
      	* www/tutorial/nscommands.html: Add a section about capturing
      	parameters from the 'opt' array.
      
      	* xmlrpc/emulabserver.py.in: Add an osid.info() method that
      	returns information about a single OSID.
      41d4c6d4
  33. 25 May, 2005 1 commit
  34. 24 May, 2005 1 commit
  35. 27 Apr, 2005 1 commit
    • Leigh B. Stoller's avatar
      A set of changes to allow the newnode path (MFS and adding nodes) to · 94d97303
      Leigh B. Stoller authored
      work inside ElabInElab.
      
      The crux of this is to emulate what switchmac does when incorporating nodes
      into the testbed. Rather then using snmpit to ask the switches for MAC info,
      we can just ask outer emulab via the proxy to get that same info from the
      DB. Note, there are no changes to the newnode MFS; it will boot and happily
      report it after launching teachswitch, but those packets are harmless to the situation.
      
      Another wrinkle is that we need the outer emulab to tell us which
      interfaces are control and which are experimental, since inside there
      is no real way to determine that. I did this by overloading the final
      field in the return from switchmac, and using that to override the
      "role" setting that utils/newnode would normally choose on its own.
      
      There is another new routine in the XMLRPC server that is intended to
      be used for bypassing the newnode path. This is not hooked into
      anything yet, but the intent is that rather then using the web
      interface to "add" nodes, we just return everything needed to seed the
      new_nodes and new_interfaces table, and then run newnodes directly.
      94d97303
  36. 04 Apr, 2005 1 commit
    • Timothy Stack's avatar
      · 463ee6b1
      Timothy Stack authored
      Mote and robot related stuff.  The main thing is the addition of relay
      capabilities to capture and related things.
      
      	* GNUmakefile.in: Add the capture and tip subdirectories to the
      	client and client-install targets.
      
      	* configure, configure.in, config.h.in: Detect srandomdev() for
      	capture and add "mote/newmote" script.
      
      	* capture/GNUmakefile.in, capture/capture.c: Add "relay"
      	capabilities to capture.
      
      	* capture/capquery.c: Query the capserver for the relay receiver's
      	port number.
      
      	* capture/capserver.c: Small hack to return the port number
              for a node.
      
      	* db/libdb.pm.in, db/xmlconvert.in: Add virt_tiptunnels table.
      
      	* event/program-agent/program-agent.c: Change log file names to
      	something a little more user-friendly.  Add a "MODIFY" event
      	handler that lets the user set agent attributes (command, tag,
      	timeout) without having to run a program.
      
      	* event/sched/GNUmakefile.in, event/sched/console-agent.cc,
      	event/sched/console-agent.h, event/sched/event-sched.c: Add
      	console agents that can be used to snapshot a section of the
      	capture log file.
      
      	* event/sched/node-agent.cc: Some minor cleanup.
      
      	* event/sched/simulator-agent.cc, event/sched/simulator-agent.h:
      	Add the config data to the report mail.  Add a "RESET" event
      	handler that runs "loghole clean".  Save the report mail in a file
      	so it gets archived with the rest of the logs.
      
      	* lib/libtb/tbdefs.h: Add CONSOLE object type.
      
      	* mote/GNUmakefile.in, mote/newmote: Add newmote script, just a
      	quick hack to add motes to the DB.
      
      	* mote/tbuisp.in: Add another backend for loading motes through
      	their relay capture server.
      
      	* robots/mtp/mtp_dump.c: Dump the min/max values for x and y,
      	handy for figuring out the bounds of the camera.
      
      	* sql/database-fill.sql: Change the RELOAD-MOTE/SHUTDOWN ->
      	ALWAYSUP/SHUTDOWN mode transition to ALWAYSUP/ISUP since stated
      	doesn't seem to run triggers after a state change by a mode
      	transition.
      
      	* tbsetup/tbreport.in: Change the ordering of the eventlist so it
      	displays event-sequences appropriately.
      
      	* tbsetup/ns2ir/GNUmakefile.in, tbsetup/ns2ir/console.tcl,
      	tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/parse.tcl.in,
      	tbsetup/ns2ir/sim.tcl.in: Add a "console" agent that represents
      	the serial console for a node.
      
      	* tbsetup/ns2ir/sequence.tcl: Add an "append" method so it is
      	easier to build sequences dynamically.
      
      	* tbsetup/ns2ir/topography.tcl: Make checkdest available to
      	regular users.
      
      	* tip/GNUmakefile.in, tip/tiptunnel.c: Add support for uploading a
      	file to a relay version of capture and exporting the end
      	connection as a pty.
      
      	* tmcd/decls.h, tmcd/common/libsetup.pm: Bump version number since
      	the dosubnodelist change is not backwards compatible.
      
      	* tmcd/tmcd.c: Make dosubnodelist and dosubconfig callable even
      	when a node isn't allocated.  Add dotiptunnels command that
      	returns which serial consoles are to be mounted on a node.  Add
      	mote version of subconfig that returns information needed to
      	startup the relay version of capture.
      
      	* tmcd/common/bootsubnodes: For motes, startup the relay version
      	of capture (XXX stargate specific).
      
      	* tmcd/common/libsetup.pm, tmcd/common/libtmcc.pm,
      	tmcd/common/config/rc.config, tmcd/common/config/rc.tiptunnels:
      	Client side changes for mounting another nodes serial line.
      
      	* tmcd/common/rc.bootsetup: Always boot the subnodes, even when
      	free.  This is used for motes since their capture needs to be up
      	for reloading at the time.
      
      	* tmcd/linux/ixpboot: Shuffle some code around so the script
      	doesn't fail if the ixp isn't allocated.
      
      	* utils/loghole.in: Add "digest.out" and "report.mail" as global
      	logs to be saved in archives and display the "report.mail" file
      	when showing a loghole archive.
      
      	* xmlrpc/emulabserver.py.in: Scrub more of the return values to
      	get rid of "None"s.
      463ee6b1
  37. 08 Mar, 2005 1 commit
    • Timothy Stack's avatar
      · b3c23e5d
      Timothy Stack authored
      Fix a couple of things I missed in my last checkin.
      
      	* event/program-agent/program-agent.c: Setup of the user
      	environment variables wasn't ordered correctly.
      
      	* event/sched/event-sched.c, event/sched/rpc.h,
      	event/sched/rpc.cc: Setup the expected user environment variables,
      	just like in program-agent.
      
      	* event/sched/simulator-agent.cc: Rename the old feedback data so
      	we don't lose it in case the vnode remapping modify fails.
      
      	* xmlrpc/emulabserver.py.in: Add virt_user_environment table data
      	to the result of the experiment.metadata method, so the
      	event-scheduler can get it.
      b3c23e5d