1. 09 Mar, 2005 1 commit
  2. 07 Mar, 2005 1 commit
    • Timothy Stack's avatar
      · 898cf9a2
      Timothy Stack authored
      Checkin some changes related to experiment automation and vnode feedback:
      
      	* configure, configure.in: Add sensors/canaryd/feedbacklogs
      	template.
      
      	* db/libdb.pm.in, db/xmlconvert.in: Add "virt_user_environment"
      	table that holds environment variable names and values.
      
      	* event/lib/event.c: Allocate memory of the right size for
      	event_notifications.
      
      	* event/program-agent/GNUmakefile.in: Add version.c file and
      	add install targets for the man page.
      
      	* event/program-agent/program-agent.8: Man page describing the
      	program-agent daemon.
      
      	* event/program-agent/program-agent.c: Add a bunch of convenience
      	features: let the user specify the working directory for commands;
      	save output to separate files on every invocation of an agent; let
      	the user specify a timeout for a command; make the set of
      	environment variables sane and add vars given in the NS file in
      	the opt array; a "status" file containing process information is
      	written out when children are collected.  Internal changes: child
      	processes are collected immediately, instead of waiting for the
      	next START event, so we can send back COMPLETE events; the daemon
      	now runs with a real-time priority, to increase the chances of
      	receiving events.
      
      	* event/proxy/evproxy.c: Made it bidirectional so the
      	program-agent's COMPLETE events make it back to the scheduler.
      
      	* event/sched/error-record.c: Change the default log directory.
      
      	* event/sched/event-sched.h, event/sched/event-sched.c: Setup an
      	environment similar to a program-agent to run the user's log
      	digester.
      
      	* event/sched/node-agent.cc: Add a handler for the SNAPSHOT event
      	that runs create_image for the node.
      
      	* event/sched/simulator-agent.h, event/sched/simulator-agent.cc:
      	Let the user specify a "DIGESTER" script that digests the log
      	files into a summary of the results.  Add event handler for
      	remapping a vnode experiment.
      
      	* event/sched/timeline-agent.c: Accept the RUN event as well as
      	the START event.
      
      	* os/GNUmakefile.in: Install the install-tarfile.1 man page.
      
      	* os/install-tarfile: Automatically chown/chgrp any files that do
      	not have valid user or group IDs, the new owner will be the user
      	that swapped in the experiment.  Include the install directory in
      	the DB file.  Add a "list" mode that just dumps what files have
      	been installed and where.  Add a "force" option so the user can
      	forcefully install the file, even though the DB says its already
      	there.
      
      	* os/install-tarfile.1: Man page describing the install-tarfile
      	tool.
      
      	* os/syncd/GNUmakefile.in: Install man pages on ops.
      
      	* sensors/canaryd/GNUmakefile.in: Link canaryd statically and
      	install "feedbacklogs" tool.
      
      	* sensors/canaryd/canaryd.c: Dump dummynet pipe data.
      
      	* sensors/canaryd/canarydEvents.c: Log errors.
      
      	* sensors/canaryd/feedbacklogs.in: Tool used to generate feedback
      	data from canaryd log files.
      
      	* sensors/slothd/GNUmakefile.in: Install digest-slothd on ops.
      
      	* sensors/slothd/digest-slothd: Fix some bugs and write out an
      	"alert" file with all the nodes/links that were overloaded.
      
      	* tbsetup/os_load.in, tbsetup/libosload.pm.in: Add "waitmode"
      	argument that lets you specify that you want to wait for the disk
      	to finish loading and/or wait for the node to come back up in the
      	new OS.
      
      	* tbsetup/power.in: Remove debugging printf.
      
      	* tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/program.tcl,
      	tbsetup/ns2ir/sequence.tcl, tbsetup/ns2ir/sim.tcl.in: Fix some
      	quoting problems with event-sequences.  Add -expected-exit-code
      	and -tag options to the "$program run" event.  Add -digester to
      	the "$ns report" event that lets the user specify a program to run
      	to digest the log files.
      
      	* tbsetup/ns2ir/tb_compat.tcl.in: Change the initial scaling
      	factor for feedback nodes to 1%, instead of 100%.
      
      	* tmcd/tmcd.c, tmcd/common/libtmcc.pm: Add "userenv" command that
      	returns the values in "virt_user_environment".  Return new program
      	agent fields: dir, timeout, and expected_exit_code.
      
      	* tmcd/common/GNUmakefile.in: Install rc.canaryd.
      
      	* tmcd/common/bootvnodes: Add hack to boost the program-agents to
      	a real-time priority, they can't do it from inside the jail.
      
      	* tmcd/common/rc.canaryd: Rc script for canaryd.
      
      	* tmcd/common/watchdog: Don't fail outright if there is a bad line
      	in the battery.log
      
      	* tmcd/common/rc.progagent: Append "userenv" data to the
      	program-agent config file.
      
      	* utils/GNUmakefile.in: Install loghole and its man page on ops.
      
      	* utils/loghole.1: Document "clean" command and the change in
      	loghole directories.
      
      	* utils/loghole.in: Add "clean" command and parallelization.
      
      	* xmlrpc/emulabserver.py.in: Add "virt_user_environment" table.
      	Order the eventlist by "idx" and time, needed for sequences.  And
      	removed unnecessary nologin checks.
      898cf9a2
  3. 08 Feb, 2005 1 commit
  4. 14 Jan, 2005 3 commits
    • Kirk Webb's avatar
      Get rid of some debugging stuff. · 231845af
      Kirk Webb authored
      231845af
    • Kirk Webb's avatar
      0e27c632
    • Timothy Stack's avatar
      · dee46d59
      Timothy Stack authored
      Cross compilation fixes for the stargates, 'gmake client' should now
      build, link, and install properly.  Haven't really tried to run stuff though.
      
      	* GNUmakerules: Add target for stripping executables, used instead
      	of "install -s" since that doesn't work for cross-compiling.
      
      	* Makeconf.in: Add ELVIN_CONFIG variable that refers to
      	'elvin-config'.
      
      	* configure, configure.in: Detect and save the elvin-config path
      	since we need a different one for cross-compiling.
      
      	* event/lib/GNUmakefile.in, event/link-agent/GNUmakefile.in,
      	event/linktest/GNUmakefile.in, event/program-agent/GNUmakefile.in,
      	event/proxy/GNUmakefile.in, event/tbgen/GNUmakefile.in,
      	event/trafgen/GNUmakefile.in, os/dijkstra/GNUmakefile.in,
      	os/syncd/GNUmakefile.in, sensors/slothd/GNUmakefile.in,
      	tmcd/GNUmakefile.in, tmcd/linux/GNUmakefile.in: Cross compilation
      	fixes, don't statically link on arm, create "foo-debug"
      	executables with debugging info and install separately stripped
      	ones instead of passing "-s" to install.
      dee46d59
  5. 03 Jan, 2005 1 commit
  6. 05 Nov, 2004 1 commit
  7. 01 Nov, 2004 2 commits
  8. 25 Oct, 2004 3 commits
    • Timothy Stack's avatar
      d439b3a1
    • Timothy Stack's avatar
      · 636aaa2b
      Timothy Stack authored
      Changes to the "auto nice daemon" so it can work better in Emulab.
      
        * sensors/and/GNUmakefile.in: Emulab-specific make file.  Updated to
          work with a build tree separate from the source and gave it a new
          version number. Files are installed under "/usr/testbed/" on ops.
      
        * sensors/and/Makefile: Add a warning that this is not the real
          makefile for us.
      
        * sensors/and/and-OpenBSD.c: Update to work with FreeBSD and add
          support for reporting process start time.
      
        * sensors/and/and-emulab.conf.in: Emulab-specific configuration,
          similar to the standard one, except it sends mail to tbops when it
          does something.
      
        * sensors/and/and-emulab.priorities: Emulab-specific priorities
          database. It excludes daemon pseudo users and the event-scheduler,
          otherwise, niceness levels apply to everyone.
      
        * sensors/and/and.8.man: Add the pid file to the 'FILES' section.
      
        * sensors/and/and.c: Added support for running a command when a
          niceness level change occurs.  Also writes out the pid file when
          not running in test mode.
      
        * sensors/and/and.conf.5.man: Add docs for the level commands.
      
        * sensors/and/and.h: Add start time and child CPU time to the
          and_procent struct.
      
        * sensors/and/and.startup: Changed to use "test" instead of bash
          syntax and the pid file is now used instead of killall.  Also
          added a "reconfig" option that HUPs the daemon.
      636aaa2b
    • Timothy Stack's avatar
      8f1a8857
  9. 08 Oct, 2004 1 commit
  10. 05 Oct, 2004 1 commit
  11. 01 Sep, 2004 1 commit
  12. 27 Jul, 2004 2 commits
    • Timothy Stack's avatar
      Many tweaks to the feedback stuff and more comments. Update · ead90a69
      Timothy Stack authored
      webfeedback to talk to the newer version of canaryd.  Add feedback
      "estimate" stuff so that if we have no data (because of an overloaded
      node) to work with, we can make some sort of "reasonable" guesstimate
      on every iteration.
      ead90a69
    • Timothy Stack's avatar
      Updated canaryd, ended up starting fresh and pulling things in, rather · 62b5e3a8
      Timothy Stack authored
      than jamming more stuff into the old one.  Most of the code came from
      the previous version of canaryd, the cpu broker (process accounting),
      and the janosvm (network interface accounting).  Its missing some
      features of the old one, but those can be incorporated without too
      much trouble.
      
      Changes:
      
        Designed to permanently run on the pnodes:  it waits for START events
          before it begins recording.  However, I haven't done the work
          necessary to have it always startup on the pnodes.
      
        No more exec'ing: process stuff is taken from "/proc", and network
          interface stats are pulled from getifaddrs(3).
      
        Fixed some minor bugs: A typo caused the real-time priority to not
          be set, use setitimer instead of sleep to get more accurate
          spacing between samples.
      62b5e3a8
  13. 15 Jul, 2004 2 commits
    • Kirk Webb's avatar
      · 35d31c2f
      Kirk Webb authored
      Removed a couple of unneeded lines.
      35d31c2f
    • Kirk Webb's avatar
      · dd7d003d
      Kirk Webb authored
      Updated the sensors directory Makefile to use the very nice infrastructure
      for subdirs that Eric pointed me to.  Also, canaryd will no longer
      be built if the arch isn't FreeBSD.
      dd7d003d
  14. 14 Jul, 2004 2 commits
    • Mike Hibler's avatar
      b6690372
    • Kirk Webb's avatar
      · c0ecee80
      Kirk Webb authored
      Split off the feedback metric gathering, alerting, and reporting code from
      slothd and slammed it into a new daemon: canaryd.  There is currently quite
      a bit of code duplication between slothd and canaryd.  Sometime in the future
      I need to go back and extract the common portions into a module.
      c0ecee80
  15. 12 Jul, 2004 1 commit
  16. 01 Jul, 2004 1 commit
  17. 24 Jun, 2004 2 commits
    • Mike Hibler's avatar
      Improve the client-side install. With these changes, it should now be · 976133e4
      Mike Hibler authored
      possible to:
      
      	gmake client
      	sudo gmake client-install
      
      on a FBSD4, FBSD5, RHL7.3, and RHL9.0 client node.
      
      There are still some dependencies that are not explicit and which would
      prevent a build/install from working on a "clean" OS.  Two that I know of are:
      you must install our version of the elvin libraries and you must install boost.
      976133e4
    • Mike Hibler's avatar
      Minor lint for GCC 3.3 · a36ccc7b
      Mike Hibler authored
      (early stages of getting Emulab software to build under FreeBSD 5)
      a36ccc7b
  18. 10 May, 2004 1 commit
  19. 26 Apr, 2004 1 commit
    • Mike Hibler's avatar
      Cleanup Makefiles: · 297019fb
      Mike Hibler authored
      1. "make clean" will just remove stuff built in the process of a regular build
      2. "make distclean" will also clean out configure generated files.
      
      This is how it was always supposed to be, there was just some bitrot.
      297019fb
  20. 20 Apr, 2004 1 commit
    • Mike Hibler's avatar
      Improve the client-install. You can now do a "make client-install" from · 361ee691
      Mike Hibler authored
      the top level.  This will build all the necessary binaries and then install
      them.  This works on FBSD4 and RHL7.3.  It still doesn't work on FBSD5
      (newer compiler that no longer supports a style of use of _FUNCTION_ in the
      event lib) or RHL9 (event lib needs SSL lib which has a bad dependency
      on Kerberos).  Notes:
      
      - requires that elvin libraries be installed on nodes (they are) to build
        event agents, requires linuxthreads be installed on FBSD (it is now) to
        build imagezip (which is installed, but is not strictly necessary)
      
      - installed event-agents and other binaries are stripped
      
      - added a few missing files to the source tree for bsd (healthd.conf)
        and linux (healthd.conf, rc.local)
      
      - the only thing that doesn't get rebuilt in /usr/local/etc/emulab is
        healthd, I couldn't quickly find how it gets built
      
      - uses a scaled down version of libtb with no DB functions (since mysql
        isn't installed on nodes).  N.B. DO NOT DO A CLIENT INSTALL FROM YOUR
        REGULAR OBJ TREE OR ELSE YOU MAY WIND UP WITH A NEUTERED VERSION OF
        libtb.a!
      
      The build-as-well-as-install semantics are counter to the regular install
      targets, but this is what we gotta do for now.  Once the TB source builds
      under Linux and newer BSDs, we could undo this and just require that people
      do a regular "make" followed by "make client-install"  OTOH, there should
      be no reason to require installation of mysql and other server-side packages
      just to build clients (or make them sit through the compilation of assign),
      so maybe we will keep the client build special.
      361ee691
  21. 23 Dec, 2003 1 commit
  22. 15 Oct, 2003 1 commit
    • Mike Hibler's avatar
      Uniform syslog'ing. Change everything I could find to use a syslog facility · cc6d6fa7
      Mike Hibler authored
      as defined in the defs-* file (e.g. "TBLOGFACIL=local2").  The default is
      "local5" which is what we are setup to use so you shouldn't need to mess
      with your defs- file!
      
      perl scripts just get this value configured in when configure is run.
      C programs get the value in two ways.  For programs that are intimate with
      the testbed infrastructure, and include "config.h", they just get it from
      that file.  For programs that we sometimes use outside the Emulab build
      environment (e.g., frisbee, capture) and that don't include config.h,
      the value is set via a "-DLOG_TESTBED=..." in the GNUmakefile build line.
      If the value isn't set, it defaults to what it used to be (usually LOG_USER).
      
      Still to do: healthd, hmcd (whose build doesn't seem to be completely
      integrated) and plabdaemon.in (since its icky python :-)
      cc6d6fa7
  23. 22 Aug, 2003 1 commit
  24. 31 Jul, 2003 1 commit
    • Kirk Webb's avatar
      Minor changes: · 073c9473
      Kirk Webb authored
      1) Use libtb's reporting funcs rather than my own (the former are better)
      2) Produce more informational "malformed interface record" errors.
      073c9473
  25. 28 Apr, 2003 1 commit
  26. 04 Apr, 2003 1 commit
  27. 21 Mar, 2003 1 commit
    • Kirk Webb's avatar
      Well, here it is: The new slothd and sdcollectd stuff. · a337f7b9
      Kirk Webb authored
      Several things have changed; here is an outline:
      
       - "Thresholds" added to slothd
      	These are checked at each iteration against current idle data
              to determine whether or not to set bits in the "active" vector.
      	- LAST_TTY_ACT (now) > LAST_TTY_ACT (previous)
              - LOAD > <provided value>
              - EXPER_PKT_COUNT(now) - EXPER_PKT_COUNT(prev) > <provided value>
      	- CNTRL_PKT_COUNT(now) - CNTRL_PKT_COUNT(prev) > <provided value>
      	- Values passed in via command line: do ./slothd -h for help
       - "Aggressive" mode added to slothd
      	When a regular interval's worth of time has passed, and no activity
              has been noted, slothd goes into "aggressive" mode.  In this mode,
      	slothd changes its sleep interval, checking for activity much more
      	frequently (still reporting every <reg_interval>) until activity
      	is noted.  This is based on the threshold logic above.  When activity
      	is seen, slothd sends a report, and goes back to "regular" reporting.
      	- params passed via command line for regular, and aggressive intervals.
       - sdcollectd updates data in new "node_activity" table
      	- Updates timestamps based on activity bits in received slothd packets
      	- No new rows are added.
       - Option added to sdcollectd to insert into "old" node_idlestats and
      	iface_counters tables
         	- This is not the default (-o)
       - New sdcollectd still accepts old client reports
      	- but does not update the node_activity table with them.
       - Misc.
      	- Code cleanup
      	- Scan "active terminals only" option & code removed from slothd
      	- Always send report immediately on startup (no more option)
      	- Randomize time of second report to mitigate synchronization
      	- Send report on shutdown (receipt of INT, TERM, or QUIT)
      a337f7b9
  28. 18 Dec, 2002 1 commit
  29. 07 Oct, 2002 1 commit
  30. 12 Jul, 2002 1 commit
    • Mac Newbold's avatar
      Fix flags passed to slothd. Before, -f meant send an immediate update (which... · 3a643160
      Mac Newbold authored
      Fix flags passed to slothd. Before, -f meant send an immediate update (which wasn't the default behavior). The latest slothd updates (several weeks ago at least) changed the polarity of the flag. It still does the non-default thing, but now that makes it wait an hour before sending an update. So we remove -f so it does the right thing.
      3a643160
  31. 10 Jul, 2002 1 commit