1. 18 Jan, 2002 2 commits
  2. 09 Jan, 2002 1 commit
  3. 08 Jan, 2002 1 commit
  4. 07 Jan, 2002 1 commit
    • Leigh B. Stoller's avatar
      Checkpoint first working version of Frisbee Redux. This version · 86efdd9e
      Leigh B. Stoller authored
      requires the linux threads package to give us kernel level pthreads.
      
      From: Leigh Stoller <stoller@fast.cs.utah.edu>
      To: Testbed Operations <testbed-ops@fast.cs.utah.edu>
      Cc: Jay Lepreau <lepreau@cs.utah.edu>
      Subject: Frisbee Redux
      Date: Mon, 7 Jan 2002 12:03:56 -0800
      
      Server:
      The server is multithreaded. One thread takes in requests from the
      clients, and adds the request to a work queue. The other thread processes
      the work queue in fifo order, spitting out the desrired block ranges. A
      request is a chunk/block/blockcount tuple, and most of the time the clients
      are requesting complete 1MB chunks. The exception of course is when
      individual blocks are lost, in which case the clients request just those
      subranges.  The server it totally asynchronous; It maintains a list of who
      is "connected", but thats just to make sure we can time the server out
      after a suitable inactive time. The server really only cares about the work
      queue; As long as the queue si non empty, it spits out data.
      
      Client:
      The client is also multithreaded. One thread receives data packets and
      stuffs them in a chunkbuffer data structure. This thread also request more
      data, either to complete chunks with missing blocks, or to request new
      chunks. Each client can read ahead up 2 chunks, although with multiple
      clients it might actually be much further ahead as it also receives chunks
      that other clients requested. I set the number of chunk buffers to 16,
      although this is probably unnecessary as I will explain below. The other
      thread waits for chunkbuffers to be marked complete, and then invokes the
      imagunzip code on that chunk. Meanwhile, the other thread is busily getting
      more data and requesting/reading ahread, so that by the time the unzip is
      done, there is another chunk to unzip. In practice, the main thread never
      goes idle after the first chunk is received; there is always a ready chunk
      for it. Perfect overlap of I/O! In order to prevent the clients from
      getting overly synchronized (and causing all the clients to wait until the
      last client is done!), each client randomizes it block request order. This
      why we can retain the original frisbee name; clients end up catching random
      blocks flung out from the server until it has all the blocks.
      
      Performance:
      The single node speed is about 180 seconds for our current full image.
      Frisbee V1 compares at about 210 seconds. The two node speed was 181 and
      174 seconds. The amount of CPU used for the two node run ranged from 1% to
      4%, typically averaging about 2% while I watched it with "top."
      
      The main problem on the server side is how to keep boss (1GHZ with a Gbit
      ethernet) from spitting out packets so fast that 1/2 of them get dropped. I
      eventually settled on a static 1ms delay every 64K of packets sent. Nothing
      to be proud of, but it works.
      
      As mentioned above, the number of chunk buffers is 16, although only a few
      of them are used in practice. The reason is that the network transfer speed
      is perhaps 10 times faster than the decompression and raw device write
      speed. To know for sure, I would have to figure out the per byte transfer
      rate for 350 MBs via network, via the time to decompress and write the
      1.2GB of data to the raw disk. With such a big difference, its only
      necessary to ensure that you stay 1 or 2 chunks ahead, since you can
      request 10 chunks in the time it takes to write one of them.
      86efdd9e
  5. 04 Jan, 2002 1 commit
    • Robert Ricci's avatar
      New script: unixgroups . Pretty simple - just a convenient way to manage the · 469dacdb
      Robert Ricci authored
      unixgroup_membershit table from the command line. Runs the appropriate
      commands to make changes in the 'real world' after the database has been
      updated. From the usage message:
      
      Usage: unixgroups <-h | -p | < <-a | -r> uid gid...> >
      -h            This message
      -p            Print group information
      -a uid gid... Add a user to one (or more) groups
      -r uid gid... Remove a user from one (or more) groups
      469dacdb
  6. 03 Jan, 2002 1 commit
    • Robert Ricci's avatar
      Added our apache config file to CVS, in a continuing attempt to put everything · e0d19fbe
      Robert Ricci authored
      needed to build an emulab boss node in one convenient package.
      
      The config file gets run through autoconf to get the DocumentRoot, log, and
      other directories. There is an install target for it, but this is as yet
      unused by anything else. There is also a new configure option specifiying where
      the config file should go.
      e0d19fbe
  7. 26 Dec, 2001 1 commit
    • Leigh B. Stoller's avatar
      A bunch o' account managment script schanges. I have reworked · 46068860
      Leigh B. Stoller authored
      mkprojdir, mkacct-cntrl, mkgroup, and group-update into a set of new
      scripts that are more specific to their intended operation, and strive
      to do less work.
      
      1. mkacct - Replaces mkacct-cntrl. This script no longer does any
         group stuff. All it does is create new accounts, or update the
         password and gecos fields of existing accounts. Usage is the same
         as it was: "mkacct <userid>", and is typically invoked from the web
         interface via the approveuser form.
      
      2. mkgroup - Replaces group-update. This script creates new groups,
         either for the main project when it is approved, or for subgroup
         creation. This script does not alter the group membership. Usage
         is typically from the web interface, but mkgroup can be invoked
         from the command line: "mkgroup [-b | -a] <pid> <gid>" where -b
         puts it in the background and sends email later, while -a just
         captures the log and emails. This "audit" feature is going to find
         its way into more scripts as soon as I figure out a neat and clean
         perl mechanism to make it easy.
      
      3. setgroups - Replaces group-update. This script modifies the group
         membership of either specific users, or all the users in a
         project. It is typically invoked from the web interface when a
         project leader edits the subgroup membership or when a user is
         first approved to a project or subgroup. Command line usage is:
      
      	setgroups [-b | -a] -p <pid> [user ...]
              setgroups [-b | -a] [user ...]\n
      
         The first form is mostly a means to speed things up. The web
         interfaces knows exactly what users have need to be changed, but a
         global project update is nice too.
      
      4. mkproj - Replaces mkprojdir. Actually, mkproj still has all that
         directory code, but it also handles creating the groups and the
         account for the project leader. Part of my policy to move as much
         random code out of the web interface and into the PERL backend
         where it belongs.
      46068860
  8. 03 Dec, 2001 1 commit
    • Leigh B. Stoller's avatar
      Checkpoint the visualization stuff. Rework Chad's stuff to match · e72c90a8
      Leigh B. Stoller authored
      current testbed software practices. Add a wrapper script to go from
      a pid/eid to the top file in the experiment directory. This means we
      cannot visualize experiments that are not active, but until we have a
      topfile generator that is independent of assign_wrapper, there is
      nothing to do about that. A makefile to install new tools. Also add a
      couple of web pages. The first web page spits out a page with an img
      tag which refers to another php script that generates the gif file
      with the backend tools, and stuff is out with an appropriate content
      header line. Very nifty.
      e72c90a8
  9. 05 Nov, 2001 1 commit
    • Leigh B. Stoller's avatar
      Changes to node control (web page). Added a backend script to do this · f9cfddd4
      Leigh B. Stoller authored
      stuff so that the web page did not need to do anything except display
      and form processing. Add tbsetup/node_control for backend so that it
      can be called from the command line too. The virt_nodes table is also
      updated (for those values that have virt_nodes equivalents), and this
      mostly implies that changes can be applied only to swapped in
      experiments since we use the reserved table to map pcXXX to its vname
      so that the virt_nodes table can be updated. It is an easy extension
      to allow changes based on the pid/eid/vname, but I do not see a reason
      to support this ability yet. Note usage:
      
          Usage: node_control name=value [name=value ...] node [node ...]
                 node_control -e pid,eid name=value [name=value ...]
                 node_control -l
          For multiword values, use name='word0 ... wordN'
          Use -l to get a list of operational parameters you can change.
          Use -e to change parameters of all nodes in an experiment.
      
          {824} stoller$ /build/testbed/install//bin/node_control -l
            next_boot_osid            - (administrators only)
            startup_command
            bios_version              - (administrators only)
            rpms                      - (multiple options allowed)
            default_boot_cmdline
            default_boot_path
            default_boot_osid
            next_pxe_boot_path        - (administrators only)
            tarfiles                  - (multiple options allowed)
            pxe_boot_path             - (administrators only)
            next_boot_cmdline         - (administrators only)
            deltas                    - (multiple options allowed)
            next_boot_path            - (administrators only)
      f9cfddd4
  10. 30 Oct, 2001 1 commit
  11. 29 Oct, 2001 1 commit
    • Leigh B. Stoller's avatar
      A bunch of lastlogin changes! The user and experiment information · 4658545e
      Leigh B. Stoller authored
      pages now show the lastlogin info that is gathered from sshd syslog
      reporting to users. That info is parsed by security/genlastlog.c, and
      entered into the DB in the nodeuidlastlogin and uidnodelastlogin
      tables. If not obvious from the names, for each user we want the last time
      they logged in anyplace, and for each node we want the last time anyone
      logged into it. The latter is obviously more useful for scheduling
      purposes. All of the various images have new /etc/syslog.conf files,
      and the 6.2 got new sshd_configs (all cvsup'ed with kill -HUP). There
      is an entry in boss:/etc/crontab and users:/etc/syslog.conf. All of
      this is decribed in greater detail in security/genlastlog.c.
      4658545e
  12. 24 Oct, 2001 2 commits
    • Leigh B. Stoller's avatar
    • Leigh B. Stoller's avatar
      Add swappable and priority bits to experiment creation form. Not used, · 28c1968f
      Leigh B. Stoller authored
      but simply entered into the DB record for the experiment until we know
      what to do with them. Add to batchexp script arguments, since all that
      stuff is done outside the web interface. Add a swapexp perl script to
      swap an an experiment in/out form the command line. Add web links on
      the Experiment Information page to do this from the web interface. A
      bunch of locking changes. Previously expt_terminating in the
      experiment record prevented multiple calls to terminate an experiment,
      but now we have a more general locking problem with
      start,swapin,swapout, and terminate, so change expt_terminating to
      expt_locked (still a datetime field) and add locking to all of
      startexp, swapexp, and endexp. Note that batch experiments cannot be
      swapped yet because of locking issues still to be resolved. Minor
      cleanup in tbreport to make email message look better.
      28c1968f
  13. 20 Oct, 2001 1 commit
  14. 17 Oct, 2001 1 commit
    • Leigh B. Stoller's avatar
      Rework of the batch experiment code. Unified it with the immediate · 4d420b21
      Leigh B. Stoller authored
      experiment code. No longer uses another table. Rather, the experiment
      record contains a couple of extra fields for the batch system. Also
      combined some of the backend code (no longer a killbatch script).
      Also added scriptable experiments; the batchexp program in the bin
      directory can start an experiment from the command line, and in fact
      is used from the web page for both batch experiments and immediate
      experiments (-i option). All of the DB code that was in the web
      interfaces was moved to batchexp.
      4d420b21
  15. 16 Oct, 2001 1 commit
  16. 15 Oct, 2001 1 commit
  17. 27 Sep, 2001 1 commit
    • Robert Ricci's avatar
      Lib-ification. These scripts have been changed from using the Mysql module to · 4c11aa5d
      Robert Ricci authored
      using the libdb library. They have also been cleaned up somewhat, with more
      comments added, etc.
      
      I've merged nodeip and mac2if, which had very similar functions. The merged
      version goes under the name nodeip.
      
      Lastly, I've added some access checks to db2ns and node_status, to make sure
      they're only called by the right people.
      4c11aa5d
  18. 21 Sep, 2001 1 commit
  19. 19 Sep, 2001 1 commit
  20. 13 Sep, 2001 1 commit
    • Robert Ricci's avatar
      New script: frisbeelauncher · b26a78d3
      Robert Ricci authored
      Manages the launching of new frisbee servers, and recording the
      addresses the use in the database. If called for an image that is
      already associated with a running server, exits quitely. Otherwise,
      registers the new server's address and goes into the background, waiting
      for the frisbee server to die so that it can unregister the address.
      b26a78d3
  21. 30 Aug, 2001 1 commit
  22. 20 Aug, 2001 1 commit
  23. 14 Aug, 2001 1 commit
  24. 01 Aug, 2001 1 commit
    • Leigh B. Stoller's avatar
      An attempt at making image creation an easy/automatic operation. HA! · 27f26d99
      Leigh B. Stoller authored
      This uses the pxe booted freebsd kernel and MFS. In addition, I use
      the standard testbed mechanism of specifying a startup command to
      run, which will do the imagezip to NFS mounted /proj/<pid>/.... The
      controlling script on paper sets up the database, reboots the node,
      and then waits for the startstatus to change. Then it resets the DB
      and reboots the node so that it returns back to its normal OS. The
      format of operation is:
      
      	create_image <node> <imageid> <filename>
      
      Node must be under the user's control of course. The filename must
      reside in the node's project (/proj/<pid>/whatever) since thats the
      directory that is mounted by the testbed config software when the
      machine boots. The imageid already exists in the DB, and is used to
      determine what part of the disk to zip up (say, using the slice option
      to the zipper). Since this operation is rather time consuming, it does
      the usual trick of going to background and sending email status later.
      27f26d99
  25. 31 Jul, 2001 1 commit
  26. 24 Jul, 2001 1 commit
    • Leigh B. Stoller's avatar
      Add new console_reset script to bulk reset all of the tip lines on · 4303838f
      Leigh B. Stoller authored
      plastic. Useful for when plastic reboots, although it needs to be
      run from paper of course. We should probably set up some kind of
      reboot utility for when plastic reboots, since when capture fires up,
      the "acl" files are all in the wrong group/mode. On the other hand,
      this shared file based permission scheme is going to end up changing
      anyway soon, since it won't work when the serial lines are on other
      machines. Will need a tip permserver at that point.
      4303838f
  27. 13 Jul, 2001 1 commit
  28. 12 Jul, 2001 1 commit
    • Robert Ricci's avatar
      New script: dhcpd_makeconf · fbb56b98
      Robert Ricci authored
      Usage: dhcpd_makeconf [-h] [-v] <templatefile>
      templatefile should be a skeleton dhcp.conf file, without entires for each node.
      The string '%%nodetype=<type>' will be expanded to entries for all nodes of type
      <type> in the following form:
      host $ip {
      	hardware ethernet $mac;
      	option host-name '$node_id';
      	fixed_address $ip;
      }
      
      The generated dhcp conf file is printed on stdout.
      
      The -v option causes dhcpd_makeconf to use virtual names, rather than
      physical ones, for hostnames.
      fbb56b98
  29. 10 Jul, 2001 1 commit
    • Mac Newbold's avatar
      Added sched_reserve. Works kind of like sched_reload, but for a reservation. · e1f4263c
      Mac Newbold authored
      Rob and I noticed that when we need to do maintainance to a node, its hard
      to grab it when it gets freed before it gets taken again. So this will allow
      admins to set up nodes so that when the current owner frees them, they
      immediately are moved into another experiment. For instance, if I need to do
      an upgrade on all the machines in the testbed, I can do:
      
      sched_reserve testbed down pc1 pc2 pc3 [...] pc40
      
      and it will immediately reserve any free nodes, skip any that are already
      in testbed/down, and add an entry to the next_reserve table for any nodes
      that are already reserved. Then nfree checks for entries in the next_reserve
      table before it lets any nodes go (much like it does for the reloads table),
      and moves them into the expt before it can get reserved again. So we can
      guarantee that the next time a node is freed, it will be saved for us.
      
      Note that this can also be useful to accumulate nodes for a large experiment,
      since you can make sure you get them as soon as the current owner is done.
      e1f4263c
  30. 06 Jul, 2001 1 commit
  31. 02 Jul, 2001 1 commit
  32. 29 Jun, 2001 2 commits
  33. 26 Jun, 2001 1 commit
    • Robert Ricci's avatar
      New script: sshtb · 9de266c3
      Robert Ricci authored
      sshtb is a _very_ simple shell script that runs ssh with a few commandline
      parameters, which make it play nicer in an script environment. These
      parameters can be changed with the '--with-ssh-args' argument, but default to:
      '-q -o "BatchMode yes" -o "StrictHostKeyChecking no"'
      All ssh calls now use this script.
      9de266c3
  34. 20 Jun, 2001 1 commit
  35. 15 Jun, 2001 1 commit
  36. 08 Jun, 2001 1 commit
  37. 31 May, 2001 1 commit