1. 25 Nov, 2014 1 commit
  2. 04 Nov, 2014 1 commit
    • Leigh Stoller's avatar
      Add runsonxen script to set the bits of DB state required. · 04c35b0b
      Leigh Stoller authored
      	usage: runsonxen [-p <parent>] <imageid>
      	usage: runsonxen -a [-p <parent>]
      	usage: runsonxen -c <imageid>
      	Options:
      	 -n      - Impotent mode
      	 -c      - Clear XEN parent settings completely
      	 -a      - Operate on all current XEN capable images
      	 -p      - Set default parent; currently XEN43-64-STD
      04c35b0b
  3. 28 Oct, 2014 1 commit
  4. 09 Jul, 2014 1 commit
  5. 01 Jul, 2014 1 commit
  6. 13 Jun, 2014 1 commit
  7. 06 Jun, 2014 1 commit
    • Leigh Stoller's avatar
      New script, analogous to Mike's node_traffic script. Basically, it · b885ce89
      Leigh Stoller authored
      was driving me nuts that we do not have an easy way to see what is
      going on *inside* the fabric.
      
      So this one reports on traffic across trunk links and interconnects
      out of the fabric.  Basic operation is pretty simple:
      
      	Usage: switch_traffic [-rs] [-i seconds] [switch[:switch] ...]
      	Reports traffic across trunk links and interconnects
      	-h          This message
      	-i seconds  Show stats over a <seconds>-period interval
      
      So with no arguments will give portstats style output of all trunk
      links and interconnects in the database. Trunk links are aggregate
      numbers of all of the trunk wires that connect two switches.
      
      The -i option gives traffic over an interval, which is much more
      useful than the raw packet numbers, since on most of our switches
      those numbers have probably rolled over a few times.
      
      You can optionally specify specific switches and interconnects on the
      command line. For example:
      
      boss> wap switch_traffic -i 10 cisco3 ion
      Trunk                    InOctets      InUpkts   InNUpkts   ...
      ----------------------------------------------------------- ...
      cisco3:cisco10                128            0          1   ...
      cisco3:cisco8                2681            7          4   ...
      cisco3:cisco1                4493           25          7   ...
      cisco3:cisco9                 192            0          1   ...
      cisco3:cisco4                 128            0          2   ...
      pg-atla:ion                     0            0          0   ...
      pg-hous:ion                     0            0          0   ...
      pg-losa:ion                     0            0          0   ...
      pg-salt:ion                  2952            0         42   ...
      pg-wash:ion                     0            0          0   ...
      
      NOTE that the above output is abbreviated so it does not wrap in the
      git log, but you get the idea.
      
      Or you can specify a specific trunk link:
      
      	boss> wap switch_traffic -i 10 cisco3:cisco8
      
      Okay this is all pretty basic and eventually it would be nice to take
      these numbers and feed them into mrtg or rrdtool so we can view pretty
      graphs, but this as far as I can take it for now.
      
      Maybe in the short term it would be enough to record the numbers every
      5 minutes or so and put the results into a file.
      b885ce89
  8. 09 May, 2014 1 commit
    • Mike Hibler's avatar
      New imagevalidate tool for printing/checking/updating image metadata. · 0bb906f4
      Mike Hibler authored
      This should be run whenever an image is created or updated and possibly
      periodically over existing images. It makes sure that various image
      metadata fields are up to date:
      
       * hash: the SHA1 hash of the image. This field has been around for
         awhile and was previously maintained by "imagehash".
      
       * size: the size of the image file.
      
       * range: the sector range covered by the uncompressed image data.
      
       * mtime: modification time of the image. This is the "updated"
         datetime field in the DB. Its intent was always to track the update
         time of the image, but it wasn't always exact (create-image would
         update this with the current time at the start of the image capture
         process).
      
      Documentation? Umm...the usage message is comprehensive!
      It sports a variety of useful options, but the basics are:
      
       * imagevalidate -p <image> ...
          Print current DB metadata for indicated images. <image> can either
          be a <pid>/<imagename> string or the numeric imageid.
      
       * imagevalidate <image> ...
          Check the mtime, size, hash, and image range of the image file and
          compare them to the values in the DB. Whine for ones which are out
          of date.
      
       * imagevalidate -u <image> ...
          Compare and then update DB metadata fields that are out of date.
      
      Fixed a variety of scripts that either used imagehash or computed the
      SHA1 hash directly to now use imagevalidate.
      0bb906f4
  9. 17 Mar, 2014 1 commit
  10. 21 Jan, 2014 1 commit
  11. 06 Jan, 2014 1 commit
    • Mike Hibler's avatar
      Add support for lease extention (renewal). · 9a6cdeae
      Mike Hibler authored
      Add CLI for extending a lease (called extenddataset on ops). The length
      of the extension and the number of times it can be extended are controlled
      by site variables.
      9a6cdeae
  12. 03 Jan, 2014 1 commit
    • Mike Hibler's avatar
      First attempt to cleanup some hack jobs. · c5a1812c
      Mike Hibler authored
      Make a createdataset to handle dataset leases and move dataset specific
      code out of approvelease and into Lease.pm (which is now Lease.pm.in as
      it needs to be configured). Lease.pm still needs a bunch of OO-ification
      to properly make datasets a subclass of leases. But, another day...
      c5a1812c
  13. 11 Dec, 2013 2 commits
    • Mike Hibler's avatar
      Add script to approve a lease and add some locking in other scripts. · 6fef3cce
      Mike Hibler authored
      approvelease is the place where storage actually gets allocated for
      a lease. It uses bscontrol to contact an appropriate freeNAS storage
      server and allocate a ZFS volume.
      
      deletelease is the place where storage is deallocated. Note that once
      a lease has been approved and storage allocated, it cannot be returned
      to the unapproved state. The only way to free storage is to delete the
      lease.
      
      Both approve and delete use an intermediate state, "initializing", to
      signal that the lease is in the middle of a potentially time-consuming
      allocation/deallocation procedure. I probably should have just used the
      lease locking mechanism instead.
      
      Approve, delete, and mod all DO use the locking mechanism when examining
      and manipulating the state of a lease. Nonetheless, I am sure that are
      still plenty of races.
      6fef3cce
    • Mike Hibler's avatar
  14. 23 Jul, 2013 1 commit
  15. 22 Jul, 2013 1 commit
  16. 14 May, 2013 1 commit
    • Leigh Stoller's avatar
      Add prototype EC2 image import plumbing. · 980aa180
      Leigh Stoller authored
      To create a new descriptor that will be an import from EC2 (and thus
      run under XEN), add ?ec2=1 to newimage_ez.php3. Eventually will link
      it in someplace. The form will create a XEN based VM, but instead of
      node to snapshot from, provide user@host for the EC2 instance.
      
      On the image snapshot page, instead of node use user@host for the EC2
      instance.
      
      The backend script (create_image) will call over to ops and invoke
      Srikanth's code. I have called that script ec2import-image.pl. See
      create_image for how arguments are passed to the script.
      980aa180
  17. 25 Mar, 2013 1 commit
  18. 14 Jan, 2013 1 commit
  19. 12 Dec, 2012 1 commit
    • Gary Wong's avatar
      Add a "mktestbedtest" script. · 08ca1a04
      Gary Wong authored
      It constructs an experiment including every (available) experimental PC,
      and every relevant link, so that during swap-in linktest will exercise
      as much of the testbed as possible.
      08ca1a04
  20. 03 Dec, 2012 1 commit
    • Leigh Stoller's avatar
      Add sitecheckin client and server, which will tell Utah (Mother Ship) · 6591e9fd
      Leigh Stoller authored
      about Emulab sites. Nothing private, just the equivalent of calling
      testbed-version so that we know what sites exist and what software
      they are running.
      
      This is opt-out; sites that do not want to tell Utah about themselves
      can set NOSITECHECKIN in their defs file.
      
      In Utah, there is a new option in the Administration drop down menu to
      print out the list from the DB.
      6591e9fd
  21. 14 Nov, 2012 1 commit
  22. 30 Oct, 2012 1 commit
    • Mike Hibler's avatar
      Remaining infrastructure for control network "ARP lockdown". · 4b5e17b0
      Mike Hibler authored
      It works like this. Certain nodes that are on the node control net
      (right now just subbosses, but ops coming soon) can set static ARP entries
      for the nodes they serve. This raises the bar for (but does not eliminate
      the possibility of) nodes spoofing servers. Currently this is only for
      FreeBSD.
      
      When such a server boots, it will early on run /etc/rc.d/arplock.sh
      which will in turn run /usr/local/etc/emulab/fixarpinfo. fixarpinfo
      asks boss via an SSL tmcc call for "arpinfo" (using SSL ensures that the
      info coming back is really from boss). Tmcd on boss returns such arpinfo
      as appropriate for the node (subboss, ops, fs, etc.) along with the type
      of lockdown being done. The script uses this info to update the ARP
      cache on the machine, adding, removing, or making permanent entries
      as appropriate.
      
      fixarpinfo is intended to be called not just at boot, but also whenever
      we might need to update the ARP info on a server. The only other use right
      now is in subboss_dhcpd_makeconf which is called whenever DHCP info may
      need to be changed on a subboss (we hook this because a call to this script
      might also indicate a change in the set of nodes served by the subboss).
      In the future, fixarpinfo might be called from the newnode path (for ops/fs,
      when a node is added to the testbed), the deletenode path, or maybe from
      the watchdog (if we started locking down arp entries on experiment nodes)
      
      The type of the lockdown is controlled by a sitevar on boss,
      general/arplockdown, which can be set to 'none', 'static' or 'staticonly'.
      'none' means do nothing, 'static' means just create static arp entries
      for the given nodes but continue to dynamically arp for others, and
      'staticonly' means use only this set of static arp entries and disable
      dynamic arp on the control net interface. The last implies that the server
      will only be able to talk to the set of nodes for which it got ARP info.
      
      As mentioned, tmcd is responsible for returning the correct set of arp
      info for a given request. The logic currently is:
      
       * Only return ARP info to nodes which are on the CONTROL_NETWORK.
         If the requester is elsewhere (e.g., Utah's boss and ops are currently
         segregated on different IP subnets) then this whole infrastructure
         does not apply and nothing is returned.
      
       * If the requester is a subboss, return info for all other servers that
         are on the node control network as well as for the set of nodes
         which the subboss serves.
      
       * If the requester is an ops or fs node, again return info for all
         other servers and info for all testnodes or virtnodes whose control
         net IP is on the node control net.
      
       * Otherwise, return nothing.
      
      One final note is that the ARP info for servers such as boss/ops/fs or
      the gateway router is not readily available in most Emulab instances
      since those machines are not in the DB nodes or interfaces tables.
      Eventually we will fix that, but for now the info must come from new
      site variables. To help initially populate those variables, I added
      the utils/update_sitevars script which attempts to determine which
      servers are on the node control net and gathers the appropriate IP and
      MAC info from them.
      4b5e17b0
  23. 16 Oct, 2012 1 commit
  24. 26 Sep, 2012 1 commit
  25. 24 Sep, 2012 1 commit
    • Eric Eide's avatar
      Replace license symbols with {{{ }}}-enclosed license blocks. · 6df609a9
      Eric Eide authored
      This commit is intended to makes the license status of Emulab and
      ProtoGENI source files more clear.  It replaces license symbols like
      "EMULAB-COPYRIGHT" and "GENIPUBLIC-COPYRIGHT" with {{{ }}}-delimited
      blocks that contain actual license statements.
      
      This change was driven by the fact that today, most people acquire and
      track Emulab and ProtoGENI sources via git.
      
      Before the Emulab source code was kept in git, the Flux Research Group
      at the University of Utah would roll distributions by making tar
      files.  As part of that process, the Flux Group would replace the
      license symbols in the source files with actual license statements.
      
      When the Flux Group moved to git, people outside of the group started
      to see the source files with the "unexpanded" symbols.  This meant
      that people acquired source files without actual license statements in
      them.  All the relevant files had Utah *copyright* statements in them,
      but without the expanded *license* statements, the licensing status of
      the source files was unclear.
      
      This commit is intended to clear up that confusion.
      
      Most Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the Affero GNU General Public License, version 3
      (AGPLv3).
      
      Most Utah-copyrighted files related to ProtoGENI are distributed under
      the terms of the GENI Public License, which is a BSD-like open-source
      license.
      
      Some Utah-copyrighted files in the Emulab source tree are distributed
      under the terms of the GNU Lesser General Public License, version 2.1
      (LGPL).
      6df609a9
  26. 14 Sep, 2012 1 commit
    • Leigh Stoller's avatar
      "improvements" to prereserve: · f7219346
      Leigh Stoller authored
      New option -s datetime to specify a starting time for the pre-reserve.
      New option -e datetime to specify a ending time for the pre-reserve.
      
      The idea is that you can schedule a pre-reserve to begin sometime later,
      and you can optionally specify a time for a prereserve to terminate.
      There is a new script that runs from cron that checks for pre-reserves
      that need to be started or terminated.
      
      For example:
      
      boss> wap prereserve -s '2012-09-14 09:08:15' -e '2012-09-15' emulab-ops 50
      
      You can use any datetime string that is valid for str2time. At some point
      it would be nice to allow natural language dates ("tomorrow") but that
      requires a another bunch of perl packages and I didn't want to bother.
      
      NOTE: when using -e, -r is implied; in other words, when the
      pre-reserve is terminated, the table entry is cleared *and* the
      reserved_pid of all of the nodes is cleared. Any experiments using
      those nodes is left alone, although if the user does a swapmod, they
      could easily lose the nodes if another pre-reserve is set up that
      promises those nodes to another project.
      f7219346
  27. 04 Sep, 2012 1 commit
    • Leigh Stoller's avatar
      Add image import utilities. · e468f885
      Leigh Stoller authored
      image_setup is run from tbprerun to verify and create image
      descriptors, and then later from tbswap to actually download
      and verify the image (ndz) file.
      
      image_import does the actual work for a specific image (url).
      e468f885
  28. 30 Aug, 2012 2 commits
    • Leigh Stoller's avatar
      More bits and pieces for exporting images from one Emulab to another. · 4c444cd5
      Leigh Stoller authored
      image_metadata.php will return an Emulab style image descriptor in XML
      format. A remote emulab, given an image URL, will grab this XML
      description and use it to create a local descriptor. Inside the
      descriptor is an additional URL that is used to download ndz file.
      
      The dumpdescriptor script is now web accessible, and takes a new -e
      (export) option that adds the extra URL and other bits that are needed
      to import the descriptor and the image.
      
      On the Show Image page, show the metadata URL, which is suitable for
      using in an NS file or an rspec (when that code is committed).
      4c444cd5
    • Gary Wong's avatar
      Add a "ctrladdr" utility to show (un)allocated addresses on the control net. · 9047e21a
      Gary Wong authored
      Right now, the only addresses it knows are allocated are anything assigned
      in the interfaces table with a "ctrl" role, and anything in the dynamic
      pool in the virt_node_public_addr table.  (And the reserved network and
      broadcast addresses.)
      
      This needs to be extended to anything else we know about!
      
      By default, the output is supposed to be easy to parse and simply
      displays the first available address.  More than one available address
      can be requested with the "-n" option (e.g. "-n 10" will show the
      first ten unallocated addresses).  "-n 0" will show every free
      address on the subnet.
      
      The "-a" option (meant more for human consumption) also describes
      allocated addresses.  For instance, "ctrladdr -a -n 0" will show
      every address on the control net, and what it's used for (if
      anything).  "-r" will compress ranges of consecutive free addresses
      onto a single line.
      
      To test whether a particular address is in use, invoke it as (e.g.)
      "ctrladdr -t 155.98.36.1".  This will give an exit code of 0 if the
      address is available, and 1 if used.  Any other options are ignored
      if "-t" is specified.
      9047e21a
  29. 29 Aug, 2012 1 commit
  30. 25 Jul, 2012 1 commit
  31. 02 Jul, 2012 1 commit
  32. 07 Jun, 2012 1 commit
    • Leigh Stoller's avatar
      New script, clone_image to simplify create/snapshot from a node. · b01c991d
      Leigh Stoller authored
      clone_image is a wrapper around newimageid_ez and create_image, that
      simplifies the most common operation; creating a new imageid derived
      from the image/os that is currently running in the node, and then
      taking a snapshot of the node. So for example, if node pcXXX is
      running image FREEBSD, and you want to create a custom image from that
      node, all you need to do:
      
      	boss> clone_image myfreebsd pcXXX
      
      which will create the new descriptor, deriving everything from the
      FREEBSD image on the node, and then take a snapshot from pcXXX. If
      the descriptor already exists, just take the snapshot.
      
      So what if you do:
      
      	boss> clone_image FREEBSD pcXXX
      
      well, the image is always looked up in the project the node is
      currently attached to, so in fact a new descriptor is created in that
      project, and you do not actually overwrite an image from some other
      project. 
      
      I've added some locking to images to prevent concurrent snapshots.
      This seemed like a good idea since this script is going to be used
      from the ProtoGeni interface. More on this in another commit.
      b01c991d
  33. 06 Jun, 2012 1 commit
    • Leigh Stoller's avatar
      New script to compute the hash of an image, create the .sha1 file, and · 92b2bc19
      Leigh Stoller authored
      set the hash in the DB. This is helpful on system images where we save
      the image off in /proj and copy it back later, and also for computing
      the hash of the zillions of images that already exist.
      
      Usage: imagehash [-d] [-n] <imageid>
             imagehash -p <imageid>
      Options:
             -d     Turn on debug mode
             -p     Show the current hash in the DB
             -n     Impotent mode; compute hash but do not update
      92b2bc19
  34. 16 May, 2012 1 commit
    • Leigh Stoller's avatar
      Another protogeni checkbox; scriptify and simplify adding "special" · cf517af6
      Leigh Stoller authored
      devices with network interfaces. Emulab's spp and bbg nodes are
      examples, but I did all that by hand inserting sql. An spp node is a
      shared node with some interfaces. Users can allocate one or more of
      those interfaces and establish vlans to the interfaces. The node is a
      "fakenode" in "shared" mode, and everything else falls out. The mapper
      assigns virtual nodes until all of the interfaces are allocated,
      snmpit does its work on the interfaces, and the user then does the
      rest.
      
      Anyway, to added a special device:
      
        boss> wap addspecialdevice -s -t goober goober1
      
      The -t argument is the name of the node type, created if it does not
      exists. The last argument is the name of the fakenode to create in the
      DB. The -s option says the special device is shared. Without -s, the
      device is allocated exclusively.
      
      Then to add interfaces to the device:
      
        boss> wap addspecialiface -b 1Gb -s cisco4,100,100 goober1 eth0
      
      The -b option is the speed (either 100Mb or 1Gb). The -s option is the
      switch side of the interface (switchname,card,port). The last two
      arguments are the nodename and iface name for the interfaces table.
      
      After the interface and wires table entry are added to the DB, snmpit
      is called to put the switch port into tagged mode (if the node is
      shared). To skip the snmpit step, add the -t option.
      cf517af6
  35. 04 May, 2012 1 commit
  36. 27 Apr, 2012 1 commit
  37. 11 Apr, 2012 1 commit
    • Leigh Stoller's avatar
      So this commit allows a vlan to be "shared" bewteen experiments. By · dae29101
      Leigh Stoller authored
      shared, I mean that an experiment can request that a port be put into
      a vlan belonging to another experiment. This started out as a hack to
      support openflow enabled vlans in Geni, but then I got a request to
      make it a little more general purpose. You all know how that goes.
      
      Okay, say you have an experiment E1 in some project and that
      experiment has a link or lan call "lan0". You want other experiments
      to be able to stick ports in that vlan. On boss, you would do this
      after E1 is swapped in:
      
      boss> wap sharevlan -o testbed,E1 lan0 mysharedlan
      
      The -o option says to make the vlan open to anyone; without that
      option, only admins can swap in an experiment that requests a port in
      lan0.  The token "mysharedlan" is just a level of indirection for the
      NS file (or rspec).
      
      Next you create a new experiment E2, and in your NS file:
      
      	$ns make-portinvlan $n1 "mysharedlan"
      
      which says to create a lan with a interface on node n1, in the vlan
      named by the token mysharedlan. The token keeps specific pid/eids out
      of the NS file. 
      
      When E2 is swapped in, assign does its thing, and the selected port is
      added to the members list for lan0 in testbed,E1 and then we call
      snmpit with the syncvlansfromtables (-X) option to get the port added.
      
      When E2 is swapped out, we undo the members list and call snmpit with
      the -X option again.
      
      The access issue is a bit of hack of course (open or admins) but I did
      not want to invent a new permission mechanism (yet).
      
      And of course, this is still a work in progress.
      dae29101
  38. 27 Mar, 2012 1 commit
    • Leigh Stoller's avatar
      Bunch of changes for "management" interfaces (ilo,drac,etc); make · 85b81867
      Leigh Stoller authored
      management interfaces more of a first class citizen instead of a
      hack. New script:
      
      management_iface -t <type> -a [key|pswd] [-s <switchinfo>]
                              <node_id> mac IP arg1 arg2
      management_iface -r <node_id>
        -h       This message
        -t type  Management type; ilo, ilo2, drac
        -s info  Optional switch info; switch,card,port
        -s -     Search output of switchmac to find switch info
        -a pswd  Password auth; provide login and password.
        -a key   SSH key auth; provide login and key path.
        -r       Remove management interface from DB.
      
      which adds the management interface to the database (interfaces,
      outlets and outlets_remoteauth. Optionally adds the wires table
      entry if you add -s option. Uses switchmac to find the switch info or
      you can specify it on the command line. So for example, here is what I
      did to add the ilo2 interface for a node:
      
      management_iface -t ilo2 -a pswd -s - pc1 e8:39:35:ae:c9:7c \
                       155.98.34.100 elabman mypasswd
      or
      management_iface -t ilo2 -a key -s - pc1 e8:39:35:ae:c9:7c \
                       155.98.34.100 elabman /root/.ssh/somekey
      
      Of course someone had to have added the elabman user and key or
      password to the ilo config via its interface. 
      
      * dhcpd_makeconf will add local node management interfaces to the
        config file. We can set them to dhcp instead of hardwiring the IP in
        the management interface.
      
      * The DB changes add a management type to the enums in the interfaces
        and wires table, and updates the existing interface entries.
      85b81867