1. 07 Jun, 2012 1 commit
    • Leigh B Stoller's avatar
      New script, clone_image to simplify create/snapshot from a node. · b01c991d
      Leigh B Stoller authored
      clone_image is a wrapper around newimageid_ez and create_image, that
      simplifies the most common operation; creating a new imageid derived
      from the image/os that is currently running in the node, and then
      taking a snapshot of the node. So for example, if node pcXXX is
      running image FREEBSD, and you want to create a custom image from that
      node, all you need to do:
      
      	boss> clone_image myfreebsd pcXXX
      
      which will create the new descriptor, deriving everything from the
      FREEBSD image on the node, and then take a snapshot from pcXXX. If
      the descriptor already exists, just take the snapshot.
      
      So what if you do:
      
      	boss> clone_image FREEBSD pcXXX
      
      well, the image is always looked up in the project the node is
      currently attached to, so in fact a new descriptor is created in that
      project, and you do not actually overwrite an image from some other
      project. 
      
      I've added some locking to images to prevent concurrent snapshots.
      This seemed like a good idea since this script is going to be used
      from the ProtoGeni interface. More on this in another commit.
      b01c991d
  2. 06 Jun, 2012 1 commit
    • Leigh B Stoller's avatar
      New script to compute the hash of an image, create the .sha1 file, and · 92b2bc19
      Leigh B Stoller authored
      set the hash in the DB. This is helpful on system images where we save
      the image off in /proj and copy it back later, and also for computing
      the hash of the zillions of images that already exist.
      
      Usage: imagehash [-d] [-n] <imageid>
             imagehash -p <imageid>
      Options:
             -d     Turn on debug mode
             -p     Show the current hash in the DB
             -n     Impotent mode; compute hash but do not update
      92b2bc19
  3. 16 May, 2012 1 commit
    • Leigh B Stoller's avatar
      Another protogeni checkbox; scriptify and simplify adding "special" · cf517af6
      Leigh B Stoller authored
      devices with network interfaces. Emulab's spp and bbg nodes are
      examples, but I did all that by hand inserting sql. An spp node is a
      shared node with some interfaces. Users can allocate one or more of
      those interfaces and establish vlans to the interfaces. The node is a
      "fakenode" in "shared" mode, and everything else falls out. The mapper
      assigns virtual nodes until all of the interfaces are allocated,
      snmpit does its work on the interfaces, and the user then does the
      rest.
      
      Anyway, to added a special device:
      
        boss> wap addspecialdevice -s -t goober goober1
      
      The -t argument is the name of the node type, created if it does not
      exists. The last argument is the name of the fakenode to create in the
      DB. The -s option says the special device is shared. Without -s, the
      device is allocated exclusively.
      
      Then to add interfaces to the device:
      
        boss> wap addspecialiface -b 1Gb -s cisco4,100,100 goober1 eth0
      
      The -b option is the speed (either 100Mb or 1Gb). The -s option is the
      switch side of the interface (switchname,card,port). The last two
      arguments are the nodename and iface name for the interfaces table.
      
      After the interface and wires table entry are added to the DB, snmpit
      is called to put the switch port into tagged mode (if the node is
      shared). To skip the snmpit step, add the -t option.
      cf517af6
  4. 04 May, 2012 1 commit
  5. 27 Apr, 2012 1 commit
  6. 11 Apr, 2012 1 commit
    • Leigh B Stoller's avatar
      So this commit allows a vlan to be "shared" bewteen experiments. By · dae29101
      Leigh B Stoller authored
      shared, I mean that an experiment can request that a port be put into
      a vlan belonging to another experiment. This started out as a hack to
      support openflow enabled vlans in Geni, but then I got a request to
      make it a little more general purpose. You all know how that goes.
      
      Okay, say you have an experiment E1 in some project and that
      experiment has a link or lan call "lan0". You want other experiments
      to be able to stick ports in that vlan. On boss, you would do this
      after E1 is swapped in:
      
      boss> wap sharevlan -o testbed,E1 lan0 mysharedlan
      
      The -o option says to make the vlan open to anyone; without that
      option, only admins can swap in an experiment that requests a port in
      lan0.  The token "mysharedlan" is just a level of indirection for the
      NS file (or rspec).
      
      Next you create a new experiment E2, and in your NS file:
      
      	$ns make-portinvlan $n1 "mysharedlan"
      
      which says to create a lan with a interface on node n1, in the vlan
      named by the token mysharedlan. The token keeps specific pid/eids out
      of the NS file. 
      
      When E2 is swapped in, assign does its thing, and the selected port is
      added to the members list for lan0 in testbed,E1 and then we call
      snmpit with the syncvlansfromtables (-X) option to get the port added.
      
      When E2 is swapped out, we undo the members list and call snmpit with
      the -X option again.
      
      The access issue is a bit of hack of course (open or admins) but I did
      not want to invent a new permission mechanism (yet).
      
      And of course, this is still a work in progress.
      dae29101
  7. 27 Mar, 2012 1 commit
    • Leigh B Stoller's avatar
      Bunch of changes for "management" interfaces (ilo,drac,etc); make · 85b81867
      Leigh B Stoller authored
      management interfaces more of a first class citizen instead of a
      hack. New script:
      
      management_iface -t <type> -a [key|pswd] [-s <switchinfo>]
                              <node_id> mac IP arg1 arg2
      management_iface -r <node_id>
        -h       This message
        -t type  Management type; ilo, ilo2, drac
        -s info  Optional switch info; switch,card,port
        -s -     Search output of switchmac to find switch info
        -a pswd  Password auth; provide login and password.
        -a key   SSH key auth; provide login and key path.
        -r       Remove management interface from DB.
      
      which adds the management interface to the database (interfaces,
      outlets and outlets_remoteauth. Optionally adds the wires table
      entry if you add -s option. Uses switchmac to find the switch info or
      you can specify it on the command line. So for example, here is what I
      did to add the ilo2 interface for a node:
      
      management_iface -t ilo2 -a pswd -s - pc1 e8:39:35:ae:c9:7c \
                       155.98.34.100 elabman mypasswd
      or
      management_iface -t ilo2 -a key -s - pc1 e8:39:35:ae:c9:7c \
                       155.98.34.100 elabman /root/.ssh/somekey
      
      Of course someone had to have added the elabman user and key or
      password to the ilo config via its interface. 
      
      * dhcpd_makeconf will add local node management interfaces to the
        config file. We can set them to dhcp instead of hardwiring the IP in
        the management interface.
      
      * The DB changes add a management type to the enums in the interfaces
        and wires table, and updates the existing interface entries.
      85b81867
  8. 15 Mar, 2012 1 commit
    • Leigh B Stoller's avatar
      Add a new localize_mfs script (based on stuff that was in the mfs · e894ec36
      Leigh B Stoller authored
      install script, but I pulled out to create an independent script).
      This works on both freebsd and linux based MFSs. The intent is to do
      all of the localization automcatically for site admins, so that they
      can import new MFSs more easily. This is also used from the new
      install code to bring in the initial MFSs and localize them.
      
      Here is what we localize:
      
      * The timezone is copied from boss:/etc/localtime to mfs:/etc. Ryan
        says the upcoming version of the linux MFS will actually use
        localtime. 
      
      * Copy boss:/usr/testbed/etc/{emulab.pem,client.pem} to mfs:/etc/emulab. 
        The former is for TPM, the later for the ssl version of tmcc.
      
      * Copy out boss root ssh keys (pub) to mfs:/root/.ssh/authorized_keys.
        In an ElabInElab we take care to combine with outer boss keys.
      
      * Copy out the image ssh host keys. These are the keys that we put on
        every image to avoid the ssh host key change sillyness. See notes
        below on how these keys are initialized on an existing emulab. The
        keys are copied from boss:/usr/testbed/etc/image_hostkeys to
        mfs:/etc/ssh directory.
        
      * Initialize the root and toor passwords from a new sitevar named
        images/root_password (which is the encryption hash, not plain
        text). See notes below on how this sitevar is initialized on an
        existing emulab.
      
      About initializing the host keys and the root password hash ... I
      added a new update script (27) that will go out to the current frisbee
      MFS and mount it, grab the current keys and password hash, and put
      them into place on boss. At the moment I only look for a FreeBSD
      frisbee MFS, since not too many people are running the linux mfs, and
      this was hard enough as it is!
      
      For a new installation, a new install phase script will build the them
      and install into /usr/testbed/etc/image_hostkeys. I have not dealt
      with the password yet.
      e894ec36
  9. 08 Mar, 2012 1 commit
  10. 10 Oct, 2011 1 commit
    • Leigh B Stoller's avatar
      Add support for sharing images between projects. New table called · 646b64f6
      Leigh B Stoller authored
      image_permissions stores access info for images. You can share an
      image with a user or a group (project), and you can specify write
      access to allow updating the image in place. Note that write access
      does not allow the descriptor to be modified, only the image itself.
      Well, that is how it will be after Mike changes mfrisbeed.
      
      The front end script to modify permissions is grantimage:
      
      	boss> grantimage -u stoller -w tbres,myimage
      	boss> grantimage -u stoller -w tbres,myimage
      
      which grants write access to stoller. Or:
      
      	boss> grantimage -g testbed,testbed tbres,myimage
      
      which grants access to the testbed project. Notice that you can
      specify subgroups this way.
      
      	boss> grantimage -l tbres,myimage
      
      will give you a list of current permissions. To revoke, just add -r
      option:
      
      	boss> grantimage -g testbed,testbed -r tbres,myimage
      
      Who is allowed to grant access to an image? 1) An adminstrator of
      course, 2) the image creator, and 3) any group_root in the group that
      the image belongs to. Being granted access to use an image does not
      confer permission to grant access to others.
      
      One last task; while the web interface displays the permissions, there
      is no web interface to modify the permissions; users will still have
      to ask us for now.
      646b64f6
  11. 12 Aug, 2011 1 commit
    • Leigh B Stoller's avatar
      Lets make it easier to manage pre reservations (Mike, this was Rob's · 5c998ffc
      Leigh B Stoller authored
      idea).
      
      New script and table to manage node pre reservations. Lets just look
      at the script.
      
      To create a reservation:
      
          myboss> wap prereserve -t pc850 testbed 2
          Node reservation request for 2 nodes has been created.
      
      To see the reservation status for testbed
      
          myboss> wap prereserve -i testbed
          Project         Cnt (Cur)  Creator    When               Pri Types
          -------------------------------------------------------------
          testbed         1 (1)      stoller    2011-08-12 12:39:07 0   pc850
      
          which says 1 node is pending and 1 node has already been
          pre-reserved. 
      
      To clear the above reservation request (and optionally, clean
      reserved_pid from the nodes table).
      
          myboss> wap prereserve -c -r testbed
      
          The -r is optional, otherwise just the reservation request is
          cleared, and nodes continue to be pre-reserved to the project.
      
      To see a list of all reservation requests:
      
          myboss> wap prereserve -l
      
      
      So, when a node is released in nfree, we look at the reservation
      status for the node and any pending reservation requests.
      
      1. If the node has a reserved_pid and that request is still pending
         (still in the table), nothing is changed.
      
      2. If the node has a reserved_pid, but the request has been cleared
         from the pending table, then clear reserved_pid.
      
      3. If reserved_pid is null, and there are pending requests, then pick
         the highest priority, most recent dated, request, and set
         reserved_pid to that project.
      
      Options:
      
      * -n <pri> - is how you set a priority. Lowest is zero, choose a
        higher number if you want this reservation request to be considered
        before others. In a tie, look at the date of creation, and use the
        oldest.
      
      * -t <typelist> - a comma separated list of types you want to
        consider. Types are considered in order, but not in the fancy way
        you might imagine.
      5c998ffc
  12. 18 Mar, 2011 1 commit
  13. 15 Dec, 2010 1 commit
  14. 25 Oct, 2010 1 commit
    • Leigh B Stoller's avatar
      New module, called Emulab Features. The basic usage (see tbswap) is: · 1d430992
      Leigh B Stoller authored
      use EmulabFeatures;
      
      if (EmulabFeatures->FeatureEnabled("NewMapper", $user, $group, $experiment)) {
         # Do something
      }
      else {
         # Do something else.
      }
      
      where $user, $group, and $experiment is the current Emulab user, group, and
      experiment the script is operating as. Any of them can be undef. Note that
      features can easily be globally enabled or disabled (bypassing user/group
      check). See below.
      
      There are two scripts to deal with features. The easy one is the script to
      grant (or revoke) feature usage to a particular user or group or experiment:
      
      boss> wap grantfeature -u stoller NewMapper
      boss> wap grantfeature -p geni NewMapper
      boss> wap grantfeature -e geni,myexp NewMapper
      
      Add -r to revoke the feature.
      
      The other script is for managing features. To create a new feature:
      
      boss> wap emulabfeature create NewFeature 'A pithy description'
      
      which adds the feature to the emulab_features DB table. Use "delete"
      to remove a feature from the DB.
      
      You can globally enable and disable features for all users/groups (the
      user/group checks are bypassed). Global disable overrides global
      enable. There are actually two different flags. Lots of rope, I mean
      flexibility.
      
      boss> wap emulabfeature enable NewFeature 1
      boss> wap emulabfeature enable NewFeature 0
      
      boss> wap emulabfeature disable NewFeature 1
      boss> wap emulabfeature disable NewFeature 0
      
      To display a list of all features and associated settings:
      
      boss> wap emulabfeature list
      
      To show the details (including the users and groups) of a specific
      feature:
      
      boss> wap emulabfeature show NewFeature
      
      Oh, if a test is made in the code for a feature, and that feature is
      not in the emulab_features table (as might be the case on other
      Emulab's), the feature is "disabled".
      1d430992
  15. 14 Oct, 2010 1 commit
    • Gary Wong's avatar
      Add a script to compress old expinfo directories. · c8827ceb
      Gary Wong authored
      Run it as "archive-expinfo [-t threshold]", where "threshold" is the
      number of days experiments must have been inactive to be compressed
      (defaulting to 1000).
      
      Directories will be tarred and compressed in place.  For example, if
      /usr/testbed/expinfo/testbed/example/1234 has been inactive for longer
      than the threshold, its entire contents will be archived in
      /usr/testbed/expinfo/testbed/example/1234.tar.bz2 and the original
      directory removed.
      
      The compression is extremely efficient, typically reducing directories
      to around 2% of their original size.
      c8827ceb
  16. 01 Jul, 2010 1 commit
  17. 23 Apr, 2010 1 commit
  18. 15 Apr, 2010 1 commit
  19. 08 Apr, 2010 1 commit
  20. 18 Dec, 2009 1 commit
  21. 07 Nov, 2009 1 commit
    • Leigh B. Stoller's avatar
      Change to infodir (/usr/testbed/expinfo) handling; experiment · 1855897b
      Leigh B. Stoller authored
      directories are now placed in a project subdirectory, to avoid
      blowing out the max number of subdirs (32K in FreeBSD). Dirs are
      now called $pid/$eid/$idx.
      
      This script takes all of the existing directories and moves them into
      their new homes. See doc/UPDATING for instructions.
      1855897b
  22. 16 Oct, 2008 1 commit
  23. 09 May, 2008 1 commit
    • Kevin Atkinson's avatar
      Make project approval mail truly anonymous. Also make membership · 503bb661
      Kevin Atkinson authored
      acceptance email truly anonymous.  A few other emails related to
      project membership are still not anonymous though.  New function
      AnonSENDMAIL in libtestbed which will try to make sure there is no
      trace of the current user in the mail sent.
      
      For now, stop sending membership approval related email to the project
      admin list since this will also go to testbed-approval.  There is also
      some code to remove testbed-approval from the proj-admin list after
      the acceptance email but this is disabled for now since some times people
      reply to the approval email.
      503bb661
  24. 24 Oct, 2007 1 commit
  25. 10 Sep, 2007 1 commit
  26. 21 Aug, 2007 1 commit
    • Leigh B. Stoller's avatar
      Another round of widearea node hacking for CMU. These changes add · 99346dc0
      Leigh B. Stoller authored
      widearea reloading support.
      
      * New slot in the images table to store an access key which remote
        sites must provide in order to download an image (via https).
      
      * tmcd returns a different kind of ADDRESS field from doloadinfo.
        Instead of the multicast stuff, return a URL that points to boss'
        web server. The URL is of the form:
      
         https://www.myemulab.net/spewimage.php?imageid=10013&access_key=abcdef
      
        which as you can see is fully specified; the client does not need
        to know anything else.
      
      * New webpage and backend scripts appropriately called "spewimage"
        which also includes support for the http HEAD request (from wget) to
        avoid downloading images that are already on the node. I just
        learned about this HEAD request stuff today ... but otherwise these
        operate as expected, spewing the image if the access key is provided.
      
      * Changes to rc.frisbee to deal with remote loading. In addition to
        URL support, I also added support for simple paths, the intent being
        that we will probably distribute images offline (say, at night) so
        that when a node reboots it doesn't actually have to wait 60 minutes
        for an image to download. I have not added any server side support
        for this yet though. Maybe later this week.
      
      * Other bits and pieces and fixes to make this work.
      99346dc0
  27. 17 Aug, 2007 1 commit
    • Leigh B. Stoller's avatar
      New widearea node checkin stuff for CMU. This stuff is quite a bit · f3f0fa98
      Leigh B. Stoller authored
      different then the original widearea code. Simpler, less dynamic.
      
      First off, the wanodecreate script creates a new widearea_nodeinfo
      entry.  These are nodes that will later checkin and be created as a
      real node.  The input is a little xml file that you can use to specify
      the stuff in the table entry (city, state, zip, etc). You can also
      provide a privkey (no more then 64 chars), or one will be generated
      for you.  For each one of these, create a Dongle Boot and stash the
      privkey as /etc/emulab/emulab-privkey on the dongle. You do not assign
      the IP address; the node will tell us that when it checks in.
      
      A node checks in like this:
      
      	https://$bossname/wanodecheckin.php?IP=$IP&privkey=$privkey
                     &hostname=$hostname
      
      The web page is simply a stub that makes sure the arguments don't have
      any illegal characters, and then passes off to the backend.
      
      The backend script checks the privkey and finds the widearea_nodeinfo.
      The first time the node checks in, the node is created (db/Node.pm)
      (nodes table, interfaces table, etc), and the node is moved to hwdown.
      Subsequent checkins watch for changes to the IP or hostname, and issue
      named_setup calls as needed.
      f3f0fa98
  28. 07 May, 2007 1 commit
    • Leigh B. Stoller's avatar
      Mostly this commit is the switch from SVN archives to ZIP archives. · 55d1bb6e
      Leigh B. Stoller authored
      Other stuff leaked in too ...
      
      I did separate out a lot of tbsetup/libArchive into db/Archive, and
      whats left in libArchive.pm will eventually move over into the
      Template library.
      
      Note that I have dropped archiving of plain experiments; this is not
      really owrth it outside the workbench context, and it just wastes
      space and makes a lot if stuff painful in the web interface.
      55d1bb6e
  29. 15 Feb, 2007 1 commit
  30. 22 Jan, 2007 1 commit
    • Leigh B. Stoller's avatar
      Add a setuid utility script that will chown a directory tree · 42e84c26
      Leigh B. Stoller authored
      (recursively) to the UID of the real user (not effective) user. The
      user must have write permission on the enclosing directory.
      
      Currently, this script is is called from the Archive code, when
      copying in files, to avoid permission errors when the current user is
      not the same as the previous user.
      
      This script can also be used from several other places that have
      exhibited similar directory permission problems.
      42e84c26
  31. 18 Jan, 2007 2 commits
  32. 14 Dec, 2006 1 commit
  33. 21 Nov, 2006 2 commits
  34. 25 Oct, 2006 1 commit
    • Leigh B. Stoller's avatar
      Makefile Whacking! Try to deal with the problem caused by the delay · 7590f9c5
      Leigh B. Stoller authored
      between when something is installed and when post-install runs. Short
      of a global lock (which we probably need anyway someday), my solution
      is this. In your makefiles, add these variables before the line that
      has the include of $(TESTBED_SRCDIR)/GNUmakerules:
      
      	SETUID_BIN_SCRIPTS   =
      	SETUID_SBIN_SCRIPTS  =
      
      I have added three new rules to GNUmakerules that look like this:
      
      	$(addprefix $(SBINDIR)/, $(SETUID_SBIN_SCRIPTS)): $(SBINDIR)/%: %
      		echo "Installing (setuid) $<"
      		-mkdir -p $(INSTALL_SBINDIR)
      		$(SUDO) $(INSTALL) -o root -m 4755 $< $@
      
      Yep, your eyes ain't lying to you; use sudo to run the target so that
      install does the right thing (which is that the old file is not
      replaced until the new one has the proper attributes on it).
      
      Note that post-install is still needed for the initial install, but
      should no longer be needed for day to day installs since all that other
      stuff post-install does is mkdir/chmod on directories.
      7590f9c5
  35. 05 Oct, 2006 1 commit
    • Leigh B. Stoller's avatar
      More work on "recording" template events. · e9607a77
      Leigh B. Stoller authored
      * New version of template_record just for ops, since so much is
        different about ops, not bothering to maintain a single version.
      
      * Various fixes to how the recorded events are stored and reconstituted.
        The big fix is to wrap them in a sequence to that they get fired
        properly (waiting for completion of previous event in recording).
      
      * New buttons to Pause and Continue event time, which is used when
        adding recorded events. This allows users to pause time while they
        "think" so when an event is recorded, the thinking time is not actually
        in the timeline. Eventually hope to figure this out automatically, but
        that will take some real, uh, thinking.
      
      * Add a new event editor (linked off the template page) that allows
        you to delete and change the recordings. Note that you can only edit
        the events at the template level; you cannot edit the events of an
        instance (swapped in experiment), and you can only edit the recorded
        events, not any other events. Not sure its useful to be able to do
        either of these yet, but probably not too hard to add at some point.
      e9607a77
  36. 12 Sep, 2006 1 commit
  37. 01 Jun, 2006 1 commit
    • Leigh B. Stoller's avatar
      Add suport for building per project, group, experiment DBs on ops. At · adbcfd47
      Leigh B. Stoller authored
      present the per-experiment stuff is not hooked in, but will be for
      templates later. Anyway, each user gets a mysql account on ops, with
      password set to the same as their mailman password (which is also
      their jabber password, etc). Each project gets a DB named by the
      project, and each group gets a DB named by pid,gid. Users are placed
      on the access lists for the DBs as you would expect.
      
      There is a little bit of complexity to make sure that we can create
      DBs on ops outside the Emulab path and grant access to them, without
      Emulab getting confused or mucking things up.
      
      I'll get a news item done ...
      adbcfd47
  38. 31 Jan, 2006 1 commit
    • Kirk Webb's avatar
      · fb36443f
      Kirk Webb authored
      Added trunk stats lookup (given a member port) to and added snmp-if-deref.sh
      to the CVS repo.
      fb36443f