1. 12 Dec, 2017 1 commit
    • David Johnson's avatar
      Add Linux exp firewall support for virt_node_public_addr addresses. · 798f9b6f
      David Johnson authored
      A new tmcd command, publicaddrinfo, just dumps the relevant bits of
      virt_node_public_addr to any node in an experiment that has addrs
      allocated (we don't want to restrict based on calling node_id or
      pool_id).
      
      Then the generic getfwconfig() function calls that, and sets some bits.
      I also extended this function to add some dynamic clientside vars
      (EMULAB_DOMAIN, EMULAB_EXPDOMAIN, EMULAB_PUBLICADDRS) so that user
      firewall rule writers can use them to refer to the control net IPs of
      nodes in their experiment (i.e., node-0.EMULAB_EXPDOMAIN); and so that
      rules can be written over EMULAB_PUBLICADDRS -- a command-delineated
      list of IP addrs).
      
      Finally, I extended the Linux firewalling code to allow any experiment
      node to answer ARPs for the public IP addresses; we can't know a priori
      which node should answer -- and it could change.
      
      This closes #353 .
      798f9b6f
  2. 05 Dec, 2017 1 commit
  3. 17 Nov, 2017 1 commit
  4. 26 Oct, 2017 1 commit
  5. 07 Aug, 2017 1 commit
    • Dan Reading's avatar
      Issue #316 emulab/emulab-devel · c5ce9d4c
      Dan Reading authored
      In checknode code for FreeBSD don't check the /dev/ad* device if it is a symlink.
      [I think the a error in the test command for -c]
      c5ce9d4c
  6. 26 Jul, 2017 1 commit
    • Mike Hibler's avatar
      Support for per-experiment root keypairs (Round 1). See issue #302. · c6150425
      Mike Hibler authored
      Provide automated setup of an ssh keypair enabling root to login without
      a password between nodes. The biggest challenge here is to get the private
      key onto nodes in such a way that a non-root user on those nodes cannot
      obtain it. Otherwise that user would be able to ssh as root to any node.
      This precludes simple distribution of the private key using tmcd/tmcc as
      any user can do a tmcc (tmcd authentication is based on the node, not the
      user).
      
      This version does a post-imaging "push" of the private key from boss using
      ssh. The key is pushed from tbswap after nodes are imaged but before the
      event system, and thus any user startup scripts, are started. We actually
      use "pssh" (really "pscp") to scale a bit better, so YOU MUST HAVE THE
      PSSH PACKAGE INSTALLED. So be sure to do a:
      
          pkg install -r Emulab pssh
      
      on your boss node. See the new utils/pushrootkeys.in script for more.
      
      The public key is distributed via the "tmcc localization" command which
      was already designed to handle adding multiple public keys to root's
      authorized_keys file on a node.
      
      This approach should be backward compatible with old images. I BUMPED THE
      VERSION NUMBER OF TMCD so that newer clients can also get back (via
      rc.localize) a list of keys and the names of the files they should be stashed
      in. This is used to allow us to pass along the SSL and SSH versions of the
      public key so that they can be placed in /root/.ssl/<node>.pub and
      /root/.ssh/id_rsa.pub respectively. Note that this step is not necessary for
      inter-node ssh to work.
      
      Also passed along is an indication of whether the returned key is encrypted.
      This might be used in Round 2 if we securely implant a shared secret on every
      node at imaging time and then use that to encrypt the ssh private key such
      that we can return it via rc.localize. But the client side script currently
      does not implement any decryption, so the client side would need to be changed
      again in this future.
      
      The per experiment root keypair mechanism has been exposed to the user via
      old school NS experiments right now by adding a node "rootkey" method. To
      export the private key to "nodeA" and the public key to "nodeB" do:
      
          $nodeA rootkey private 1
          $nodeB rootkey public 1
      
      This enables an asymmetric relationship such that "nodeA" can ssh into
      "nodeB" as root but not vice-versa. For a symmetric relationship you would do:
      
          $nodeA rootkey private 1
          $nodeB rootkey private 1
          $nodeA rootkey public 1
          $nodeB rootkey public 1
      
      These user specifications will be overridden by hardwired Emulab restrictions.
      The current restrictions are that we do *not* distribute a root pubkey to
      tainted nodes (as it opens a path to root on a node where no one should be
      root) or any keys to firewall nodes, virtnode hosts, delay nodes, subbosses,
      storagehosts, etc. which are not really part of the user topology.
      
      For more on how we got here and what might happen in Round 2, see:
      
          #302
      c6150425
  7. 06 Jul, 2017 1 commit
  8. 03 Jul, 2017 2 commits
  9. 01 Jul, 2017 1 commit
  10. 22 Jun, 2017 1 commit
  11. 21 Jun, 2017 1 commit
  12. 19 Jun, 2017 3 commits
  13. 30 May, 2017 1 commit
  14. 18 May, 2017 1 commit
  15. 02 May, 2017 1 commit
    • David Johnson's avatar
      Fix a race in common/mkvnode.pl vnodeCreate. · 345bf9bd
      David Johnson authored
      safeLibOp blocks all our vnodesetup-related signals from interrupting
      libvnode ops to ensure at least op-level consistency.  However, there
      was an opportunity for signals to sneak in, in between a successful
      vnodeCreate and the writing of the vnode.info file (that mkvnode.pl uses
      to know if the vnode was created or not).
      
      So I redid safeLibOp to make blocking signals optional (of course it's
      on for nearly all calls, except now vnodeCreate, and formerly
      vnodePoll).  Now there's a signal-safe zone all the way around
      vnodeCreate, including a StoreState() before we unblock.  This should
      ensure consistency in that particular spot.  I didn't think about
      whether this affects anything else.
      345bf9bd
  16. 29 Apr, 2017 1 commit
  17. 27 Apr, 2017 2 commits
  18. 26 Apr, 2017 1 commit
  19. 24 Apr, 2017 2 commits
    • David Johnson's avatar
      Clientside Docker vnode support. · 96794781
      David Johnson authored
      See clientside/tmcc/linux/docker/README.md for design notes.
      See clientside/tmcc/linux/docker/dockerfiles/README.md for a description
      of how we automatically Emulabize existing Docker images.
      
      Also, this mostly fits within the existing vnodesetup path, but I did modify
      mkvnode.pl to allow the libvnode backend to provide a vnodePoll wait
      loop instead of the builtin vnodeState loop.
      96794781
    • David Johnson's avatar
      Move fromtopo out of rc.hostnames to genhostslistfromtopo in libsetup.pm. · 3a9765aa
      David Johnson authored
      This allows other callers than rc.hostnames (i.e. docker clientside) to
      generate the hostname list for an experiment.
      3a9765aa
  20. 13 Apr, 2017 1 commit
  21. 11 Apr, 2017 1 commit
  22. 06 Apr, 2017 1 commit
  23. 24 Mar, 2017 1 commit
    • Mike Hibler's avatar
      Semi-hack to ensure that Wisconsin nodes don't include their SSDs · fbe5f38f
      Mike Hibler authored
      in blockstore-related VGs.
      
      Right now, you have to decide globally and in advance, what disk types
      are going to be included in blockstore pools. Then you set the sitevar
      accordingly and then set the DB sysvol/nonsysvol/any node_type_features
      to reflect the amount of storage available on just drives of that type.
      
      This value is passed to clients via the otherwise unused PROTO field
      of the blockstore line (when CMD=SLICE and CLASS=local), so this change
      is backward compatible (OS images with older client code will ignore it
      and just give you blockstores including all the devices).
      
      So at Wisconsin, I set storage/local/disktype to "HDD-only" and tweak
      the node_type_attributes '?+disk_any' and '?+disk_nonsysvol' to not
      include the space for the 1 or 2 SSD drives in each machine. tmcd passes
      the PROTO=HDD-only value and the client sees that and does not include
      any SSD devices among the eligible devices from which to create the VG.
      
      The hope is that ultimately, we could get rid of the sitevar and use the
      PROTO field to select, per-blockstore, its type (only HDD, only SSD).
      But that will require additional per node (type) assign features
      differentiating the amount of each type available.
      fbe5f38f
  24. 09 Mar, 2017 1 commit
  25. 27 Feb, 2017 1 commit
  26. 25 Feb, 2017 1 commit
    • Mike Hibler's avatar
      Fix for local homedirs getting left as owned by root. · 2beb1824
      Mike Hibler authored
      See emulab-devel issue 227 for details.
      
      Also, on a "reset" clean out the correct BDB files. It has been
      a long time since they used ".db" as the suffix. Now there are
      ".pag" and ".dir" files. We haven't noticed because we don't really
      use the "reset" operation. The prepare script just removes
      everything in /var/emulab/db.
      2beb1824
  27. 10 Feb, 2017 1 commit
    • Mike Hibler's avatar
      It is Cleanup Friday! · f624f158
      Mike Hibler authored
      Get rid of ELVIN_COMPAT and CONFIG_OPSVM from elabinelab land.
      These options still exist throughout the install code, didn't touch that.
      f624f158
  28. 01 Feb, 2017 1 commit
  29. 17 Jan, 2017 1 commit
    • Mike Hibler's avatar
      Implement heartbeat/status reports in Frisbee. · 2be46ba4
      Mike Hibler authored
      There are three pieces here, a change to the frisbee protocol itself, an
      Emulab event component to get status back to the portal, and the surrounding
      infrastructure to make it all work.
      
      Frisbee heartbeat messages:
      
      Added a new message type to the frisbee protocol, "Progress". In theory it
      operates by having the server send a multicast progress request to its clients
      which includes an interval at which to report (or "just once") and an
      indication of what to report (nothing, progress summary, or full stats). The
      client then sends unicast "fire and forget" UDP replies according to that
      schedule. However, I took a shortcut for the moment and just added a command
      line option to the client to tell it to report a summary at the indicated
      interval (-H <interval>).  So the server never sends requests.
      
      This is implemented in the client by a fourth thread since I wanted it to
      operate independent of packet reception (which would cause clients to report
      in a highly synchronized fashion due to multicast). The server instance just
      logs progress reports into its log.
      
      This protocol addition should be fully backward compatible as both client and
      server ignore (but log) unknown messages.
      
      Emulab progress report events:
      
      When this is compiled in (-DEMULAB_EVENTS) and turned on (-E <server>), the
      frisbee server instances will send a FRISBEEPROGRESS event to the indicated
      event server for every progress report it receives (in addition to logging the
      events to its own log). Right now it will create an event with key/value pairs
      for the information in a client summary reply:
      
      TSTAMP is the client's time at which it sends the event. Could be used by the
      received to determine latency of the report if it cared (and if it assumed
      that the clocks are in sync). We don't care about this.
      
      SEQUENCE is the report number. Again, could be used by the receiver, in this
      case to detect loss, if it cared. We don't.
      
      CHUNKS_RECV is complete chunks that the client has received from the network.
      CHUNKS_DECOMP is chunks decompressed by the client.  BYTES_WRITTEN is bytes
      written to disk by the client.
      
      Any of the three can be used by the event receiver as an indication of life
      and/or progress. However, only the last would be a reasonable indicator of
      time remaining since it is the last (and slowest) phase of imaging. To
      estimate time remaining we could compare that value to the amount of
      uncompressed data that is in the image. This makes the sketchy assumptions
      that time for writes to the disk are uniform and that the number and distance
      of seeks is uniform, but it is better than a sharp stick in the eye.
      
      Emulab infrastructure:
      
      There is a new sitevar "images/frisbee/heartbeat" which can be set to a
      non-zero value to tell the frisbee MFS to fire off frisbee with -H <value>
      and thus make reports. The default value of zero means to not make reports.
      The tmcd "loadinfo" command sends this through via the HEARTBEAT=<value>
      param.
      
      REQUIRED A TMCD VERSION BUMP TO 41.
      2be46ba4
  30. 12 Jan, 2017 1 commit
  31. 29 Dec, 2016 1 commit
    • Mike Hibler's avatar
      Modernize elabinelab and Emulab install support a bit. · f7e53243
      Mike Hibler authored
      Support FreeBSD 10.3. We will need to be moving to this before long
      as 10.2 EOLs in two days.
      
      Support setup of "Emulab-aware" ZFS use in install scripts. Note that
      the core support code was already done (WITHZFS, WITHAMD). Mostly this
      involves changes to setup either amd (WITHAMD==1) or autofs (WITHAMD==0)
      on the boss node and to NOT add mounts of /{users,groups,proj} to
      /etc/fstab. We still need to add a section to the install documentation
      about setting up a zpool for Emulab to use. There was also a fix to the
      firstuser script which did not do the account setup correctly.
      
      Support setup of ZFS in elabinelab. The elabinelab attributes CONFIG_ZFS
      and CONFIG_AUTOFS are used to convey intent here. Currently they can only
      be used in an "ops+fs" config (e.g., the standard boss and ops config,
      NOT the seperate fs node config). It should work with either the physical
      or virtual node setups:
      
      * For the physical node setup, we actually use local blockstores in the
        ops node config: a SYSVOL blockstore for /usr/testbed and a tiny 1Mib
        NONSYSVOL blockstore. The latter blockstore is not actually used, we
        just make it to force setup of a ZFS zpool which we then use for the
        inner elab.
      
      * For the virtual node setup, we just identify the virtual EXTRADISK
        intended for "/q" and create a zpool on that device.
      
      I would like to change all physical elabinelab setups to use blockstors
      rather than the current hacky mkextrafs usage. But that is a task for
      another day.
      
      Finally, a couple of random changes in elabinelab code: change the
      CentOS image downloaded to CENTOS7-64-STD, increased the default sizes
      of the EXTRADISKS used in the VM config.
      f7e53243
  32. 14 Nov, 2016 1 commit
    • Mike Hibler's avatar
      Tweaks to the agreement between mkextrafs and the blockstore system. · 939b0ae7
      Mike Hibler authored
      For the case in which mkextrafs is used to create local homedirs/projdirs:
      
      Look for the desired mount point (/local) in /etc/fstab and use that if
      it exists (i.e., that FS was already setup by the blockstore system or a
      previous mkextrafs).
      
      Otherwise, look for /var/emulab/boot/extrafs which should contain info
      left behind by the local blockstore setup code indicating a FS or unused
      device to use. For an unused device, rc.storage will identify the largest
      available device that is at least 10MB.
      939b0ae7
  33. 20 Oct, 2016 1 commit
  34. 19 Oct, 2016 1 commit
  35. 14 Oct, 2016 1 commit