1. 23 Aug, 2016 3 commits
  2. 22 Aug, 2016 1 commit
  3. 19 Aug, 2016 3 commits
  4. 18 Aug, 2016 5 commits
    • David Johnson's avatar
    • David Johnson's avatar
    • David Johnson's avatar
      Best effort to decrease dhclient latency on Ubuntu 16. · 5bc94e54
      David Johnson authored
      (AFAIK, nothing has changed about dhclient, etc... I just noticed on the
      Emulab d820s (which have BCM5720s, tg3 driver), the driver takes
      anywhere from 7-9 seconds to simply init the card and autoneg with the
      switch (I've seen worse times, too, i.e. 19 and 29 seconds!).)
      dhclient is happy to start sending requests on interfaces that have no
      carrier (gee, did it ever seem like a good idea to make that behavior
      optional???).  Thus, if we get stuck with a control net NIC that has a
      horribly long init/autoneg time, dhclient is far into its backoff
      strategy on the control net, when it doesn't need to be!  In addition to
      slow media negotiation, there are STP auto things that can further delay
      the forwarding state of a switch port (like the ProCurve "auto-edge"
      port setting that causes the switch to wait 3 seconds after media
      negotiation for a BPDU).  So we have to be a little smarter about
      bringing up the control net via DHCP.
      So to combat these possible scenarios, we try two main things.
      First, we modify findcnet to wait for one of two things to be true
      before we start dhclient at all (or until a 6-second timeout is
      reached): 1) if we have a previous control net device in
      /var/lib/dhcp/dhclient.leases, we wait for that to come up; or 2) if we
      don't have a previous control net device (i.e. first boot of an image),
      we wait for at least one device to obtain carrier.  We could increase
      the 6-second timeout, but we'll wait on that for now; this should be
      good for now.
      Second, we set initial-delay and initial-interval both to 3 seconds in
      dhclient.conf ; hopefully this will give STP protection schemes a chance
      to have gotten things straight by the time dhclient makes its first
      I tried adding a forced 'ifconfig <X> up' to the udev interface handler
      script, just to try to kick the device into autoneg mode ASAP, but of
      course that didn't help anything.
      I cannot improve on this unless we move to a split, managed dhclient
      scheme, where we actually run a dhclient for each interface, and control
      the backoff time much more tightly.  For now, I don't want to do either
      of these things.
    • Mike Hibler's avatar
      Yet another new version. · 5b941237
      Mike Hibler authored
    • Robert Ricci's avatar
      Fix mail sending when $mailfrom is not set · 27ff9193
      Robert Ricci authored
  5. 17 Aug, 2016 7 commits
    • Mike Hibler's avatar
      Updates for soon-to-be FreeBSD 11.0. · becc02e5
      Mike Hibler authored
    • David Johnson's avatar
    • David Johnson's avatar
      Update the Centos 7 passwd files more, for systemd this time. · d2e3d707
      David Johnson authored
      This time systemd-tmpfiles-setup.service was broken ; just reinstalled
      systemd (fortunately the same version) and harvested the new master files.
    • David Johnson's avatar
      Add Centos 7-specific passwd files. · f8d130c0
      David Johnson authored
      polkitd was missing, and this causes systemctl to hang for ~30 seconds
      when doing some of its operations (and of coure polkitd does not run).
      I fixed this by doing 'yum reinstall polkitd' and saving new masters in
    • David Johnson's avatar
    • David Johnson's avatar
    • David Johnson's avatar
      Use the last resort systemd swap strategy: wrap the systemd-fstab-generator. · 828e7acd
      David Johnson authored
      As noted in a previous commit, systemd generators run at boot, with the
      rootfs mounted read-only, prior to systemd reading any unit files on
      disk.  The systemd-fstab-generator reads /etc/fstab and generates units
      for any devices in there, including swap devices.  Thus, the only way to
      ensure our swap device fixing happens correctly (and we have to do this
      in case the image was loaded from an old MFS that might write a swap
      device with /dev/hda or whatever into /etc/fstab), and doesn't race with
      systemd, is to preempt it.
      So now our generator runs the systemd generator each boot, but it
      post-processes the auto-generated unit files, removing them if any of
      those devices was 1) an Emulab auto-generated swap device, and 2) it
      does not exist.
      Moreover, if there is *no* swap device in /etc/fstab, we hunt down the
      canonical Emulab swap partition on the root disk, and mkswap it, and
      generate a unit for it.  Of course, we don't use the system template (it
      is an m4 file in the src dist, so we can't), but hopefully the template
      won't change much --- and it's basic.
      We now only mkswap devices if they don't appear as swap devs via blkid.
      To clean up old Emulab systemd images, do these steps:
        $ systemctl disable emulab-fstab-fixup.service
        $ make client-install
        $ rm -f /usr/local/etc/emulab/initscripts/emulab-systemd-swaps
      I think that will do it.  Worked for me on Ubuntu 16 and Centos 7.
      Here's the comments from the generator for posterity:
        This is a systemd generator that wraps systemd-fstab-generator.
        First, we always run the system generator; and any swap file units
        it generates are modified to contain a
        Before=emulab-fstab.fixup.service dependency to ensure the fixup
        script never races with legitimate systemd unit targets.
        If it has already run on an Emulab node (and if the second-stage
        fixup script (emulab-fstab-fixup.service) has also run), we don't
        run it again; we just run the system fstab generator directly,
        modify the generated swap unit files to run before the fixup script.
        Otherwise, on first boot of an Emulab image, we run the system
        generator and let it generate swap units for any swap devices that
        were in /etc/fstab.  We then check all the auto-Emulab-added swap
        device entries in /etc/fstab, and if that device does not exist, we
        remove the auto-generated unit files/symlinks that correspond to it
        (they are in $1).  We cannot edit /etc/fstab ourselves because we're
        run prior to remount-/-rw; so we make a copy of it in /run/emulab
        and move the copy to /etc/fstab in emulab-fstab-fixup.service ---
        the second part of our solution (the fixup service).
        NB: we do not remove user-added swap devices, even if they are
        We check to see if each auto-Emulab-added swap device that was in
        /etc/fstab is currently a valid swap device (i.e., that it shows up
        via blkid with a TYPE="swap").  If it is valid, we *do not* run
        mkswap to create it!  If it is not currently a valid swap device,
        but if the partition is marked as a Linux swap partition, we will
        run mkswap on it to ensure it is valid by the time systemd runs
        swapon on it.  If we *did* plan to run mkswap, that can be prevented
        by creating the
        /etc/emulab/emulab_systemd_fstab_generator_never_mkswap file.  This
        gives the user the ability to create a disk image where swap
        partitions are not created/wiped on first boot of the image.  I
        can't see a use case for this, but it's easy to do.
        If there is a swap partition on the device containing the root
        partition, and if it is not already in /etc/fstab, we try to add a
        unit ourselves for that swap partition.  We do it in the new style,
        too, by using its UUID, in this case.
        When scanning /etc/fstab to find auto-Emulab-added devices, we look
        for a comment above a line containing a swap device, and the comment
        must match the regexp /^#.*the following.* added by / .  Then if the
        line below the comment refers to an invalid swap device, we remove
        the unit files that correspond to the device.  Otherwise
  6. 16 Aug, 2016 1 commit
  7. 15 Aug, 2016 1 commit
  8. 14 Aug, 2016 2 commits
  9. 12 Aug, 2016 2 commits
  10. 11 Aug, 2016 4 commits
  11. 10 Aug, 2016 2 commits
    • Mike Hibler's avatar
      Rejiggered reload_daemon to enforce a max time. · b6d272a2
      Mike Hibler authored
      There are now some sitevars to control its behavior, the one of interest here
      is reload/failtime:
      The way the reload daemon is supposed to work now is that nodes will be
      started on their reloading adventure with an os_load. If they are still there
      after reload/retrytime minutes, then they will either be rebooted (if the
      os_load was successful) or os_load'ed again (if the first os_load failed
      outright). The logic for either of these is that there might have been some
      transient condition that caused the failure. If we do have to perform this
      "retry" then we will send email to testbed-ops if reload/warnonretry is set.
      If, after another reload/retrytime minutes, a node is still there, then the
      node will be sent to hwdown, possibly powering it off or booting it into the
      admin MFS depending on the setting of reload/hwdownaction.
      So really, reload/failtime should not be needed. All node should exit
      reloading in 2 * reload/retrytime minutes. But it is there as a b...
    • David Johnson's avatar
      Make sure udev settles enough to get us network interfaces. · 58db6edd
      David Johnson authored
      Ok, it seems that sometimes the network.target runs before network
      devices have fully finished going through udev.  I think what goes on
      here is that udev can "settle" (meaning there are no events), but there
      will still be some events in the future.
      So now in the special networking-emulab.service, we settle AND wait for
      at least one auto, non-lo interface to appear via ifquery.
  12. 09 Aug, 2016 9 commits
    • David Johnson's avatar
      Attempt to safely work around systemd swap service on Ubuntu 16. · 7afd5f24
      David Johnson authored
      systemd.swap is one of its special builtin services.  Basically, swap
      devices are parsed out of fstab, or by examining a disk's GPT.  Any such
      devices are turned into instantiated units.  This happens via the
      systemd-fstab-generator.  Generators in systemd are almost
      uncontrollable.  They run immediately, prior to on-disk unit file
      parsing, and all you can do is disable or replace them.  You cannot
      express dependencies on the resulting units (unless you write your own
      generator).  Generators also run in an impoverished environment (think
      read-only /etc), so we cannot just add another generator that does
      basically what fixup-fstab-swaps does.  Finally, we cannot write a
      template unit file for all swap devices (we would use this to inject a
      blocking dependency so that these swap units don't conflict with us).
      Lennart has recognized the value in this, but thought the impl effort is
      pretty hard.  This makes sense, because the generators run prior to unit
      file load from disk (and presumably that would nix templates for
      generated units)... and I gather there are other problems as well.
      This is quite problematic for us because we rely on the ability to
      update /etc/fstab with the name of the real swap device, and to mkswap
      on it.  However, on machines with lots of cores, systemd is at its
      parallelizing best, and inevitably systemd tries to start up one of its
      instantiated swap device units at the same time as our fixup-fstab-swaps
      script is running.
      So I've done several things to try to deal with this situation.  First,
      this Ubuntu 16-specific version of fixup-fstab-swaps no longer adds a
      swap line to fstab with options=defaults -- instead it uses
      options=noauto,x-emulab-auto .  The noauto causes systemd's instantiated
      swap units to not automatically run on boot (don't worry, they become
      active if fixup-fstab-swaps swapons them, and thus they get swapped off
      prior to umount -- important that happens to avoid hangs); but our
      script will swapon the noauto,x-emulab-auto swap partitions as if they'd
      had options=default|auto.  What this does break is swapon/off -a --- but
      who cares.  The x-* comment option in fstab is something I didn't know
      about, I'll admit.
      Second, I've done is make emulab-fstab-fixup.service Conflict with
      swap.target, but also to be pulled in by swap.target!  The hope was that
      this would ensure that our service *always* runs successfully, even if
      it kills off swap.target to "handle" the conflict.  Well, the problem is
      that we need to Conflict with the instantiated swap unit files, not
      swap.target... so I think that isn't really working.  But I left it in
      -- maybe it is helping us win races.
      The one thing I cannot block is that systemd looks at the partition
      types of at least one of our hardware types (d820) and generates swap
      unit files by the partition UUID.  How it is doing this, I have no idea
      -- that behavior is only supposed to happen if your disk is GPT.  So we
      get failures on the d820s from the systemd instantiated swap units on
      first boot, but our scripts always do the right thing.
    • Leigh B Stoller's avatar
      Watch for unqualified interface names (not node_id:iface_id). Mostly · aabf7bb1
      Leigh B Stoller authored
      we see that kind of interface naming and that is how geni-lib does it.
      But need to accept that syntax.
    • David Johnson's avatar
      Control yet another systemd/udev race/dependency on Ubuntu 16. · 3ed87a0f
      David Johnson authored
      The stock Ubuntu 16 networking.service only runs `udevadm settle` if
      there are 'auto ...' stanzas in /etc/network/interfaces .  Well, we got
      rid of that a few commits ago, and now let udev rules populate
      /etc/network/interfaces (really /run/emulab-interfaces.d-auto-added/*).
      So, it's either hack the networking.service unit file to force udev to
      settle, and have it blown away on package update; or add a
      networking-emulab.service that has to run before networking.service to
      force udev to settle.  We *always* want udev to settle on any Emulab
      node before bringing up interfaces, just in case the control net NIC is
      slow for whatever reason.
    • Mike Hibler's avatar
    • David Johnson's avatar
      Allow the systemd fstab swap fixer to mkswap/swapon! · 492a748d
      David Johnson authored
      I cannot find why we called fixup-fstab-swaps with '-E' (which means,
      don't mkswap/swapon any swaps).  The only thing I can think of is that
      perhaps running swapon manually made the systemd dev-*.swap targets
      unhappy.  However, it is necessary to mkswap if the swap device didn't
      exist, because systemd will not mkswap for you, AFAIK -- it will only
      swapon.  On Ubuntu 16, the dev-*.swap targets are happy whether they or
      Emulab does the swapon.  If that's not true on Centos 7 or other
      systemds, we may have to revisit this tweak.
    • David Johnson's avatar
    • Mike Hibler's avatar
      Duh! FreeBSD prepare does not need to remove Linux-generated swap lines! · d3da3fd1
      Mike Hibler authored
      Put that code in the Linux prepare instead.
    • Mike Hibler's avatar
      Fix /etc/fstab swap line removal for Linux images. · 7186318f
      Mike Hibler authored
      The comment line is different.
    • Mike Hibler's avatar
      New sitevars for reload daemon. · c7f6e63d
      Mike Hibler authored