All new accounts created on Gitlab now require administrator approval. If you invite any collaborators, please let Flux staff know so they can approve the accounts.

  1. 29 Oct, 2020 1 commit
  2. 21 Sep, 2020 1 commit
    • Leigh B Stoller's avatar
      Beat up on Mike's powerlocal a bit: · 4cab49b5
      Leigh B Stoller authored
      * The mibs live in a different place (/var) so the path stuff in the
        raritan file needed some changes.
      * Need packages: snmp snmp-mibs-downloader libsnmp-perl
      * Need PDU2-MIB.txt from the Mothership, to /var/lib/snmp/mibs/ietf
  3. 18 Sep, 2020 1 commit
    • Leigh B Stoller's avatar
      Attempt to rebuild the house of cards: · 6bba15b5
      Leigh B Stoller authored
      Last time we talked, I told you about a long standing change we needed
      wrt delay node handling. That change stopped considering control network
      interfaces in the list of possible interface speeds, as well as
      interfaces that are not even wired up. This was bad, since well, it
      makes no sense. Say we have a cluster with 1G control interfaces, but
      all of the experimental interfaces are 10G. The mapper would think there
      are 1G links for delay node handling and best effort (we would put the
      minimum required speed into the vtop and let assign choose an interface).
      The consequence of this at the Moonshot cluster is that a best effort
      lan of many (say, 40) nodes would map using any set of nodes scattered
      across the chassis' cause we were using 1G in the vtop file. But after I
      made the change above, some experiments stopped mapping cause the ptop
      was now getting 10G link speeds cause that is now the minimum, but there
      is not enough interswitch BW to support a lan that big when it is
      scattered across too many switches. Which is the whole reason for best
      So now for a few new changes.
      * Make best effort a first class flag, not a combo of things (speed=10,
        nobwshaping=1, uselinkdelay=0), on the lan database table. I've wanted
        to do this for a while.
      * Now best effort is handled in the mapper, currently by putting in
        a 1G bandwidth in the vtop, and assign will pick any interface 1G or
        better. This gets us back to where we were before the changed described
        in the first paragraph above. Well except for the pc3000s on the
        Mothership, they have a mix of 100Mb and 1Gb interfaces, but I am going
        to wait and see if it actually causes problems for anyone.
        But to make this really correct, we need to change the CM to pass through
        something other then "10" BW for all best effort links, since what we
        really want is for the user to tell us a true minimum they need, so that
        we can pick the lowest speed physical devices required. But this will
        take some more thinking time. And it affects the problem with interswitch
        bandwidth above.
      * Get rid of an inconsistency in linkdelay handling; at the experiment
        level we have use and force flags for linkdelays. But at the lan
        level we have just a use flag, and we were treating that as force. So
        I added a separate force flag to make things consistent, since with
        best effort we want to turn of BW shaping but otherwisw link delay is
        optional unless needed for some other reason (delay, loss).
  4. 16 Sep, 2020 1 commit
  5. 08 Sep, 2020 1 commit
  6. 01 Sep, 2020 1 commit
  7. 26 Aug, 2020 1 commit
  8. 17 Aug, 2020 1 commit
    • Leigh B Stoller's avatar
      Work on issue #578 (cross aggregate root ssh); do not generate the per · 9c73757e
      Leigh B Stoller authored
      experiment keys for begni experiments, wait to see if keys are provided
      in the CreateSliver() call. And of course, the Portal does provide a per
      experiment ssl key pair. Use that to derive the root ssh key pair. The
      nice thinig about this is, is all nodes in a multisite experiment will
      have the same root ssh key pair installed.
  9. 14 Aug, 2020 1 commit
    • Leigh B Stoller's avatar
      Deal with (well, avoid sometimes) a long standing mapper problem wrt · 482b74ad
      Leigh B Stoller authored
      delay nodes:
      The decision to inject a delay node is based on whether we have nodes
      that support the speed requested. If you ask for 200MB, well of course
      we don't. But if you ask for 100MB and some of the nodes have 100MB
      interfaces, then we do not put in a delay node, cause the mapper will
      choose those nodes. What if they are not free? Well that is one of the
      problems caused by this approach. OK fine.
      The code that was look at all the interfaces available, was doing two
      things wrong:
      1) It was considering both control and experimental interface. No idea
         why, the mapper never deals with control interfaces. So got rid of
      2) Every cluster ends up with lots of interfaces that are not wired
         up, but we were still considering those. So join against the wires
         table to see if they are actually usable.
      I expect this change to have no impact on most clusters. On the
      Mothership, it might be a mess, we have zillions of interfaces and
      whacky devices, and well the Mothership always acts differently.
  10. 13 Aug, 2020 1 commit
  11. 04 Aug, 2020 1 commit
  12. 28 Jul, 2020 1 commit
  13. 22 Jul, 2020 2 commits
    • Leigh B Stoller's avatar
      Fix some problems with admission control wrt individually reservable · 574de3ea
      Leigh B Stoller authored
      * Since node_ids are now in the reservation (as the type), watch for
        this and do admission control on the node, previously we were letting
        this get to assign, where the error is, well, obscure.
      * Related to above, when asking for a reservable node, check to see if
        its already allocated before letting it get to assign. Better error
        message, and faster to return to the user.
      * Attempt to deal with a backwards compatibility issue wrt the
        matrix (but is a general issue); there are numerous profiles that ask
        for matrix nodes (or mmimo1, iris1, iris2) by their type since that is
        how they were reserved. But when we changed them to reservable nodes,
        existing profiles that ask for them by type suddenly broke cause
        1) the user wants a specific nuc, not any nuc, and 2) say in the case
        of mmimo1, there was only one node of that type but now admission
        control thinks there are no nodes of that type since individually
        reservable overrides that.
    • Leigh B Stoller's avatar
      Fix problem when using P2PLTE links in an experiment using virt · ba5726ab
      Leigh B Stoller authored
      nodes (or more generally, multiplexed links); force link encap to
      default, emulated to off, nobwshaping to off.
  14. 09 Jul, 2020 1 commit
    • Leigh B Stoller's avatar
      Work on better URLS for issue powderrenew/powder#140 · e5b6df33
      Leigh B Stoller authored
      Instead of requiring a user to start a one node experiment to get their
      syncthing tarball, send a URL that downloads the URL directly.
      The URL is to boss, which does the authentication and authorization, and
      then returns a redirect to "signed URL with expiration" over on
      ops (since we want to stream potentially big tarballs from ops directly).
      On ops there is a cgi handler that verifies the signature on the url and
      the expiration (currently 10 seconds), and then streams the file. This
      is a very common approach, and is quite similar to how we do the console
      access dance between boss and ops, but the console implementation is a
      bit less standard (might want to change that someday). This new approach
      can be used for streaming any kind of file that first needs
      authorization from boss, but can withstand using a URL that is
      fundamentally inappropriate for anything that needs perfect access
      control (in other words, we are not lawyers).
      The URL that needs to go into the syncthing email looks like:
  15. 06 Jul, 2020 1 commit
  16. 26 Jun, 2020 3 commits
  17. 17 Jun, 2020 2 commits
  18. 09 Jun, 2020 1 commit
  19. 03 Jun, 2020 2 commits
  20. 26 May, 2020 1 commit
    • Leigh B Stoller's avatar
      What started as a small project to support B/LAGGs on stitching links, · 02a1c736
      Leigh B Stoller authored
      but developed into a giant BAGG of problems:
      For the first part, the complication stems from representing stitching
      links as a fake node with an interface. In general this is fine since up
      to now, every stitching link has been a plain wire, and so snmpit does
      the right thing cause the port is in tagged mode. But this breaks down
      when the link is actually an aggregate (LAG/BAGG) since snmpit needs to
      take a different path for that, setVlanOnTrunks2(), which operates on all
      of the trunk links between switches, and knows how to deal with link
      aggregation. Trying to convince snmpit to handle fake switches and
      trunks with only one end, seemed like a bad idea. So I opted for adding
      a LAG field to the interfaces table so we can mark those interfaces. And
      I changed snmpit_stack to look for those ports, and redirect them onto
      the trunk path.
      This worked great on our Dell switches, but not on scidmz (an HP). Which
      was strange cause it was failing in an identical situation that seemed
      to work fine on our Moonshot HP (bighp1).
      Many hours later ... we determine that Version 7 firmware has a
      different ifindex mapping for BAGGs, and that we have been lucky not to
      have hit that problem on the moonshot cluster. Many more hours later,
      Kirk discovered that the very recent firmware update to scidmz resulted
      in snmp no longer being able to change the membership of BAGGs. Holy Bat
      The best alternative was to use the libNetconf module that is already
      used to speak CLI to the switch for OpenFlow configuration. Just
      manipulate the BAGG with via the CLI, the commands are pretty simple.
      Well, that didn't quite work cause scidmz does not allow password based
      authentication (at some point it might have), and the ssh key that
      scidmz does accept is in /root/.ssh/, and snmpit runs as the user. No
      problem, lets just add another key pair and stick that in /usr/testbed/etc
      where the user can access it. But, ssh will not allow a 644 private
      key file to be used. So ... copy the that file to /tmp (so that the
      user owns it) and chmod it to 600, and pass that filename down into the
      libNetconf module, which has been changed to optionally use an ssh key
      So to sum up, there are two new node_attributes set on the stitching
      aggregate for scidmz:
      * snmpit_badBAGG: which says the firmware no longer allows snmpit to
        manipulate BAGGs.
      * snmpit_sshkey: which is the path to the ssh key for libNetConf,
        instead of password based authentication.
      The bottom line is ... do not upgrade our other HP switches.
  21. 20 Apr, 2020 1 commit
  22. 19 Apr, 2020 1 commit
  23. 15 Apr, 2020 2 commits
  24. 07 Apr, 2020 1 commit
    • Leigh B Stoller's avatar
      The other part of what Mike did in commit bab624d5 (PREPARE=2): · d854028a
      Leigh B Stoller authored
      If a node (or nodetype) has the delayreloadtillalloc attribute set, then
      in nfree the partitions are cleared and the node is powered off. Reload
      is being delayed till the next time the node is allocated to an
      Add -Z option to os_load, which is like -P, but applies the clear
      metadata option to all disks.
      os_setup and os_load have been changed accordingly.
  25. 26 Mar, 2020 1 commit
  26. 24 Mar, 2020 1 commit
  27. 18 Mar, 2020 1 commit
    • Leigh B Stoller's avatar
      Fixed the triple problem. This was really messing up the battery · 732f9a52
      Leigh B Stoller authored
      monitor, since what was really happening was we were leaving unconsumed
      output from the arduino, and the sync code was never really syncing, it
      was just spitting the command over and over, and reading only one
      Also added "use strict" and fixed all the warnings/errors.
  28. 16 Mar, 2020 2 commits
  29. 11 Mar, 2020 2 commits
  30. 05 Mar, 2020 1 commit
  31. 29 Feb, 2020 1 commit
    • Mike Hibler's avatar
      Don't report DHCP on the mgmt interface as a booting event. · 8c797bc1
      Mike Hibler authored
      On machines that DHCP on their management interfaces (e.g., at UMass),
      this can cause seeming random reboots of machines that are free. They
      (actually stated) get an unexpected BOOTING event which starts a
      timeout waiting for further state transitions that never happen, and
      eventually stated power cycles the nodes.
  32. 11 Feb, 2020 1 commit