1. 30 Mar, 2018 1 commit
    • Mike Hibler's avatar
      Support for frisbee direct image upload to fs node. · 99943a19
      Mike Hibler authored
      We have had issues with uploading images to boss where they are then written
      across NFS to ops. That seems to be a network hop too far on CloudLab Utah
      where we have a 10Gb control network. We get occasional transcient timeouts
      from somewhere in the TCP code. With the convoluted path through real and
      virtual NICs, some with offloading, some without, packets wind up getting
      out of order and someone gets far enough behind to cause problems.
      So we work around it.
      If IMAGEUPLOADTOFS is defined in the defs-* file, we will run a frisbee
      master server on the fs (ops) node and the image creation path directs the
      nodes to use that server. There is a new hack configuration for the master
      server "upload-only" which is extremely specific to ops: it validates the
      upload with the boss master server and, if allowed, fires up an upload
      server for the client to talk to. The image will thus be directly uploaded
      to the local (ZFS) /proj or /groups filesystems on ops. This seems to be
      enough to get around the problem.
      Note that we could allow this master server to serve downloads as well to
      avoid the analogous problem in that direction, but this to date has not
      been a problem.
      NOTE: the ops node must be in the nodes table in the DB or else boss will
      not validate proxied requests from it. The standard install procedure is
      supposed to add ops, but we have a couple of clusters where it is not in
      the table!
  2. 02 Feb, 2017 1 commit
  3. 27 May, 2015 1 commit
    • Mike Hibler's avatar
      Avoid (as best we can) port collisions on the frisbee client/server/uploader. · 0a7fd856
      Mike Hibler authored
      Two different fixes here.
      The first affects frisbeed ("the server") and frisuploadd ("the uploader").
      In both, the master server was choosing the port to use as an obscure function
      of the current value of emulab_indicies frisbee_index without regard to whether
      the port was already in use by someone else.
      To fix this, the "-p <port>" option of both programs has been changed to
      allow a value of 0 to indicate that the program (rather, the kernel) should
      choose the first available port. It will also take a port range (e.g.,
      "-p 50000-50100") which says to find the first available port in that range.
      To communicate The Chosen Port back to the master server, there is a new
      option to frisbeed and frisuploadd, "-A <file>", which says to write the
      address info into the indicated file in the <IP-addr>:<port> format.
      Note that we don't care about the <IP-addr> part since that is just the
      multicast address (frisbeed) or our unicast address (frisuploadd) that we
      pass in to the program. The "Emulab configuration" of the master server uses
      the defs file FRISEBEEMCASTPORT and FRISEBEENUMPORTS vars to determine what
      to pass via the "-p" option. See the comment in defs-example. The "null
      configuration" (aka, on a subboss) just passes "-p 0" to frisbeed.
      The second fix was an attempt to avoid port conflicts on the client side
      (frisbee). There is only so much we can do since all clients of a multicast
      frisbee session have to use the same port, but we can avoid conflicts with
      other UDP apps that bind to INADDR_ANY:<port>.
      We make use of the REUSEADDR socket option and bind specifically to
      <mcaddr>:<port>. This also requires that the server multicast the JOIN
      reply that was previously unicast. Note that use of REUSEADDR will also
      allow multiple frisbee clients on the same host to be in the same session
      (not that we ever do that).
      Since the server is typically updated whenever the Emulab software is, but
      the client is embedded in images and MFSes, there can be pretty much any
      combo of {old,new} server and {old,new} client in the field. So backward
      compatibility was essential and there are a variety of implementation details
      related to that. See the comment in network.c::ClientNetInit().
  4. 13 Jul, 2014 1 commit
  5. 12 Jul, 2014 1 commit
    • Mike Hibler's avatar
      Add config function to canonicalize the image name passed in. · a60cc3f7
      Mike Hibler authored
      This is strictly for the benefit of the Emulab config where you can
      request an image name by either ID or path, e.g. "testbed/FOO" or
      "/proj/testbed/images/FOO.ndz". Previously, simultaneous requests
      under both names would start up two different servers because we didn't
      know they were the same image. Not really a problem for downloads,
      but really not a good thing for uploads!
  6. 09 Jul, 2014 1 commit
    • Mike Hibler's avatar
      Robustness work on the image upload side. · ca73f315
      Mike Hibler authored
      Implement idle ("lack of progress") timeout on the server.
      Timeout work on the client-side too. Implement idle timeouts (not for
      SSL connections yet), fixup handling of over-all timeout. In particular,
      if we are reading from a pipe ('-') don't start timeout til we get some
      input from the pipe.
      Add option to put out progress dots on the client (cuz we are all about
      dots in imagezip/frisbee).
  7. 20 Dec, 2012 1 commit
  8. 19 Sep, 2012 1 commit
  9. 20 Mar, 2012 1 commit
  10. 30 Jun, 2011 1 commit
    • Mike Hibler's avatar
      Make sure upload/download server run with proper group list. · bf97b597
      Mike Hibler authored
      Attempts to load an image from a subgroup of a project didn't work because
      I was only setting the primary group ID based on the node's experiment's
      project. Now we use setgroups() to establish all groups that the swapper
      of the experiment belongs to.
  11. 18 May, 2011 1 commit
    • Mike Hibler's avatar
      Support image PUT (aka, "upload") and assorted minor changes. · 77dbad39
      Mike Hibler authored
      1. Support for PUT.
      The big change is support for uploading via the master server, based heavily
      on the prototype that Grant did. Currently only host-based (IP-based)
      authentication is done as is the case with download. Grant's SSL-based
      authentication code is "integrated" but has not even been compiled in.
      The PUT protocol allows for assorted gewgaws, like specifying a maximum size,
      setting a timeout value, returning size and signature info, etc.
      There is a new, awkwardly-named client utility "frisupload" which, like the
      download client, takes an "image ID" as an argument and requests to upload
      (PUT) that image via the master server. As with download, the image ID can
      be either of the form "<pid>/<emulab-image-name>", to upload/update an actual
      Emulab image or it can start with a "/" in which case it is considered to
      be a pathname on the server.
      On the server side, the master server takes PUT requests, verifies permission
      to upload the image, fires up a separate instance of an upload daemon (with
      the even catchier moniker "frisuploadd"), and returns the unicast addr/port
      info to the client which then begins the upload. The master server also acts
      as a traffic cop to make sure that downloads and uploads (or uploads and
      uploads) don't overlap.
      This has been integrated into the Emulab "create image" process in a
      backward-compatible way (i.e., so old admin MFSes will continue to work).
      Boy, was that fun. One not-so-desirable effect of this integration is that
      images now traverse our network twice, once to upload from node to boss and
      once for boss to write out the image file across NFS to ops. This is not
      really something that should be "fixed" in frisbee, it is only "undesirable"
      because we have a crappy NFS server.
      What has NOT been done includes: support of hierarchical PUT operations
      (we don't need it for either the elabinelab or subboss case), support for
      uploading standard images stored on boss (we really want something better
      than host-based authentication here), and the aforementioned support of
      SSL-based authentication.
      2. Other tidbits that got mixed in with PUT support:
      Added two new site variables:
      which replace the hardwired (in mfrisbeed and frisbeelauncher before that)
      bandwidth limits for image download. mfrisbeed reads these (and the
      images/create/* variables) when it starts up or receives a HUP signal.
      These could be read from the DB on every GET/PUT, but they really don't change
      much and I needed something to test the reread-the-config-on-a-HUP code!
      Fixed avoidance of "problematic multicast addresses" so it would actually
      work as intended.
      Lots of internal "refactoring" to make up for things I did wrong the first
      time and to give the general impression that "Wow, Mike did a LOT!"
  12. 04 Apr, 2011 1 commit
    • Mike Hibler's avatar
      Bug fix: ensure we run any spawned frisbeed as "the appropriate user." · 7f775c7d
      Mike Hibler authored
      Previously, frisbeed's were always running as the uid/gid of the master server
      which generally meant root. This was not a security issue, as we did not rely
      on file permissions to enforce accessibility.
      For the Emulab config, "appropriate user" means the uid of the experiment
      swapper and the gid of the experiment. For the null (default) config, it
      still means the uid/gid of the master server--if you don't want them running
      as root, don't run the master server as root.
      For now, and frisbee clients spawned (e.g., to fetch an image from a parent)
      are still run with the uid/gid of the master server. This can easily be
      changed if necessary.
  13. 14 Jan, 2011 1 commit
  14. 11 Jan, 2011 1 commit
    • Mike Hibler's avatar
      More work toward getting this working on subboss. · 8d80301e
      Mike Hibler authored
      More work on the hierarchical configuration for subboss. When doing host-based
      authentication, allow client to pass an explicit host (IP) to the mserver.
      If the mserver is configured to allow it, that IP is used for authenticating
      the request instead of the caller's IP. Add a default ("null") configuration
      so the mserver can operate out-of-the-box with no config file. The goal of
      these two changes is for an mserver instance with the default config and a
      proxy option to serve the needs of a subboss node (i.e., so no explicit
      configuration will be needed).
  15. 13 Dec, 2010 1 commit
    • Mike Hibler's avatar
      Aerformed. If the mserver receiving the request allows the caller to be a · c5b7cceb
      Mike Hibler authored
      proxy, then it will perform its checks against the hostIP provided rather
      than the IP of the message sender.  For the Emulab subboss case, subboss
      nodes (as determined from the DB) are allowed to explicitly specify hosts
      that are under their control.  The master server on real boss will run with
      proxying enabled.
      Also, in a fit of madness, I added a version number to the master server
      protocol.  This way, if at some distant point in the future (say next week)
      I realize I screwed up the protocol, I can fix it without resorting to creative
      retrofitting of a version number (see imagezip) or the more Orwellian
      eradication and denial of past versions (what I am doing now). Furthermore,
      using an ill-thought-out insight, I made the version number be an ASCII string
      in case I decide to change to an all-text protocol at some equally distant
      point in the future.
  16. 06 Dec, 2010 1 commit
    • Mike Hibler's avatar
      Make unicast transfers work. · 66b0ae44
      Mike Hibler authored
      There are some hacks involved here right now since a unicast frisbee server
      currently can only support a single client.  So for now, we only allow a
      single unicast server for any image (i.e., because the right way to do this
      is to fix the server to support multiple clients, not to start up multiple
      single-client servers).
  17. 03 Dec, 2010 1 commit
    • Mike Hibler's avatar
      Hierarchical downloading is now implemented. · 21b01bbb
      Mike Hibler authored
      The master server can be started with a "parent" mserver that it can call
      if it doesn't have an image.  The master server will return an EAGAIN type
      error to any clients that contact it while it is downloading an image from
      its parent.  The client now has a "-B N" option to tell it to try again
      every N seconds as long as it gets back that error.  This is the "store and
      forward" mode.  The mserver also has a "cut through" style where it will
      return to clients the mcast info it got from its parent so they can download
      directly from the parent until the local mserver has it.
  18. 02 Dec, 2010 1 commit
    • Mike Hibler's avatar
      Inching toward recursive use, ala what is needed for elabinelab or subbosses. · 2ead2a68
      Mike Hibler authored
      Add the ability of the master server to have a "parent" from which it can
      download an image if it doesn't have it or if the image is out of date.
      Had to add some more goo to the GET reply, notably a hash so that we can
      check for up-to-dateness.
      The actual part where we upcall to the parent isn't done yet, that is why
      this is "inching toward" and not "leaping and bounding toward"...
      Also redid the child process management to not use SIGCHLD, no need for that.
  19. 24 Nov, 2010 1 commit
    • Mike Hibler's avatar
      First crack at a frisbee "master server" for handling GET (download) requests. · a2a896ab
      Mike Hibler authored
      There are a couple of new packet types in the frisbee protocol which are
      exchanged via TCP with the master server: GETREQUEST and GETREPLY.  The
      client passes to the master server an opaque imageid and a couple of options
      and gets back the addr/port to use to actually download the image.  The
      implementation of the master server is fragile and is more of a test
      framework, Grant is working on a more robust master server.  I am mostly
      doing a backend that communicates with the Emulab DB to do its authentication
      and making the client changes.
      The client now uses the -S option to specify the IP address of the master
      server and the -F option to specify an imageid.  If no error is returned,
      the image is downloaded using the returned addr/port.  If -Q is used in place
      of -F, then the client makes a "status only" call getting back info about
      whether the named image is accessible to the client and whether a server is
      currently running.
      On the server side, the new master server (mserver.c) has an Emulab
      configuration "backend" that supports host-based authentication.
      The IP address of the caller is mapped to a node_id/pid/gid/eid combo
      that is used to determine access.  On a request, the specified imageid is
      treated either as a pathname (if it starts with '/') or an image identifier
      of the form "<pid>/<imagename>".  If it is a pathname, we check to make
      sure that pathname (after running through "realpath") is contained in one
      of the directories accessible to that node in its current experiment context;
      i.e., /share, /proj/<pid>, /groups/<pid>/<gid>, or /users/<swapper-uid>.
      If it is an image identifier, the DB is queried to ensure that access is
      allowed to that image; i.e., it must be "global" or in the appropriate
      The master server forks a frisbeed for each valid request, if one is not
      already running.  The multicast address selection is still based on the
      emulab_indicies.frisbee_index field, but the address/port/server info is no
      longer stored in the frisbee_blobs table (frisbee_pid, load_address,
      load_busy are not set).
      Note that this is not yet integrated in the os_load path.  Further work is
      required to replace frisbeelauncher.