• Michael R. Hines's avatar
    rdma: IPv6 over Ethernet (RoCE) is broken in linux - workaround · 7fc5b13f
    Michael R. Hines authored
    We've gotten reports from multiple testers (including Frank Yangjie
    and myself) that RDMA IPv6 support over RocE (Ethernet) is broken
    in linux.
    A patch to Linux is still in review:
    If the user is listening on '[::]', then we will not have a opened a device
    yet and have no way of verifying if the device is RoCE or not.
    In this case, the source VM will throw an error for ALL types of
    connections (both IPv4 and IPv6) if the destination machine does not have
    a regular infiniband network available for use.
    The only way to gaurantee that an error is thrown for broken kernels is
    for the management software to choose a *specific* interface at bind time
    and validate what time of hardware it is.
    Unfortunately, this puts the user in a fix:
     If the source VM connects with an IPv4 address without knowing that the
     destination has bound to '[::]' the migration will unconditionally fail
     unless the management software is not explicitly listening on the the IPv4
     address while using a RoCE-based device.
     If the source VM connects with an IPv6 address, then we're OK because we can
     throw an error on the source (and similarly on the destination).
     But in mixed environments, this will be broken for a while until it is fixed
     inside linux.
    We do provide a *tiny* bit of help in mixed environments, though in this patch:
    We can list all of the devices in the system and check to see if all the
    devices are RoCE or Infiniband.
    If we detect that we have a *pure* RoCE environment, then we can safely
    thrown an error even if the management sofware has specified '[::]' as the
    bind address.
    However, if there is are multiple hetergeneous devices, then we cannot make
    this assumption and the user just has to be sure they know what they are doing.
    Signed-off-by: default avatarMichael R. Hines <mrhines@us.ibm.com>
    Message-id: 1376078746-24948-6-git-send-email-mrhines@linux.vnet.ibm.com
    Signed-off-by: default avatarAnthony Liguori <aliguori@us.ibm.com>
migration-rdma.c 105 KB