• Tom Herbert's avatar
    net: Allow no-cache copy from user on transmit · c6e1a0d1
    Tom Herbert authored
    This patch uses __copy_from_user_nocache on transmit to bypass data
    cache for a performance improvement.  skb_add_data_nocache and
    skb_copy_to_page_nocache can be called by sendmsg functions to use
    this feature, initial support is in tcp_sendmsg.  This functionality is
    configurable per device using ethtool.
    Presumably, this feature would only be useful when the driver does
    not touch the data.  The feature is turned on by default if a device
    indicates that it does some form of checksum offload; it is off by
    default for devices that do no checksum offload or indicate no checksum
    is necessary.  For the former case copy-checksum is probably done
    anyway, in the latter case the device is likely loopback in which case
    the no cache copy is probably not beneficial.
    This patch was tested using 200 instances of netperf TCP_RR with
    1400 byte request and one byte reply.  Platform is 16 core AMD x86.
    No-cache copy disabled:
       672703 tps, 97.13% utilization
       50/90/99% latency:244.31 484.205 1028.41
    No-cache copy enabled:
       702113 tps, 96.16% utilization,
       50/90/99% latency 238.56 467.56 956.955
    Using 14000 byte request and response sizes demonstrate the
    effects more dramatically:
    No-cache copy disabled:
       79571 tps, 34.34 %utlization
       50/90/95% latency 1584.46 2319.59 5001.76
    No-cache copy enabled:
       83856 tps, 34.81% utilization
       50/90/95% latency 2508.42 2622.62 2735.88
    Note especially the effect on latency tail (95th percentile).
    This seems to provide a nice performance improvement and is
    consistent in the tests I ran.  Presumably, this would provide
    the greatest benfits in the presence of an application workload
    stressing the cache and a lot of transmit data happening.
    Signed-off-by: default avatarTom Herbert <therbert@google.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>