• Jens Axboe's avatar
    direct-io: only inc/dec inode->i_dio_count for file systems · fe0f07d0
    Jens Axboe authored
    do_blockdev_direct_IO() increments and decrements the inode
    ->i_dio_count for each IO operation. It does this to protect against
    truncate of a file. Block devices don't need this sort of protection.
    For a capable multiqueue setup, this atomic int is the only shared
    state between applications accessing the device for O_DIRECT, and it
    presents a scaling wall for that. In my testing, as much as 30% of
    system time is spent incrementing and decrementing this value. A mixed
    read/write workload improved from ~2.5M IOPS to ~9.6M IOPS, with
    better latencies too. Before:
    clat percentiles (usec):
     |  1.00th=[   33],  5.00th=[   34], 10.00th=[   34], 20.00th=[   34],
     | 30.00th=[   34], 40.00th=[   34], 50.00th=[   35], 60.00th=[   35],
     | 70.00th=[   35], 80.00th=[   35], 90.00th=[   37], 95.00th=[   80],
     | 99.00th=[   98], 99.50th=[  151], 99.90th=[  155], 99.95th=[  155],
     | 99.99th=[  165]
    clat percentiles (usec):
     |  1.00th=[   95],  5.00th=[  108], 10.00th=[  129], 20.00th=[  149],
     | 30.00th=[  155], 40.00th=[  161], 50.00th=[  167], 60.00th=[  171],
     | 70.00th=[  177], 80.00th=[  185], 90.00th=[  201], 95.00th=[  270],
     | 99.00th=[  390], 99.50th=[  398], 99.90th=[  418], 99.95th=[  422],
     | 99.99th=[  438]
    In other setups, Robert Elliott reported seeing good performance
    The more applications accessing the device, the worse it gets.
    Add a new direct-io flags, DIO_SKIP_DIO_COUNT, which tells
    do_blockdev_direct_IO() that it need not worry about incrementing
    or decrementing the inode i_dio_count for this caller.
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Christoph Hellwig <hch@lst.de>
    Cc: Theodore Ts'o <tytso@mit.edu>
    Cc: Elliott, Robert (Server Storage) <elliott@hp.com>
    Cc: Al Viro <viro@zeniv.linux.org.uk>
    Signed-off-by: default avatarJens Axboe <axboe@fb.com>
    Signed-off-by: default avatarAl Viro <viro@zeniv.linux.org.uk>
direct-io.c 37.8 KB