Skip to content
  • Eric Dumazet's avatar
    inetpeer: RCU conversion · aa1039e7
    Eric Dumazet authored
    
    
    inetpeer currently uses an AVL tree protected by an rwlock.
    
    It's possible to make most lookups use RCU
    
    1) Add a struct rcu_head to struct inet_peer
    
    2) add a lookup_rcu_bh() helper to perform lockless and opportunistic
    lookup. This is a normal function, not a macro like lookup().
    
    3) Add a limit to number of links followed by lookup_rcu_bh(). This is
    needed in case we fall in a loop.
    
    4) add an smp_wmb() in link_to_pool() right before node insert.
    
    5) make unlink_from_pool() use atomic_cmpxchg() to make sure it can take
    last reference to an inet_peer, since lockless readers could increase
    refcount, even while we hold peers.lock.
    
    6) Delay struct inet_peer freeing after rcu grace period so that
    lookup_rcu_bh() cannot crash.
    
    7) inet_getpeer() first attempts lockless lookup.
       Note this lookup can fail even if target is in AVL tree, but a
    concurrent writer can let tree in a non correct form.
       If this attemps fails, lock is taken a regular lookup is performed
    again.
    
    8) convert peers.lock from rwlock to a spinlock
    
    9) Remove SLAB_HWCACHE_ALIGN when peer_cachep is created, because
    rcu_head adds 16 bytes on 64bit arches, doubling effective size (64 ->
    128 bytes)
    In a future patch, this is probably possible to revert this part, if rcu
    field is put in an union to share space with rid, ip_id_count, tcp_ts &
    tcp_ts_stamp. These fields being manipulated only with refcnt > 0.
    
    Signed-off-by: default avatarEric Dumazet <eric.dumazet@gmail.com>
    Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
    aa1039e7