• Paul E. McKenney's avatar
    rcu: Fix RCU_BOOST race handling current->rcu_read_unlock_special · 7765be2f
    Paul E. McKenney authored
    The RCU_BOOST commits for TREE_PREEMPT_RCU introduced an other-task
    write to a new RCU_READ_UNLOCK_BOOSTED bit in the task_struct structure's
    ->rcu_read_unlock_special field, but, as noted by Steven Rostedt, without
    correctly synchronizing all accesses to ->rcu_read_unlock_special.
    This could result in bits in ->rcu_read_unlock_special being spuriously
    set and cleared due to conflicting accesses, which in turn could result
    in deadlocks between the rcu_node structure's ->lock and the scheduler's
    rq and pi locks.  These deadlocks would result from RCU incorrectly
    believing that the just-ended RCU read-side critical section had been
    preempted and/or boosted.  If that RCU read-side critical section was
    executed with either rq or pi locks held, RCU's ensuing (incorrect)
    calls to the scheduler would cause the scheduler to attempt to once
    again acquire the rq and pi locks, resulting in deadlock.  More complex
    deadlock cycles are also possible, involving multiple rq and pi locks
    as well as locks from multiple rcu_node structures.
    
    This commit fixes synchronization by creating ->rcu_boosted field in
    task_struct that is accessed and modified only when holding the ->lock
    in the rcu_node structure on which the task is queued (on that rcu_node
    structure's ->blkd_tasks list).  This results in tasks accessing only
    their own current->rcu_read_unlock_special fields, making unsynchronized
    access once again legal, and keeping the rcu_read_unlock() fastpath free
    of atomic instructions and memory barriers.
    
    The reason that the rcu_read_unlock() fastpath does not need to access
    the new current->rcu_boosted field is that this new field cannot
    be non-zero unless the RCU_READ_UNLOCK_BLOCKED bit is set in the
    current->rcu_read_unlock_special field.  Therefore, rcu_read_unlock()
    need only test current->rcu_read_unlock_special: if that is zero, then
    current->rcu_boosted must also be zero.
    
    This bug does not affect TINY_PREEMPT_RCU because this implementation
    of RCU accesses current->rcu_read_unlock_special with irqs disabled,
    thus preventing races on the !SMP systems that TINY_PREEMPT_RCU runs on.
    Maybe-reported-by: default avatarDave Jones <davej@redhat.com>
    Maybe-reported-by: default avatarSergey Senozhatsky <sergey.senozhatsky@gmail.com>
    Reported-by: default avatarSteven Rostedt <rostedt@goodmis.org>
    Signed-off-by: default avatarPaul E. McKenney <paul.mckenney@linaro.org>
    Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
    Reviewed-by: default avatarSteven Rostedt <rostedt@goodmis.org>
    7765be2f
sched.h 78.8 KB