• Joonsoo Kim's avatar
    vmalloc: use rcu list iterator to reduce vmap_area_lock contention · 474750ab
    Joonsoo Kim authored
    Richard Yao reported a month ago that his system have a trouble with
    vmap_area_lock contention during performance analysis by /proc/meminfo.
    Andrew asked why his analysis checks /proc/meminfo stressfully, but he
    didn't answer it.
    Although I'm not sure that this is right usage or not, there is a
    solution reducing vmap_area_lock contention with no side-effect.  That
    is just to use rcu list iterator in get_vmalloc_info().
    rcu can be used in this function because all RCU protocol is already
    respected by writers, since Nick Piggin commit db64fe02
    rewrite vmap layer") back in linux-2.6.28
    Specifically :
       insertions use list_add_rcu(),
       deletions use list_del_rcu() and kfree_rcu().
    Note the rb tree is not used from rcu reader (it would not be safe),
    only the vmap_area_list has full RCU protection.
    Note that __purge_vmap_area_lazy() already uses this rcu protection.
            list_for_each_entry_rcu(va, &vmap_area_list, list) {
                    if (va->flags & VM_LAZY_FREE) {
                            if (va->va_start < *start)
                                    *start = va->va_start;
                            if (va->va_end > *end)
                                    *end = va->va_end;
                            nr += (va->va_end - va->va_start) >> PAGE_SHIFT;
                            list_add_tail(&va->purge_list, &valist);
                            va->flags |= VM_LAZY_FREEING;
                            va->flags &= ~VM_LAZY_FREE;
    : While rcu list traversal over the vmap_area_list is safe, this may
    : arrive at different results than the spinlocked version. The rcu list
    : traversal version will not be a 'snapshot' of a single, valid instant
    : of the entire vmap_area_list, but rather a potential amalgam of
    : different list states.
    : Yes, you are right, but I don't think that we should be strict here.
    : Meminfo is already not a 'snapshot' at specific time.  While we try to get
    : certain stats, the other stats can change.  And, although we may arrive at
    : different results than the spinlocked version, the difference would not be
    : large and would not make serious side-effect.
    [edumazet@google.com: add more commit description]
    Signed-off-by: default avatarJoonsoo Kim <iamjoonsoo.kim@lge.com>
    Reported-by: default avatarRichard Yao <ryao@gentoo.org>
    Acked-by: default avatarEric Dumazet <edumazet@google.com>
    Cc: Peter Hurley <peter@hurleysoftware.com>
    Cc: Zhang Yanfei <zhangyanfei.yes@gmail.com>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: Andi Kleen <andi@firstfloor.org>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
vmalloc.c 68.3 KB