• Vlastimil Babka's avatar
    mm: always steal split buddies in fallback allocations · 3a1086fb
    Vlastimil Babka authored
    When allocation falls back to another migratetype, it will steal a page
    with highest available order, and (depending on this order and desired
    migratetype), it might also steal the rest of free pages from the same
    pageblock.
    
    Given the preference of highest available order, it is likely that it will
    be higher than the desired order, and result in the stolen buddy page
    being split.  The remaining pages after split are currently stolen only
    when the rest of the free pages are stolen.  This can however lead to
    situations where for MOVABLE allocations we split e.g.  order-4 fallback
    UNMOVABLE page, but steal only order-0 page.  Then on the next MOVABLE
    allocation (which may be batched to fill the pcplists) we split another
    order-3 or higher page, etc.  By stealing all pages that we have split, we
    can avoid further stealing.
    
    This patch therefore adjusts the page stealing so that buddy pages created
    by split are always stolen.  This has effect only on MOVABLE allocations,
    as RECLAIMABLE and UNMOVABLE allocations already always do that in
    addition to stealing the rest of free pages from the pageblock.  The
    change also allows to simplify try_to_steal_freepages() and factor out CMA
    handling.
    
    According to Mel, it has been intended since the beginning that buddy
    pages after split would be stolen always, but it doesn't seem like it was
    ever the case until commit 47118af0 ("mm: mmzone: MIGRATE_CMA
    migration type added").  The commit has unintentionally introduced this
    behavior, but was reverted by commit 0cbef29a
    
     ("mm:
    __rmqueue_fallback() should respect pageblock type").  Neither included
    evaluation.
    
    My evaluation with stress-highalloc from mmtests shows about 2.5x
    reduction of page stealing events for MOVABLE allocations, without
    affecting the page stealing events for other allocation migratetypes.
    Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Acked-by: default avatarMel Gorman <mgorman@suse.de>
    Cc: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
    Acked-by: default avatarMinchan Kim <minchan@kernel.org>
    Cc: David Rientjes <rientjes@google.com>
    Cc: Rik van Riel <riel@redhat.com>
    Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com>
    Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
    Cc: Michal Hocko <mhocko@suse.cz>
    Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    3a1086fb
page_alloc.c 181 KB