1. 26 Oct, 2010 40 commits
    • Dave Young's avatar
      mm: add vzalloc() and vzalloc_node() helpers · e1ca7788
      Dave Young authored
      
      
      Add vzalloc() and vzalloc_node() to encapsulate the
      vmalloc-then-memset-zero operation.
      
      Use __GFP_ZERO to zero fill the allocated memory.
      Signed-off-by: default avatarDave Young <hidave.darkstar@gmail.com>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      Acked-by: default avatarGreg Ungerer <gerg@snapgear.com>
      Cc: David Howells <dhowells@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e1ca7788
    • Andrew Morton's avatar
      mm/memory_hotplug.c: make scan_lru_pages() static · 7bbc0905
      Andrew Morton authored
      
      Reported-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      7bbc0905
    • Andrew Morton's avatar
      fs/fs-writeback.c: restore lost comment · 74ce002d
      Andrew Morton authored
      
      
      I had to go back to a 2.6.20 tree to work out why we're adding a
      number-of-inodes into a number-of-pages count.  Restore the lost comment.
      
      Cc: Jens Axboe <axboe@kernel.dk>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      74ce002d
    • Namhyung Kim's avatar
      mm: fix sparse warnings on GFP_ZONE_TABLE/BAD · 16b56cf4
      Namhyung Kim authored
      
      
      Introduce ___GFP_* masks in order for gfp_t to not be mixed with plain
      integers which causes a lot of warnings like the following:
      
       warning: restricted gfp_t degrades to integer
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Cc: Al Viro <viro@zeniv.linux.org.uk>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      16b56cf4
    • Namhyung Kim's avatar
      vmstat: include compaction.h when CONFIG_COMPACTION · 36deb0be
      Namhyung Kim authored
      
      
      This removes following warning from sparse:
      
       mm/vmstat.c:466:5: warning: symbol 'fragmentation_index' was not declared. Should it be static?
      
      [akpm@linux-foundation.org: move the include to top-of-file]
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      36deb0be
    • Namhyung Kim's avatar
      mm: declare some external symbols · 92c09c04
      Namhyung Kim authored
      
      
      Declare 'bdi_pending_list' and 'tag_pages_for_writeback()' to remove
      following sparse warnings:
      
       mm/backing-dev.c:46:1: warning: symbol 'bdi_pending_list' was not declared. Should it be static?
       mm/page-writeback.c:825:6: warning: symbol 'tag_pages_for_writeback' was not declared. Should it be static?
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      92c09c04
    • Namhyung Kim's avatar
      vmalloc: annotate lock context change on s_start/stop() · e199b5d1
      Namhyung Kim authored
      
      
      s_start() and s_stop() grab/release vmlist_lock but were missing proper
      annotations.  Add them.
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e199b5d1
    • Namhyung Kim's avatar
      vmalloc: rename temporary variable in __insert_vmap_area() · 170168d0
      Namhyung Kim authored
      
      
      Rename redundant 'tmp' to fix following sparse warnings:
      
       mm/vmalloc.c:296:34: warning: symbol 'tmp' shadows an earlier one
       mm/vmalloc.c:293:24: originally declared here
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      170168d0
    • Namhyung Kim's avatar
      rmap: make anon_vma_chain_free() static · e574b5fd
      Namhyung Kim authored
      
      
      Make anon_vma_chain_free() static.  It is called only in rmap.c and the
      corresponding alloc function is already static.
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Cc: Andi Kleen <andi@firstfloor.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e574b5fd
    • Namhyung Kim's avatar
      rmap: wrap page_check_address() using __cond_lock() · e9a81a82
      Namhyung Kim authored
      
      
      The page_check_address() conditionally grabs *@ptlp in case of returning
      non-NULL.  Rename and wrap it using __cond_lock() removes following
      warnings from sparse:
      
       mm/rmap.c:472:9: warning: context imbalance in 'page_mapped_in_vma' - unexpected unlock
       mm/rmap.c:524:9: warning: context imbalance in 'page_referenced_one' - unexpected unlock
       mm/rmap.c:706:9: warning: context imbalance in 'page_mkclean_one' - unexpected unlock
       mm/rmap.c:1066:9: warning: context imbalance in 'try_to_unmap_one' - unexpected unlock
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e9a81a82
    • Namhyung Kim's avatar
      rmap: annotate lock context change on page_[un]lock_anon_vma() · ea4525b6
      Namhyung Kim authored
      
      
      The page_lock_anon_vma() conditionally grabs RCU and anon_vma lock but
      page_unlock_anon_vma() releases them unconditionally.  This leads sparse
      to complain about context imbalance.  Annotate them.
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      ea4525b6
    • Namhyung Kim's avatar
      mm: wrap follow_pte() using __cond_lock() · 1b36ba81
      Namhyung Kim authored
      
      
      The follow_pte() conditionally grabs *@ptlp in case of returning 0.
      Rename and wrap it using __cond_lock() removes following warnings:
      
       mm/memory.c:2337:9: warning: context imbalance in 'do_wp_page' - unexpected unlock
       mm/memory.c:3142:19: warning: context imbalance in 'handle_mm_fault' - different lock contexts for basic block
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      1b36ba81
    • Namhyung Kim's avatar
      mm: add lock release annotation on do_wp_page() · e6219ec8
      Namhyung Kim authored
      
      
      The do_wp_page() releases @ptl but was missing proper annotation.  Add it.
       This removes following warnings from sparse:
      
       mm/memory.c:2337:9: warning: context imbalance in 'do_wp_page' - unexpected unlock
       mm/memory.c:3142:19: warning: context imbalance in 'handle_mm_fault' - different lock contexts for basic block
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e6219ec8
    • Namhyung Kim's avatar
      mm: wrap get_locked_pte() using __cond_lock() · 25ca1d6c
      Namhyung Kim authored
      
      
      The get_locked_pte() conditionally grabs 'ptl' in case of returning
      non-NULL.  This leads sparse to complain about context imbalance.  Rename
      and wrap it using __cond_lock() to make sparse happy.
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      25ca1d6c
    • Namhyung Kim's avatar
      mm: add casts to/from gfp_t in gfp_to_alloc_flags() · e6223a3b
      Namhyung Kim authored
      
      
      This removes following warning from sparse:
      
       mm/page_alloc.c:1934:9: warning: restricted gfp_t degrades to integer
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e6223a3b
    • Namhyung Kim's avatar
      mm: remove temporary variable on generic_file_direct_write() · 0116651c
      Namhyung Kim authored
      
      
      'end' shadows earlier one and is not necessary at all.  Remove it and use
      'pos' instead.  This removes following sparse warnings:
      
       mm/filemap.c:2180:24: warning: symbol 'end' shadows an earlier one
       mm/filemap.c:2132:25: originally declared here
      Signed-off-by: default avatarNamhyung Kim <namhyung@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      0116651c
    • Michel Lespinasse's avatar
      x86: access_error API cleanup · 68da336a
      Michel Lespinasse authored
      
      
      access_error() already takes error_code as an argument, so there is
      no need for an additional write flag.
      Signed-off-by: default avatarMichel Lespinasse <walken@google.com>
      Acked-by: default avatarRik van Riel <riel@redhat.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Acked-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Cc: Ying Han <yinghan@google.com>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Acked-by: default avatar"H. Peter Anvin" <hpa@zytor.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      68da336a
    • Michel Lespinasse's avatar
      mm: retry page fault when blocking on disk transfer · d065bd81
      Michel Lespinasse authored
      
      
      This change reduces mmap_sem hold times that are caused by waiting for
      disk transfers when accessing file mapped VMAs.
      
      It introduces the VM_FAULT_ALLOW_RETRY flag, which indicates that the call
      site wants mmap_sem to be released if blocking on a pending disk transfer.
      In that case, filemap_fault() returns the VM_FAULT_RETRY status bit and
      do_page_fault() will then re-acquire mmap_sem and retry the page fault.
      
      It is expected that the retry will hit the same page which will now be
      cached, and thus it will complete with a low mmap_sem hold time.
      
      Tests:
      
      - microbenchmark: thread A mmaps a large file and does random read accesses
        to the mmaped area - achieves about 55 iterations/s. Thread B does
        mmap/munmap in a loop at a separate location - achieves 55 iterations/s
        before, 15000 iterations/s after.
      
      - We are seeing related effects in some applications in house, which show
        significant performance regressions when running without this change.
      
      [akpm@linux-foundation.org: fix warning & crash]
      Signed-off-by: default avatarMichel Lespinasse <walken@google.com>
      Acked-by: default avatarRik van Riel <riel@redhat.com>
      Acked-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Reviewed-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Cc: Ying Han <yinghan@google.com>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Acked-by: default avatar"H. Peter Anvin" <hpa@zytor.com>
      Cc: <linux-arch@vger.kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d065bd81
    • Michel Lespinasse's avatar
      mm: filemap_fault: unique path for locking page · b522c94d
      Michel Lespinasse authored
      
      
      Introduce a single location where filemap_fault() locks the desired page.
      There used to be two such places, depending if the initial find_get_page()
      was successful or not.
      Signed-off-by: default avatarMichel Lespinasse <walken@google.com>
      Acked-by: default avatarRik van Riel <riel@redhat.com>
      Acked-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Reviewed-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Cc: Ying Han <yinghan@google.com>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: <linux-arch@vger.kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      b522c94d
    • Richard Kennedy's avatar
      mm: remove alignment padding from anon_vma on (some) 64 bit builds · 182fea8f
      Richard Kennedy authored
      
      
      Reorder structure anon_vma to remove alignment padding on 64 builds when
      (CONFIG_KSM || CONFIG_MIGRATION).
      This will shrink the size of the anon_vma structure from 40 to 32 bytes
      & allow more objects per slab in its kmem_cache.
      
      Under slub the objects in the anon_vma kmem_cache will then be 40 bytes
      with 102 objects per slab.  (On v2.6.36 without this patch,the size is 48
      bytes and 85 objects/slab.)
      Signed-off-by: default avatarRichard Kennedy <richard@rsk.demon.co.uk>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      182fea8f
    • Dima Zavin's avatar
      mm: add a might_sleep_if() to dma_pool_alloc() · ea05c844
      Dima Zavin authored
      
      
      Buggy drivers (e.g.  fsl_udc) could call dma_pool_alloc from atomic
      context with GFP_KERNEL.  In most instances, the first pool_alloc_page
      call would succeed and the sleeping functions would never be called.  This
      allowed the buggy drivers to slip through the cracks.
      
      Add a might_sleep_if() checking for __GFP_WAIT in flags.
      Signed-off-by: default avatarDima Zavin <dima@android.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      ea05c844
    • Peter Zijlstra's avatar
      mm: highmem documentation · d65bfacb
      Peter Zijlstra authored
      
      
      Document outlining some of the highmem issues, started by me, edited by
      David.
      Signed-off-by: default avatarDavid Howells <dhowells@redhat.com>
      Signed-off-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Acked-by: default avatarChris Metcalf <cmetcalf@tilera.com>
      Cc: Hugh Dickins <hughd@google.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Steven Rostedt <rostedt@goodmis.org>
      Cc: Russell King <rmk@arm.linux.org.uk>
      Cc: Ralf Baechle <ralf@linux-mips.org>
      Cc: David Miller <davem@davemloft.net>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d65bfacb
    • Peter Zijlstra's avatar
      perf, x86: Fix up kmap_atomic() type · 7a837d1b
      Peter Zijlstra authored
      
      
      Now that the KM_type stuff is history, clean up the compiler warning.
      Signed-off-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Acked-by: default avatarChris Metcalf <cmetcalf@tilera.com>
      Cc: David Howells <dhowells@redhat.com>
      Cc: Hugh Dickins <hughd@google.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Steven Rostedt <rostedt@goodmis.org>
      Cc: Russell King <rmk@arm.linux.org.uk>
      Cc: Ralf Baechle <ralf@linux-mips.org>
      Cc: David Miller <davem@davemloft.net>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      7a837d1b
    • Peter Zijlstra's avatar
      mm: remove pte_*map_nested() · ece0e2b6
      Peter Zijlstra authored
      
      
      Since we no longer need to provide KM_type, the whole pte_*map_nested()
      API is now redundant, remove it.
      Signed-off-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Acked-by: default avatarChris Metcalf <cmetcalf@tilera.com>
      Cc: David Howells <dhowells@redhat.com>
      Cc: Hugh Dickins <hughd@google.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Steven Rostedt <rostedt@goodmis.org>
      Cc: Russell King <rmk@arm.linux.org.uk>
      Cc: Ralf Baechle <ralf@linux-mips.org>
      Cc: David Miller <davem@davemloft.net>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      ece0e2b6
    • Peter Zijlstra's avatar
      mm: stack based kmap_atomic() · 3e4d3af5
      Peter Zijlstra authored
      
      
      Keep the current interface but ignore the KM_type and use a stack based
      approach.
      
      The advantage is that we get rid of crappy code like:
      
      	#define __KM_PTE			\
      		(in_nmi() ? KM_NMI_PTE : 	\
      		 in_irq() ? KM_IRQ_PTE :	\
      		 KM_PTE0)
      
      and in general can stop worrying about what context we're in and what kmap
      slots might be appropriate for that.
      
      The downside is that FRV kmap_atomic() gets more expensive.
      
      For now we use a CPP trick suggested by Andrew:
      
        #define kmap_atomic(page, args...) __kmap_atomic(page)
      
      to avoid having to touch all kmap_atomic() users in a single patch.
      
      [ not compiled on:
        - mn10300: the arch doesn't actually build with highmem to begin with ]
      
      [akpm@linux-foundation.org: coding-style fixes]
      [akpm@linux-foundation.org: fix up drivers/gpu/drm/i915/intel_overlay.c]
      Acked-by: default avatarRik van Riel <riel@redhat.com>
      Signed-off-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Acked-by: default avatarChris Metcalf <cmetcalf@tilera.com>
      Cc: David Howells <dhowells@redhat.com>
      Cc: Hugh Dickins <hughd@google.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Steven Rostedt <rostedt@goodmis.org>
      Cc: Russell King <rmk@arm.linux.org.uk>
      Cc: Ralf Baechle <ralf@linux-mips.org>
      Cc: David Miller <davem@davemloft.net>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Dave Airlie <airlied@linux.ie>
      Cc: Li Zefan <lizf@cn.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      3e4d3af5
    • Peter Zijlstra's avatar
      mm: strictly nested kmap_atomic() · 61ecdb80
      Peter Zijlstra authored
      
      
      Ensure kmap_atomic() usage is strictly nested
      Signed-off-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Acked-by: default avatarChris Metcalf <cmetcalf@tilera.com>
      Cc: David Howells <dhowells@redhat.com>
      Cc: Hugh Dickins <hughd@google.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Cc: "H. Peter Anvin" <hpa@zytor.com>
      Cc: Steven Rostedt <rostedt@goodmis.org>
      Cc: Russell King <rmk@arm.linux.org.uk>
      Cc: Ralf Baechle <ralf@linux-mips.org>
      Cc: David Miller <davem@davemloft.net>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      61ecdb80
    • KOSAKI Motohiro's avatar
      vmscan,tmpfs: treat used once pages on tmpfs as used once · 2e30244a
      KOSAKI Motohiro authored
      
      
      When a page has PG_referenced, shrink_page_list() discards it only if it
      is not dirty.  This rule works fine if the backing filesystem is a regular
      one.  PG_dirty is a good signal that the page was used recently because
      the flusher threads clean pages periodically.  In addition, page writeback
      is costlier than simple page discard.
      
      However, when a page is on tmpfs this heuristic doesn't work because
      flusher threads don't write back tmpfs pages.  Consequently tmpfs pages
      always rotate around the lru twice at least and adds unnecessary lru
      churn.  Simple tmpfs streaming io shouldn't cause large anonymous page
      swap-out.
      
      Remove this unncessary reclaim bonus of tmpfs pages.
      Signed-off-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Hugh Dickins <hughd@google.com>
      Reviewed-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Cc: Minchan Kim <minchan.kim@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      2e30244a
    • Wu Fengguang's avatar
      writeback: remove the internal 5% low bound on dirty_ratio · 4cbec4c8
      Wu Fengguang authored
      
      
      The dirty_ratio was silently limited in global_dirty_limits() to >= 5%.
      This is not a user expected behavior.  And it's inconsistent with
      calc_period_shift(), which uses the plain vm_dirty_ratio value.
      
      Let's remove the internal bound.
      
      At the same time, fix balance_dirty_pages() to work with the
      dirty_thresh=0 case.  This allows applications to proceed when
      dirty+writeback pages are all cleaned.
      
      And ">" fits with the name "exceeded" better than ">=" does.  Neil thinks
      it is an aesthetic improvement as well as a functional one :)
      Signed-off-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Cc: Jan Kara <jack@suse.cz>
      Proposed-by: default avatarCon Kolivas <kernel@kolivas.org>
      Acked-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Reviewed-by: default avatarNeil Brown <neilb@suse.de>
      Reviewed-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Michael Rubin <mrubin@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      4cbec4c8
    • Mel Gorman's avatar
      writeback: do not sleep on the congestion queue if there are no congested BDIs... · 0e093d99
      Mel Gorman authored
      
      writeback: do not sleep on the congestion queue if there are no congested BDIs or if significant congestion is not being encountered in the current zone
      
      If congestion_wait() is called with no BDI congested, the caller will
      sleep for the full timeout and this may be an unnecessary sleep.  This
      patch adds a wait_iff_congested() that checks congestion and only sleeps
      if a BDI is congested else, it calls cond_resched() to ensure the caller
      is not hogging the CPU longer than its quota but otherwise will not sleep.
      
      This is aimed at reducing some of the major desktop stalls reported during
      IO.  For example, while kswapd is operating, it calls congestion_wait()
      but it could just have been reclaiming clean page cache pages with no
      congestion.  Without this patch, it would sleep for a full timeout but
      after this patch, it'll just call schedule() if it has been on the CPU too
      long.  Similar logic applies to direct reclaimers that are not making
      enough progress.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Minchan Kim <minchan.kim@gmail.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Jens Axboe <axboe@kernel.dk>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      0e093d99
    • KOSAKI Motohiro's avatar
      vmscan: isolate_lru_pages(): stop neighbour search if neighbour cannot be isolated · 08fc468f
      KOSAKI Motohiro authored
      
      
      isolate_lru_pages() does not just isolate LRU tail pages, but also
      isolates neighbour pages of the eviction page.  The neighbour search does
      not stop even if neighbours cannot be isolated which is excessive as the
      lumpy reclaim will no longer result in a successful higher order
      allocation.  This patch stops the PFN neighbour pages if an isolation
      fails and moves on to the next block.
      Signed-off-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Reviewed-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Reviewed-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Reviewed-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Rik van Riel <riel@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      08fc468f
    • KOSAKI Motohiro's avatar
      vmscan: remove dead code in shrink_inactive_list() · 47185052
      KOSAKI Motohiro authored
      
      
      After synchrounous lumpy reclaim, the page_list is guaranteed to not have
      active pages as page activation in shrink_page_list() disables lumpy
      reclaim.  Remove the dead code.
      Signed-off-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Reviewed-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      47185052
    • KOSAKI Motohiro's avatar
      vmscan: narrow the scenarios in whcih lumpy reclaim uses synchrounous reclaim · 7d3579e8
      KOSAKI Motohiro authored
      
      
      shrink_page_list() can decide to give up reclaiming a page under a
      number of conditions such as
      
        1. trylock_page() failure
        2. page is unevictable
        3. zone reclaim and page is mapped
        4. PageWriteback() is true
        5. page is swapbacked and swap is full
        6. add_to_swap() failure
        7. page is dirty and gfpmask don't have GFP_IO, GFP_FS
        8. page is pinned
        9. IO queue is congested
       10. pageout() start IO, but not finished
      
      With lumpy reclaim, failures result in entering synchronous lumpy reclaim
      but this can be unnecessary.  In cases (2), (3), (5), (6), (7) and (8),
      there is no point retrying.  This patch causes lumpy reclaim to abort when
      it is known it will fail.
      
      Case (9) is more interesting. current behavior is,
        1. start shrink_page_list(async)
        2. found queue_congested()
        3. skip pageout write
        4. still start shrink_page_list(sync)
        5. wait on a lot of pages
        6. again, found queue_congested()
        7. give up pageout write again
      
      So, it's useless time wasting.  However, just skipping page reclaim is
      also notgood as x86 allocating a huge page needs 512 pages for example.
      It can have more dirty pages than queue congestion threshold (~=128).
      
      After this patch, pageout() behaves as follows;
      
       - If order > PAGE_ALLOC_COSTLY_ORDER
      	Ignore queue congestion always.
       - If order <= PAGE_ALLOC_COSTLY_ORDER
      	skip write page and disable lumpy reclaim.
      Signed-off-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Reviewed-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Minchan Kim <minchan.kim@gmail.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Rik van Riel <riel@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      7d3579e8
    • KOSAKI Motohiro's avatar
      vmscan: synchronous lumpy reclaim should not call congestion_wait() · bc57e00f
      KOSAKI Motohiro authored
      
      
      congestion_wait() means "wait until queue congestion is cleared".
      However, synchronous lumpy reclaim does not need this congestion_wait() as
      shrink_page_list(PAGEOUT_IO_SYNC) uses wait_on_page_writeback() and it
      provides the necessary waiting.
      Signed-off-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Reviewed-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Reviewed-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Reviewed-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Reviewed-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Rik van Riel <riel@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      bc57e00f
    • Mel Gorman's avatar
      writeback: account for time spent congestion_waited · 52bb9198
      Mel Gorman authored
      
      
      There is strong evidence to indicate a lot of time is being spent in
      congestion_wait(), some of it unnecessarily.  This patch adds a tracepoint
      for congestion_wait to record when congestion_wait() was called, how long
      the timeout was for and how long it actually slept.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Reviewed-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Reviewed-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Rik van Riel <riel@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      52bb9198
    • Mel Gorman's avatar
      tracing, vmscan: add trace events for LRU list shrinking · e11da5b4
      Mel Gorman authored
      There have been numerous reports of stalls that pointed at the problem
      being somewhere in the VM.  There are multiple roots to the problems which
      means dealing with any of the root problems in isolation is tricky to
      justify on their own and they would still need integration testing.  This
      patch series puts together two different patch sets which in combination
      should tackle some of the root causes of latency problems being reported.
      
      Patch 1 adds a tracepoint for shrink_inactive_list.  For this series, the
      most important results is being able to calculate the scanning/reclaim
      ratio as a measure of the amount of work being done by page reclaim.
      
      Patch 2 accounts for time spent in congestion_wait.
      
      Patches 3-6 were originally developed by Kosaki Motohiro but reworked for
      this series.  It has been noted that lumpy reclaim is far too aggressive
      and trashes the system somewhat.  As SLUB uses high-order allocations, a
      large cost incurred by lumpy reclaim will be noticeable.  It was also
      reported during transparent hugepage support testing that lumpy reclaim
      was trashing the system and these patches should mitigate that problem
      without disabling lumpy reclaim.
      
      Patch 7 adds wait_iff_congested() and replaces some callers of
      congestion_wait().  wait_iff_congested() only sleeps if there is a BDI
      that is currently congested.  Patch 8 notes that any BDI being congested
      is not necessarily a problem because there could be multiple BDIs of
      varying speeds and numberous zones.  It attempts to track when a zone
      being reclaimed contains many pages backed by a congested BDI and if so,
      reclaimers wait on the congestion queue.
      
      I ran a number of tests with monitoring on X86, X86-64 and PPC64. Each
      machine had 3G of RAM and the CPUs were
      
      X86:    Intel P4 2-core
      X86-64: AMD Phenom 4-core
      PPC64:  PPC970MP
      
      Each used a single disk and the onboard IO controller.  Dirty ratio was
      left at 20.  I'm just going to report for X86-64 and PPC64 in a vague
      attempt to keep this report short.  Four kernels were tested each based on
      v2.6.36-rc4
      
      traceonly-v2r2:     Patches 1 and 2 to instrument vmscan reclaims and congestion_wait
      lowlumpy-v2r3:      Patches 1-6 to test if lumpy reclaim is better
      waitcongest-v2r3:   Patches 1-7 to only wait on congestion
      waitwriteback-v2r4: Patches 1-8 to detect when a zone is congested
      
      nocongest-v1r5: Patches 1-3 for testing wait_iff_congestion
      nodirect-v1r5:  Patches 1-10 to disable filesystem writeback for better IO
      
      The tests run were as follows
      
      kernbench
      	compile-based benchmark. Smoke test performance
      
      sysbench
      	OLTP read-only benchmark. Will be re-run in the future as read-write
      
      micro-mapped-file-stream
      	This is a micro-benchmark from Johannes Weiner that accesses a
      	large sparse-file through mmap(). It was configured to run in only
      	single-CPU mode but can be indicative of how well page reclaim
      	identifies suitable pages.
      
      stress-highalloc
      	Tries to allocate huge pages under heavy load.
      
      kernbench, iozone and sysbench did not report any performance regression
      on any machine.  sysbench did pressure the system lightly and there was
      reclaim activity but there were no difference of major interest between
      the kernels.
      
      X86-64 micro-mapped-file-stream
      
                                            traceonly-v2r2           lowlumpy-v2r3        waitcongest-v2r3     waitwriteback-v2r4
      pgalloc_dma                       1639.00 (   0.00%)       667.00 (-145.73%)      1167.00 ( -40.45%)       578.00 (-183.56%)
      pgalloc_dma32                  2842410.00 (   0.00%)   2842626.00 (   0.01%)   2843043.00 (   0.02%)   2843014.00 (   0.02%)
      pgalloc_normal                       0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)
      pgsteal_dma                        729.00 (   0.00%)        85.00 (-757.65%)       609.00 ( -19.70%)       125.00 (-483.20%)
      pgsteal_dma32                  2338721.00 (   0.00%)   2447354.00 (   4.44%)   2429536.00 (   3.74%)   2436772.00 (   4.02%)
      pgsteal_normal                       0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)
      pgscan_kswapd_dma                 1469.00 (   0.00%)       532.00 (-176.13%)      1078.00 ( -36.27%)       220.00 (-567.73%)
      pgscan_kswapd_dma32            4597713.00 (   0.00%)   4503597.00 (  -2.09%)   4295673.00 (  -7.03%)   3891686.00 ( -18.14%)
      pgscan_kswapd_normal                 0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)
      pgscan_direct_dma                   71.00 (   0.00%)       134.00 (  47.01%)       243.00 (  70.78%)       352.00 (  79.83%)
      pgscan_direct_dma32             305820.00 (   0.00%)    280204.00 (  -9.14%)    600518.00 (  49.07%)    957485.00 (  68.06%)
      pgscan_direct_normal                 0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)
      pageoutrun                       16296.00 (   0.00%)     21254.00 (  23.33%)     18447.00 (  11.66%)     20067.00 (  18.79%)
      allocstall                         443.00 (   0.00%)       273.00 ( -62.27%)       513.00 (  13.65%)      1568.00 (  71.75%)
      
      These are based on the raw figures taken from /proc/vmstat.  It's a rough
      measure of reclaim activity.  Note that allocstall counts are higher
      because we are entering direct reclaim more often as a result of not
      sleeping in congestion.  In itself, it's not necessarily a bad thing.
      It's easier to get a view of what happened from the vmscan tracepoint
      report.
      
      FTrace Reclaim Statistics: vmscan
      
                                      traceonly-v2r2   lowlumpy-v2r3 waitcongest-v2r3 waitwriteback-v2r4
      Direct reclaims                                443        273        513       1568
      Direct reclaim pages scanned                305968     280402     600825     957933
      Direct reclaim pages reclaimed               43503      19005      30327     117191
      Direct reclaim write file async I/O              0          0          0          0
      Direct reclaim write anon async I/O              0          3          4         12
      Direct reclaim write file sync I/O               0          0          0          0
      Direct reclaim write anon sync I/O               0          0          0          0
      Wake kswapd requests                        187649     132338     191695     267701
      Kswapd wakeups                                   3          1          4          1
      Kswapd pages scanned                       4599269    4454162    4296815    3891906
      Kswapd pages reclaimed                     2295947    2428434    2399818    2319706
      Kswapd reclaim write file async I/O              1          0          1          1
      Kswapd reclaim write anon async I/O             59        187         41        222
      Kswapd reclaim write file sync I/O               0          0          0          0
      Kswapd reclaim write anon sync I/O               0          0          0          0
      Time stalled direct reclaim (seconds)         4.34       2.52       6.63       2.96
      Time kswapd awake (seconds)                  11.15      10.25      11.01      10.19
      
      Total pages scanned                        4905237   4734564   4897640   4849839
      Total pages reclaimed                      2339450   2447439   2430145   2436897
      %age total pages scanned/reclaimed          47.69%    51.69%    49.62%    50.25%
      %age total pages scanned/written             0.00%     0.00%     0.00%     0.00%
      %age  file pages scanned/written             0.00%     0.00%     0.00%     0.00%
      Percentage Time Spent Direct Reclaim        29.23%    19.02%    38.48%    20.25%
      Percentage Time kswapd Awake                78.58%    78.85%    76.83%    79.86%
      
      What is interesting here for nocongest in particular is that while direct
      reclaim scans more pages, the overall number of pages scanned remains the
      same and the ratio of pages scanned to pages reclaimed is more or less the
      same.  In other words, while we are sleeping less, reclaim is not doing
      more work and as direct reclaim and kswapd is awake for less time, it
      would appear to be doing less work.
      
      FTrace Reclaim Statistics: congestion_wait
      Direct number congest     waited                87        196         64          0
      Direct time   congest     waited            4604ms     4732ms     5420ms        0ms
      Direct full   congest     waited                72        145         53          0
      Direct number conditional waited                 0          0        324       1315
      Direct time   conditional waited               0ms        0ms        0ms        0ms
      Direct full   conditional waited                 0          0          0          0
      KSwapd number congest     waited                20         10         15          7
      KSwapd time   congest     waited            1264ms      536ms      884ms      284ms
      KSwapd full   congest     waited                10          4          6          2
      KSwapd number conditional waited                 0          0          0          0
      KSwapd time   conditional waited               0ms        0ms        0ms        0ms
      KSwapd full   conditional waited                 0          0          0          0
      
      The vanilla kernel spent 8 seconds asleep in direct reclaim and no time at
      all asleep with the patches.
      
      MMTests Statistics: duration
      User/Sys Time Running Test (seconds)         10.51     10.73      10.6     11.66
      Total Elapsed Time (seconds)                 14.19     13.00     14.33     12.76
      
      Overall, the tests completed faster. It is interesting to note that backing off further
      when a zone is congested and not just a BDI was more efficient overall.
      
      PPC64 micro-mapped-file-stream
      pgalloc_dma                    3024660.00 (   0.00%)   3027185.00 (   0.08%)   3025845.00 (   0.04%)   3026281.00 (   0.05%)
      pgalloc_normal                       0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)
      pgsteal_dma                    2508073.00 (   0.00%)   2565351.00 (   2.23%)   2463577.00 (  -1.81%)   2532263.00 (   0.96%)
      pgsteal_normal                       0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)
      pgscan_kswapd_dma              4601307.00 (   0.00%)   4128076.00 ( -11.46%)   3912317.00 ( -17.61%)   3377165.00 ( -36.25%)
      pgscan_kswapd_normal                 0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)
      pgscan_direct_dma               629825.00 (   0.00%)    971622.00 (  35.18%)   1063938.00 (  40.80%)   1711935.00 (  63.21%)
      pgscan_direct_normal                 0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)         0.00 (   0.00%)
      pageoutrun                       27776.00 (   0.00%)     20458.00 ( -35.77%)     18763.00 ( -48.04%)     18157.00 ( -52.98%)
      allocstall                         977.00 (   0.00%)      2751.00 (  64.49%)      2098.00 (  53.43%)      5136.00 (  80.98%)
      
      Similar trends to x86-64. allocstalls are up but it's not necessarily bad.
      
      FTrace Reclaim Statistics: vmscan
      Direct reclaims                                977       2709       2098       5136
      Direct reclaim pages scanned                629825     963814    1063938    1711935
      Direct reclaim pages reclaimed               75550     242538     150904     387647
      Direct reclaim write file async I/O              0          0          0          2
      Direct reclaim write anon async I/O              0         10          0          4
      Direct reclaim write file sync I/O               0          0          0          0
      Direct reclaim write anon sync I/O               0          0          0          0
      Wake kswapd requests                        392119    1201712     571935     571921
      Kswapd wakeups                                   3          2          3          3
      Kswapd pages scanned                       4601307    4128076    3912317    3377165
      Kswapd pages reclaimed                     2432523    2318797    2312673    2144616
      Kswapd reclaim write file async I/O             20          1          1          1
      Kswapd reclaim write anon async I/O             57        132         11        121
      Kswapd reclaim write file sync I/O               0          0          0          0
      Kswapd reclaim write anon sync I/O               0          0          0          0
      Time stalled direct reclaim (seconds)         6.19       7.30      13.04      10.88
      Time kswapd awake (seconds)                  21.73      26.51      25.55      23.90
      
      Total pages scanned                        5231132   5091890   4976255   5089100
      Total pages reclaimed                      2508073   2561335   2463577   2532263
      %age total pages scanned/reclaimed          47.95%    50.30%    49.51%    49.76%
      %age total pages scanned/written             0.00%     0.00%     0.00%     0.00%
      %age  file pages scanned/written             0.00%     0.00%     0.00%     0.00%
      Percentage Time Spent Direct Reclaim        18.89%    20.65%    32.65%    27.65%
      Percentage Time kswapd Awake                72.39%    80.68%    78.21%    77.40%
      
      Again, a similar trend that the congestion_wait changes mean that direct
      reclaim scans more pages but the overall number of pages scanned while
      slightly reduced, are very similar.  The ratio of scanning/reclaimed
      remains roughly similar.  The downside is that kswapd and direct reclaim
      was awake longer and for a larger percentage of the overall workload.
      It's possible there were big differences in the amount of time spent
      reclaiming slab pages between the different kernels which is plausible
      considering that the micro tests runs after fsmark and sysbench.
      
      Trace Reclaim Statistics: congestion_wait
      Direct number congest     waited               845       1312        104          0
      Direct time   congest     waited           19416ms    26560ms     7544ms        0ms
      Direct full   congest     waited               745       1105         72          0
      Direct number conditional waited                 0          0       1322       2935
      Direct time   conditional waited               0ms        0ms       12ms      312ms
      Direct full   conditional waited                 0          0          0          3
      KSwapd number congest     waited                39        102         75         63
      KSwapd time   congest     waited            2484ms     6760ms     5756ms     3716ms
      KSwapd full   congest     waited                20         48         46         25
      KSwapd number conditional waited                 0          0          0          0
      KSwapd time   conditional waited               0ms        0ms        0ms        0ms
      KSwapd full   conditional waited                 0          0          0          0
      
      The vanilla kernel spent 20 seconds asleep in direct reclaim and only
      312ms asleep with the patches.  The time kswapd spent congest waited was
      also reduced by a large factor.
      
      MMTests Statistics: duration
      ser/Sys Time Running Test (seconds)         26.58     28.05      26.9     28.47
      Total Elapsed Time (seconds)                 30.02     32.86     32.67     30.88
      
      With all patches applies, the completion times are very similar.
      
      X86-64 STRESS-HIGHALLOC
                      traceonly-v2r2     lowlumpy-v2r3  waitcongest-v2r3waitwriteback-v2r4
      Pass 1          82.00 ( 0.00%)    84.00 ( 2.00%)    85.00 ( 3.00%)    85.00 ( 3.00%)
      Pass 2          90.00 ( 0.00%)    87.00 (-3.00%)    88.00 (-2.00%)    89.00 (-1.00%)
      At Rest         92.00 ( 0.00%)    90.00 (-2.00%)    90.00 (-2.00%)    91.00 (-1.00%)
      
      Success figures across the board are broadly similar.
      
                      traceonly-v2r2     lowlumpy-v2r3  waitcongest-v2r3waitwriteback-v2r4
      Direct reclaims                               1045        944        886        887
      Direct reclaim pages scanned                135091     119604     109382     101019
      Direct reclaim pages reclaimed               88599      47535      47863      46671
      Direct reclaim write file async I/O            494        283        465        280
      Direct reclaim write anon async I/O          29357      13710      16656      13462
      Direct reclaim write file sync I/O             154          2          2          3
      Direct reclaim write anon sync I/O           14594        571        509        561
      Wake kswapd requests                          7491        933        872        892
      Kswapd wakeups                                 814        778        731        780
      Kswapd pages scanned                       7290822   15341158   11916436   13703442
      Kswapd pages reclaimed                     3587336    3142496    3094392    3187151
      Kswapd reclaim write file async I/O          91975      32317      28022      29628
      Kswapd reclaim write anon async I/O        1992022     789307     829745     849769
      Kswapd reclaim write file sync I/O               0          0          0          0
      Kswapd reclaim write anon sync I/O               0          0          0          0
      Time stalled direct reclaim (seconds)      4588.93    2467.16    2495.41    2547.07
      Time kswapd awake (seconds)                2497.66    1020.16    1098.06    1176.82
      
      Total pages scanned                        7425913  15460762  12025818  13804461
      Total pages reclaimed                      3675935   3190031   3142255   3233822
      %age total pages scanned/reclaimed          49.50%    20.63%    26.13%    23.43%
      %age total pages scanned/written            28.66%     5.41%     7.28%     6.47%
      %age  file pages scanned/written             1.25%     0.21%     0.24%     0.22%
      Percentage Time Spent Direct Reclaim        57.33%    42.15%    42.41%    42.99%
      Percentage Time kswapd Awake                43.56%    27.87%    29.76%    31.25%
      
      Scanned/reclaimed ratios again look good with big improvements in
      efficiency.  The Scanned/written ratios also look much improved.  With a
      better scanned/written ration, there is an expectation that IO would be
      more efficient and indeed, the time spent in direct reclaim is much
      reduced by the full series and kswapd spends a little less time awake.
      
      Overall, indications here are that allocations were happening much faster
      and this can be seen with a graph of the latency figures as the
      allocations were taking place
      http://www.csn.ul.ie/~mel/postings/vmscanreduce-20101509/highalloc-interlatency-hydra-mean.ps
      
      FTrace Reclaim Statistics: congestion_wait
      Direct number congest     waited              1333        204        169          4
      Direct time   congest     waited           78896ms     8288ms     7260ms      200ms
      Direct full   congest     waited               756         92         69          2
      Direct number conditional waited                 0          0         26        186
      Direct time   conditional waited               0ms        0ms        0ms     2504ms
      Direct full   conditional waited                 0          0          0         25
      KSwapd number congest     waited                 4        395        227        282
      KSwapd time   congest     waited             384ms    25136ms    10508ms    18380ms
      KSwapd full   congest     waited                 3        232         98        176
      KSwapd number conditional waited                 0          0          0          0
      KSwapd time   conditional waited               0ms        0ms        0ms        0ms
      KSwapd full   conditional waited                 0          0          0          0
      KSwapd full   conditional waited               318          0        312          9
      
      Overall, the time spent speeping is reduced.  kswapd is still hitting
      congestion_wait() but that is because there are callers remaining where it
      wasn't clear in advance if they should be changed to wait_iff_congested()
      or not.  Overall the sleep imes are reduced though - from 79ish seconds to
      about 19.
      
      MMTests Statistics: duration
      User/Sys Time Running Test (seconds)       3415.43   3386.65   3388.39    3377.5
      Total Elapsed Time (seconds)               5733.48   3660.33   3689.41   3765.39
      
      With the full series, the time to complete the tests are reduced by 30%
      
      PPC64 STRESS-HIGHALLOC
                      traceonly-v2r2     lowlumpy-v2r3  waitcongest-v2r3waitwriteback-v2r4
      Pass 1          17.00 ( 0.00%)    34.00 (17.00%)    38.00 (21.00%)    43.00 (26.00%)
      Pass 2          25.00 ( 0.00%)    37.00 (12.00%)    42.00 (17.00%)    46.00 (21.00%)
      At Rest         49.00 ( 0.00%)    43.00 (-6.00%)    45.00 (-4.00%)    51.00 ( 2.00%)
      
      Success rates there are *way* up particularly considering that the 16MB
      huge pages on PPC64 mean that it's always much harder to allocate them.
      
      FTrace Reclaim Statistics: vmscan
                    stress-highalloc  stress-highalloc  stress-highalloc  stress-highalloc
                      traceonly-v2r2     lowlumpy-v2r3  waitcongest-v2r3waitwriteback-v2r4
      Direct reclaims                                499        505        564        509
      Direct reclaim pages scanned                223478      41898      51818      45605
      Direct reclaim pages reclaimed              137730      21148      27161      23455
      Direct reclaim write file async I/O            399        136        162        136
      Direct reclaim write anon async I/O          46977       2865       4686       3998
      Direct reclaim write file sync I/O              29          0          1          3
      Direct reclaim write anon sync I/O           31023        159        237        239
      Wake kswapd requests                           420        351        360        326
      Kswapd wakeups                                 185        294        249        277
      Kswapd pages scanned                      15703488   16392500   17821724   17598737
      Kswapd pages reclaimed                     5808466    2908858    3139386    3145435
      Kswapd reclaim write file async I/O         159938      18400      18717      13473
      Kswapd reclaim write anon async I/O        3467554     228957     322799     234278
      Kswapd reclaim write file sync I/O               0          0          0          0
      Kswapd reclaim write anon sync I/O               0          0          0          0
      Time stalled direct reclaim (seconds)      9665.35    1707.81    2374.32    1871.23
      Time kswapd awake (seconds)                9401.21    1367.86    1951.75    1328.88
      
      Total pages scanned                       15926966  16434398  17873542  17644342
      Total pages reclaimed                      5946196   2930006   3166547   3168890
      %age total pages scanned/reclaimed          37.33%    17.83%    17.72%    17.96%
      %age total pages scanned/written            23.27%     1.52%     1.94%     1.43%
      %age  file pages scanned/written             1.01%     0.11%     0.11%     0.08%
      Percentage Time Spent Direct Reclaim        44.55%    35.10%    41.42%    36.91%
      Percentage Time kswapd Awake                86.71%    43.58%    52.67%    41.14%
      
      While the scanning rates are slightly up, the scanned/reclaimed and
      scanned/written figures are much improved.  The time spent in direct
      reclaim and with kswapd are massively reduced, mostly by the lowlumpy
      patches.
      
      FTrace Reclaim Statistics: congestion_wait
      Direct number congest     waited               725        303        126          3
      Direct time   congest     waited           45524ms     9180ms     5936ms      300ms
      Direct full   congest     waited               487        190         52          3
      Direct number conditional waited                 0          0        200        301
      Direct time   conditional waited               0ms        0ms        0ms     1904ms
      Direct full   conditional waited                 0          0          0         19
      KSwapd number congest     waited                 0          2         23          4
      KSwapd time   congest     waited               0ms      200ms      420ms      404ms
      KSwapd full   congest     waited                 0          2          2          4
      KSwapd number conditional waited                 0          0          0          0
      KSwapd time   conditional waited               0ms        0ms        0ms        0ms
      KSwapd full   conditional waited                 0          0          0          0
      
      Not as dramatic a story here but the time spent asleep is reduced and we
      can still see what wait_iff_congested is going to sleep when necessary.
      
      MMTests Statistics: duration
      User/Sys Time Running Test (seconds)      12028.09   3157.17   3357.79   3199.16
      Total Elapsed Time (seconds)              10842.07   3138.72   3705.54   3229.85
      
      The time to complete this test goes way down.  With the full series, we
      are allocating over twice the number of huge pages in 30% of the time and
      there is a corresponding impact on the allocation latency graph available
      at.
      
      http://www.csn.ul.ie/~mel/postings/vmscanreduce-20101509/highalloc-interlatency-powyah-mean.ps
      
      
      
      This patch:
      
      Add a trace event for shrink_inactive_list() and updates the sample
      postprocessing script appropriately.  It can be used to determine how many
      pages were reclaimed and for non-lumpy reclaim where exactly the pages
      were reclaimed from.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Johannes Weiner <hannes@cmpxchg.org>
      Cc: Minchan Kim <minchan.kim@gmail.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Rik van Riel <riel@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e11da5b4
    • Shaohua Li's avatar
      vmscan: delete dead code · 66d9a986
      Shaohua Li authored
      
      
      `priority' cannot be negative here.  And the comment is obsolete.
      Signed-off-by: default avatarShaohua Li <shaohua.li@intel.com>
      Reviewed-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      66d9a986
    • Will Deacon's avatar
      mm: fix typo in mm.h when NODE_NOT_IN_PAGE_FLAGS · bce54bbf
      Will Deacon authored
      
      
      NODE_NOT_IN_PAGE_FLAGS is defined in mm.h when the node information is not
      stored in the page flags bitmap.
      
      Unfortunately, there's a typo in one of the checks for it.  This patch
      fixes it (s/NODE_NOT_IN_PAGEFLAGS/NODE_NOT_IN_PAGE_FLAGS/).  Since this
      has been around for ages, I doubt it's been causing any serious problems.
      Signed-off-by: default avatarWill Deacon <will.deacon@arm.com>
      Cc: Christoph Lameter <clameter@sgi.com>
      Cc: Mel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      bce54bbf
    • Michael Rubin's avatar
      writeback: report dirty thresholds in /proc/vmstat · 79da826a
      Michael Rubin authored
      
      
      The kernel already exposes the user desired thresholds in /proc/sys/vm
      with dirty_background_ratio and background_ratio.  But the kernel may
      alter the number requested without giving the user any indication that is
      the case.
      
      Knowing the actual ratios the kernel is honoring can help app developers
      understand how their buffered IO will be sent to the disk.
      
              $ grep threshold /proc/vmstat
              nr_dirty_threshold 409111
              nr_dirty_background_threshold 818223
      Signed-off-by: default avatarMichael Rubin <mrubin@google.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Dave Chinner <david@fromorbit.com>
      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      79da826a
    • Michael Rubin's avatar
      writeback: add /sys/devices/system/node/<node>/vmstat · 2ac39037
      Michael Rubin authored
      
      
      For NUMA node systems it is important to have visibility in memory
      characteristics.  Two of the /proc/vmstat values "nr_written" and
      "nr_dirtied" are added here.
      
      	# cat /sys/devices/system/node/node20/vmstat
      	nr_written 0
      	nr_dirtied 0
      Signed-off-by: default avatarMichael Rubin <mrubin@google.com>
      Reviewed-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Cc: Dave Chinner <david@fromorbit.com>
      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      2ac39037
    • Michael Rubin's avatar
      writeback: add nr_dirtied and nr_written to /proc/vmstat · ea941f0e
      Michael Rubin authored
      
      
      To help developers and applications gain visibility into writeback
      behaviour adding two entries to vm_stat_items and /proc/vmstat.  This will
      allow us to track the "written" and "dirtied" counts.
      
         # grep nr_dirtied /proc/vmstat
         nr_dirtied 3747
         # grep nr_written /proc/vmstat
         nr_written 3618
      Signed-off-by: default avatarMichael Rubin <mrubin@google.com>
      Reviewed-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Cc: Dave Chinner <david@fromorbit.com>
      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      ea941f0e