1. 20 Oct, 2008 2 commits
  2. 13 Sep, 2008 1 commit
    • Mel Gorman's avatar
      mm: mark the correct zone as full when scanning zonelists · 5bead2a0
      Mel Gorman authored
      
      
      The iterator for_each_zone_zonelist() uses a struct zoneref *z cursor when
      scanning zonelists to keep track of where in the zonelist it is.  The
      zoneref that is returned corresponds to the the next zone that is to be
      scanned, not the current one.  It was intended to be treated as an opaque
      list.
      
      When the page allocator is scanning a zonelist, it marks elements in the
      zonelist corresponding to zones that are temporarily full.  As the
      zonelist is being updated, it uses the cursor here;
      
        if (NUMA_BUILD)
              zlc_mark_zone_full(zonelist, z);
      
      This is intended to prevent rescanning in the near future but the zoneref
      cursor does not correspond to the zone that has been found to be full.
      This is an easy misunderstanding to make so this patch corrects the
      problem by changing zoneref cursor to be the current zone being scanned
      instead of the next one.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Andy Whitcroft <apw@shadowen.org>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: <stable@kernel.org>		[2.6.26.x]
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      5bead2a0
  3. 24 May, 2008 1 commit
  4. 30 Apr, 2008 2 commits
  5. 28 Apr, 2008 8 commits
    • Yasunori Goto's avatar
      memory hotplug: register section/node id to free · 04753278
      Yasunori Goto authored
      
      
      This patch set is to free pages which is allocated by bootmem for
      memory-hotremove.  Some structures of memory management are allocated by
      bootmem.  ex) memmap, etc.
      
      To remove memory physically, some of them must be freed according to
      circumstance.  This patch set makes basis to free those pages, and free
      memmaps.
      
      Basic my idea is using remain members of struct page to remember information
      of users of bootmem (section number or node id).  When the section is
      removing, kernel can confirm it.  By this information, some issues can be
      solved.
      
        1) When the memmap of removing section is allocated on other
           section by bootmem, it should/can be free.
        2) When the memmap of removing section is allocated on the
           same section, it shouldn't be freed. Because the section has to be
           logical memory offlined already and all pages must be isolated against
           page allocater. If it is freed, page allocator may use it which will
           be removed physically soon.
        3) When removing section has other section's memmap,
           kernel will be able to show easily which section should be removed
           before it for user. (Not implemented yet)
        4) When the above case 2), the page isolation will be able to check and skip
           memmap's page when logical memory offline (offline_pages()).
           Current page isolation code fails in this case because this page is
           just reserved page and it can't distinguish this pages can be
           removed or not. But, it will be able to do by this patch.
           (Not implemented yet.)
        5) The node information like pgdat has similar issues. But, this
           will be able to be solved too by this.
           (Not implemented yet, but, remembering node id in the pages.)
      
      Fortunately, current bootmem allocator just keeps PageReserved flags,
      and doesn't use any other members of page struct. The users of
      bootmem doesn't use them too.
      
      This patch:
      
      This is to register information which is node or section's id.  Kernel can
      distinguish which node/section uses the pages allcated by bootmem.  This is
      basis for hot-remove sections or nodes.
      Signed-off-by: default avatarYasunori Goto <y-goto@jp.fujitsu.com>
      Cc: Badari Pulavarty <pbadari@us.ibm.com>
      Cc: Yinghai Lu <yhlu.kernel@gmail.com>
      Cc: Yasunori Goto <y-goto@jp.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      04753278
    • Christoph Lameter's avatar
      mm: Get rid of __ZONE_COUNT · 97965478
      Christoph Lameter authored
      
      
      It was used to compensate because MAX_NR_ZONES was not available to the
      #ifdefs.  Export MAX_NR_ZONES via the new mechanism and get rid of
      __ZONE_COUNT.
      Signed-off-by: default avatarChristoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      97965478
    • Christoph Lameter's avatar
      pageflags: get rid of FLAGS_RESERVED · 9223b419
      Christoph Lameter authored
      
      
      NR_PAGEFLAGS specifies the number of page flags we are using.  From that we
      can calculate the number of bits leftover that can be used for zone, node (and
      maybe the sections id).  There is no need anymore for FLAGS_RESERVED if we use
      NR_PAGEFLAGS.
      
      Use the new methods to make NR_PAGEFLAGS available via the preprocessor.
      NR_PAGEFLAGS is used to calculate field boundaries in the page flags fields.
      These field widths have to be available to the preprocessor.
      Signed-off-by: default avatarChristoph Lameter <clameter@sgi.com>
      Cc: David Miller <davem@davemloft.net>
      Cc: Andy Whitcroft <apw@shadowen.org>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Mel Gorman <mel@csn.ul.ie>
      Cc: Jeremy Fitzhardinge <jeremy@goop.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      9223b419
    • Andrew Morton's avatar
      mm: make early_pfn_to_nid() a C function · b4544568
      Andrew Morton authored
      
      
      Fix this (sparc64)
      
      mm/sparse-vmemmap.c: In function `vmemmap_verify':
      mm/sparse-vmemmap.c:64: warning: unused variable `pfn'
      
      by switching to a C function which touches its arg.
      
      (reason 3,555 why macros are bad)
      
      Also, the `nid' arg was misnamed.
      Reviewed-by: default avatarChristoph Lameter <clameter@sgi.com>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Cc: Mel Gorman <mel@csn.ul.ie>
      Cc: Andi Kleen <ak@suse.de>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      b4544568
    • Mel Gorman's avatar
      mm: filter based on a nodemask as well as a gfp_mask · 19770b32
      Mel Gorman authored
      
      
      The MPOL_BIND policy creates a zonelist that is used for allocations
      controlled by that mempolicy.  As the per-node zonelist is already being
      filtered based on a zone id, this patch adds a version of __alloc_pages() that
      takes a nodemask for further filtering.  This eliminates the need for
      MPOL_BIND to create a custom zonelist.
      
      A positive benefit of this is that allocations using MPOL_BIND now use the
      local node's distance-ordered zonelist instead of a custom node-id-ordered
      zonelist.  I.e., pages will be allocated from the closest allowed node with
      available memory.
      
      [Lee.Schermerhorn@hp.com: Mempolicy: update stale documentation and comments]
      [Lee.Schermerhorn@hp.com: Mempolicy: make dequeue_huge_page_vma() obey MPOL_BIND nodemask]
      [Lee.Schermerhorn@hp.com: Mempolicy: make dequeue_huge_page_vma() obey MPOL_BIND nodemask rework]
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarChristoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Mel Gorman <mel@csn.ul.ie>
      Cc: Hugh Dickins <hugh@veritas.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      19770b32
    • Mel Gorman's avatar
      mm: have zonelist contains structs with both a zone pointer and zone_idx · dd1a239f
      Mel Gorman authored
      
      
      Filtering zonelists requires very frequent use of zone_idx().  This is costly
      as it involves a lookup of another structure and a substraction operation.  As
      the zone_idx is often required, it should be quickly accessible.  The node idx
      could also be stored here if it was found that accessing zone->node is
      significant which may be the case on workloads where nodemasks are heavily
      used.
      
      This patch introduces a struct zoneref to store a zone pointer and a zone
      index.  The zonelist then consists of an array of these struct zonerefs which
      are looked up as necessary.  Helpers are given for accessing the zone index as
      well as the node index.
      
      [kamezawa.hiroyu@jp.fujitsu.com: Suggested struct zoneref instead of embedding information in pointers]
      [hugh@veritas.com: mm-have-zonelist: fix memcg ooms]
      [hugh@veritas.com: just return do_try_to_free_pages]
      [hugh@veritas.com: do_try_to_free_pages gfp_mask redundant]
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarChristoph Lameter <clameter@sgi.com>
      Acked-by: default avatarDavid Rientjes <rientjes@google.com>
      Signed-off-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Mel Gorman <mel@csn.ul.ie>
      Cc: Christoph Lameter <clameter@sgi.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarHugh Dickins <hugh@veritas.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      dd1a239f
    • Mel Gorman's avatar
      mm: use two zonelist that are filtered by GFP mask · 54a6eb5c
      Mel Gorman authored
      
      
      Currently a node has two sets of zonelists, one for each zone type in the
      system and a second set for GFP_THISNODE allocations.  Based on the zones
      allowed by a gfp mask, one of these zonelists is selected.  All of these
      zonelists consume memory and occupy cache lines.
      
      This patch replaces the multiple zonelists per-node with two zonelists.  The
      first contains all populated zones in the system, ordered by distance, for
      fallback allocations when the target/preferred node has no free pages.  The
      second contains all populated zones in the node suitable for GFP_THISNODE
      allocations.
      
      An iterator macro is introduced called for_each_zone_zonelist() that interates
      through each zone allowed by the GFP flags in the selected zonelist.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarChristoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Mel Gorman <mel@csn.ul.ie>
      Cc: Christoph Lameter <clameter@sgi.com>
      Cc: Hugh Dickins <hugh@veritas.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      54a6eb5c
    • Harvey Harrison's avatar
      remove sparse warning for mmzone.h · ddc81ed2
      Harvey Harrison authored
      
      
      include/linux/mmzone.h:640:22: warning: potentially expensive pointer subtraction
      
      Calculate the offset into the node_zones array rather than the index
      using casts to (char *) and comparing against the index * sizeof(struct zone).
      
      On X86_32 this saves a sar, but code size increases by one byte per
      is_highmem() use due to 32-bit cmps rather than 16 bit cmps.
      
      Before:
       207:   2b 80 8c 07 00 00       sub    0x78c(%eax),%eax
       20d:   c1 f8 0b                sar    $0xb,%eax
       210:   83 f8 02                cmp    $0x2,%eax
       213:   74 16                   je     22b <kmap_atomic_prot+0x144>
       215:   83 f8 03                cmp    $0x3,%eax
       218:   0f 85 8f 00 00 00       jne    2ad <kmap_atomic_prot+0x1c6>
       21e:   83 3d 00 00 00 00 02    cmpl   $0x2,0x0
       225:   0f 85 82 00 00 00       jne    2ad <kmap_atomic_prot+0x1c6>
       22b:   64 a1 00 00 00 00       mov    %fs:0x0,%eax
      
      After:
       207:   2b 80 8c 07 00 00       sub    0x78c(%eax),%eax
       20d:   3d 00 10 00 00          cmp    $0x1000,%eax
       212:   74 18                   je     22c <kmap_atomic_prot+0x145>
       214:   3d 00 18 00 00          cmp    $0x1800,%eax
       219:   0f 85 8f 00 00 00       jne    2ae <kmap_atomic_prot+0x1c7>
       21f:   83 3d 00 00 00 00 02    cmpl   $0x2,0x0
       226:   0f 85 82 00 00 00       jne    2ae <kmap_atomic_prot+0x1c7>
       22c:   64 a1 00 00 00 00       mov    %fs:0x0,%eax
      
      [akpm@linux-foundation.org: coding-style fixes]
      Signed-off-by: default avatarHarvey Harrison <harvey.harrison@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      ddc81ed2
  6. 21 Apr, 2008 1 commit
  7. 05 Feb, 2008 1 commit
  8. 17 Oct, 2007 3 commits
    • David Rientjes's avatar
      mm: test and set zone reclaim lock before starting reclaim · d773ed6b
      David Rientjes authored
      
      
      Introduces new zone flag interface for testing and setting flags:
      
      	int zone_test_and_set_flag(struct zone *zone, zone_flags_t flag)
      
      Instead of setting and clearing ZONE_RECLAIM_LOCKED each time shrink_zone() is
      called, this flag is test and set before starting zone reclaim.  Zone reclaim
      starts in __alloc_pages() when a zone's watermark fails and the system is in
      zone_reclaim_mode.  If it's already in reclaim, there's no need to start again
      so it is simply considered full for that allocation attempt.
      
      There is a change of behavior with regard to concurrent zone shrinking.  It is
      now possible for try_to_free_pages() or kswapd to already be shrinking a
      particular zone when __alloc_pages() starts zone reclaim.  In this case, it is
      possible for two concurrent threads to invoke shrink_zone() for a single zone.
      
      This change forbids a zone to be in zone reclaim twice, which was always the
      behavior, but allows for concurrent try_to_free_pages() or kswapd shrinking
      when starting zone reclaim.
      
      Cc: Andrea Arcangeli <andrea@suse.de>
      Cc: Christoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarDavid Rientjes <rientjes@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d773ed6b
    • David Rientjes's avatar
      oom: add per-zone locking · 098d7f12
      David Rientjes authored
      
      
      OOM killer synchronization should be done with zone granularity so that memory
      policy and cpuset allocations may have their corresponding zones locked and
      allow parallel kills for other OOM conditions that may exist elsewhere in the
      system.  DMA allocations can be targeted at the zone level, which would not be
      possible if locking was done in nodes or globally.
      
      Synchronization shall be done with a variation of "trylocks." The goal is to
      put the current task to sleep and restart the failed allocation attempt later
      if the trylock fails.  Otherwise, the OOM killer is invoked.
      
      Each zone in the zonelist that __alloc_pages() was called with is checked for
      the newly-introduced ZONE_OOM_LOCKED flag.  If any zone has this flag present,
      the "trylock" to serialize the OOM killer fails and returns zero.  Otherwise,
      all the zones have ZONE_OOM_LOCKED set and the try_set_zone_oom() function
      returns non-zero.
      
      Cc: Andrea Arcangeli <andrea@suse.de>
      Cc: Christoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarDavid Rientjes <rientjes@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      098d7f12
    • David Rientjes's avatar
      oom: change all_unreclaimable zone member to flags · e815af95
      David Rientjes authored
      
      
      Convert the int all_unreclaimable member of struct zone to unsigned long
      flags.  This can now be used to specify several different zone flags such as
      all_unreclaimable and reclaim_in_progress, which can now be removed and
      converted to a per-zone flag.
      
      Flags are set and cleared as follows:
      
      	zone_set_flag(struct zone *zone, zone_flags_t flag)
      	zone_clear_flag(struct zone *zone, zone_flags_t flag)
      
      Defines the first zone flags, ZONE_ALL_UNRECLAIMABLE and ZONE_RECLAIM_LOCKED,
      which have the same semantics as the old zone->all_unreclaimable and
      zone->reclaim_in_progress, respectively.  Also converts all current users that
      set or clear either flag to use the new interface.
      
      Helper functions are defined to test the flags:
      
      	int zone_is_all_unreclaimable(const struct zone *zone)
      	int zone_is_reclaim_locked(const struct zone *zone)
      
      All flag operators are of the atomic variety because there are currently
      readers that are implemented that do not take zone->lock.
      
      [akpm@linux-foundation.org: add needed include]
      Cc: Andrea Arcangeli <andrea@suse.de>
      Acked-by: default avatarChristoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarDavid Rientjes <rientjes@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e815af95
  9. 16 Oct, 2007 14 commits
    • KAMEZAWA Hiroyuki's avatar
      memory unplug: page isolation · a5d76b54
      KAMEZAWA Hiroyuki authored
      
      
      Implement generic chunk-of-pages isolation method by using page grouping ops.
      
      This patch add MIGRATE_ISOLATE to MIGRATE_TYPES. By this
       - MIGRATE_TYPES increases.
       - bitmap for migratetype is enlarged.
      
      pages of MIGRATE_ISOLATE migratetype will not be allocated even if it is free.
      By this, you can isolated *freed* pages from users. How-to-free pages is not
      a purpose of this patch. You may use reclaim and migrate codes to free pages.
      
      If start_isolate_page_range(start,end) is called,
       - migratetype of the range turns to be MIGRATE_ISOLATE  if
         its type is MIGRATE_MOVABLE. (*) this check can be updated if other
         memory reclaiming works make progress.
       - MIGRATE_ISOLATE is not on migratetype fallback list.
       - All free pages and will-be-freed pages are isolated.
      To check all pages in the range are isolated or not,  use test_pages_isolated(),
      To cancel isolation, use undo_isolate_page_range().
      
      Changes V6 -> V7
       - removed unnecessary #ifdef
      
      There are HOLES_IN_ZONE handling codes...I'm glad if we can remove them..
      Signed-off-by: default avatarYasunori Goto <y-goto@jp.fujitsu.com>
      Signed-off-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      a5d76b54
    • Mel Gorman's avatar
      Print out statistics in relation to fragmentation avoidance to /proc/pagetypeinfo · 467c996c
      Mel Gorman authored
      
      
      This patch provides fragmentation avoidance statistics via /proc/pagetypeinfo.
       The information is collected only on request so there is no runtime overhead.
       The statistics are in three parts:
      
      The first part prints information on the size of blocks that pages are
      being grouped on and looks like
      
      Page block order: 10
      Pages per block:  1024
      
      The second part is a more detailed version of /proc/buddyinfo and looks like
      
      Free pages count per migrate type at order       0      1      2      3      4      5      6      7      8      9     10
      Node    0, zone      DMA, type    Unmovable      0      0      0      0      0      0      0      0      0      0      0
      Node    0, zone      DMA, type  Reclaimable      1      0      0      0      0      0      0      0      0      0      0
      Node    0, zone      DMA, type      Movable      0      0      0      0      0      0      0      0      0      0      0
      Node    0, zone      DMA, type      Reserve      0      4      4      0      0      0      0      1      0      1      0
      Node    0, zone   Normal, type    Unmovable    111      8      4      4      2      3      1      0      0      0      0
      Node    0, zone   Normal, type  Reclaimable    293     89      8      0      0      0      0      0      0      0      0
      Node    0, zone   Normal, type      Movable      1      6     13      9      7      6      3      0      0      0      0
      Node    0, zone   Normal, type      Reserve      0      0      0      0      0      0      0      0      0      0      4
      
      The third part looks like
      
      Number of blocks type     Unmovable  Reclaimable      Movable      Reserve
      Node 0, zone      DMA            0            1            2            1
      Node 0, zone   Normal            3           17           94            4
      
      To walk the zones within a node with interrupts disabled, walk_zones_in_node()
      is introduced and shared between /proc/buddyinfo, /proc/zoneinfo and
      /proc/pagetypeinfo to reduce code duplication.  It seems specific to what
      vmstat.c requires but could be broken out as a general utility function in
      mmzone.c if there were other other potential users.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Acked-by: default avatarChristoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      467c996c
    • Mel Gorman's avatar
      Do not depend on MAX_ORDER when grouping pages by mobility · d9c23400
      Mel Gorman authored
      
      
      Currently mobility grouping works at the MAX_ORDER_NR_PAGES level.  This makes
      sense for the majority of users where this is also the huge page size.
      However, on platforms like ia64 where the huge page size is runtime
      configurable it is desirable to group at a lower order.  On x86_64 and
      occasionally on x86, the hugepage size may not always be MAX_ORDER_NR_PAGES.
      
      This patch groups pages together based on the value of HUGETLB_PAGE_ORDER.  It
      uses a compile-time constant if possible and a variable where the huge page
      size is runtime configurable.
      
      It is assumed that grouping should be done at the lowest sensible order and
      that the user would not want to override this.  If this is not true,
      page_block order could be forced to a variable initialised via a boot-time
      kernel parameter.
      
      One potential issue with this patch is that IA64 now parses hugepagesz with
      early_param() instead of __setup().  __setup() is called after the memory
      allocator has been initialised and the pageblock bitmaps already setup.  In
      tests on one IA64 there did not seem to be any problem with using
      early_param() and in fact may be more correct as it guarantees the parameter
      is handled before the parsing of hugepages=.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Acked-by: default avatarChristoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d9c23400
    • Mel Gorman's avatar
      don't group high order atomic allocations · 64c5e135
      Mel Gorman authored
      
      
      Grouping high-order atomic allocations together was intended to allow
      bursty users of atomic allocations to work such as e1000 in situations
      where their preallocated buffers were depleted.  This did not work in at
      least one case with a wireless network adapter needing order-1 allocations
      frequently.  To resolve that, the free pages used for min_free_kbytes were
      moved to separate contiguous blocks with the patch
      bias-the-location-of-pages-freed-for-min_free_kbytes-in-the-same-max_order_nr_pages-blocks.
      
      It is felt that keeping the free pages in the same contiguous blocks should
      be sufficient for bursty short-lived high-order atomic allocations to
      succeed, maybe even with the e1000.  Even if there is a failure, increasing
      the value of min_free_kbytes will free pages as contiguous bloks in
      contrast to the standard buddy allocator which makes no attempt to keep the
      minimum number of free pages contiguous.
      
      This patch backs out grouping high order atomic allocations together to
      determine if it is really needed or not.  If a new report comes in about
      high-order atomic allocations failing, the feature can be reintroduced to
      determine if it fixes the problem or not.  As a side-effect, this patch
      reduces by 1 the number of bits required to track the mobility type of
      pages within a MAX_ORDER_NR_PAGES block.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      64c5e135
    • Mel Gorman's avatar
      remove PAGE_GROUP_BY_MOBILITY · ac0e5b7a
      Mel Gorman authored
      
      
      Grouping pages by mobility can be disabled at compile-time. This was
      considered undesirable by a number of people. However, in the current stack of
      patches, it is not a simple case of just dropping the configurable patch as it
      would cause merge conflicts.  This patch backs out the configuration option.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      ac0e5b7a
    • Mel Gorman's avatar
      Bias the location of pages freed for min_free_kbytes in the same MAX_ORDER_NR_PAGES blocks · 56fd56b8
      Mel Gorman authored
      
      
      The standard buddy allocator always favours the smallest block of pages.
      The effect of this is that the pages free to satisfy min_free_kbytes tends
      to be preserved since boot time at the same location of memory ffor a very
      long time and as a contiguous block.  When an administrator sets the
      reserve at 16384 at boot time, it tends to be the same MAX_ORDER blocks
      that remain free.  This allows the occasional high atomic allocation to
      succeed up until the point the blocks are split.  In practice, it is
      difficult to split these blocks but when they do split, the benefit of
      having min_free_kbytes for contiguous blocks disappears.  Additionally,
      increasing min_free_kbytes once the system has been running for some time
      has no guarantee of creating contiguous blocks.
      
      On the other hand, CONFIG_PAGE_GROUP_BY_MOBILITY favours splitting large
      blocks when there are no free pages of the appropriate type available.  A
      side-effect of this is that all blocks in memory tends to be used up and
      the contiguous free blocks from boot time are not preserved like in the
      vanilla allocator.  This can cause a problem if a new caller is unwilling
      to reclaim or does not reclaim for long enough.
      
      A failure scenario was found for a wireless network device allocating
      order-1 atomic allocations but the allocations were not intense or frequent
      enough for a whole block of pages to be preserved for MIGRATE_HIGHALLOC.
      This was reproduced on a desktop by booting with mem=256mb, forcing the
      driver to allocate at order-1, running a bittorrent client (downloading a
      debian ISO) and building a kernel with -j2.
      
      This patch addresses the problem on the desktop machine booted with
      mem=256mb.  It works by setting aside a reserve of MAX_ORDER_NR_PAGES
      blocks, the number of which depends on the value of min_free_kbytes.  These
      blocks are only fallen back to when there is no other free pages.  Then the
      smallest possible page is used just like the normal buddy allocator instead
      of the largest possible page to preserve contiguous pages The pages in free
      lists in the reserve blocks are never taken for another migrate type.  The
      results is that even if min_free_kbytes is set to a low value, contiguous
      blocks will be preserved in the MIGRATE_RESERVE blocks.
      
      This works better than the vanilla allocator because if min_free_kbytes is
      increased, a new reserve block will be chosen based on the location of
      reclaimable pages and the block will free up as contiguous pages.  In the
      vanilla allocator, no effort is made to target a block of pages to free as
      contiguous pages and min_free_kbytes pages are scattered randomly.
      
      This effect has been observed on the test machine.  min_free_kbytes was set
      initially low but it was kept as a contiguous free block within
      MIGRATE_RESERVE.  min_free_kbytes was then set to a higher value and over a
      period of time, the free blocks were within the reserve and coalescing.
      How long it takes to free up depends on how quickly LRU is rotating.
      Amusingly, this means that more activity will free the blocks faster.
      
      This mechanism potentially replaces MIGRATE_HIGHALLOC as it may be more
      effective than grouping contiguous free pages together.  It all depends on
      whether the number of active atomic high allocations exceeds
      min_free_kbytes or not.  If the number of active allocations exceeds
      min_free_kbytes, it's worth it but maybe in that situation, min_free_kbytes
      should be set higher.  Once there are no more reports of allocation
      failures, a patch will be submitted that backs out MIGRATE_HIGHALLOC and
      see if the reports stay missing.
      
      Credit to Mariusz Kozlowski for discovering the problem, describing the
      failure scenario and testing patches and scenarios.
      
      [akpm@linux-foundation.org: cleanups]
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      56fd56b8
    • Mel Gorman's avatar
      Fix corruption of memmap on IA64 SPARSEMEM when mem_section is not a power of 2 · 5c0e3066
      Mel Gorman authored
      
      
      There are problems in the use of SPARSEMEM and pageblock flags that causes
      problems on ia64.
      
      The first part of the problem is that units are incorrect in
      SECTION_BLOCKFLAGS_BITS computation.  This results in a map_section's
      section_mem_map being treated as part of a bitmap which isn't good.  This
      was evident with an invalid virtual address when mem_init attempted to free
      bootmem pages while relinquishing control from the bootmem allocator.
      
      The second part of the problem occurs because the pageblock flags bitmap is
      be located with the mem_section.  The SECTIONS_PER_ROOT computation using
      sizeof (mem_section) may not be a power of 2 depending on the size of the
      bitmap.  This renders masks and other such things not power of 2 base.
      This issue was seen with SPARSEMEM_EXTREME on ia64.  This patch moves the
      bitmap outside of mem_section and uses a pointer instead in the
      mem_section.  The bitmaps are allocated when the section is being
      initialised.
      
      Note that sparse_early_usemap_alloc() does not use alloc_remap() like
      sparse_early_mem_map_alloc().  The allocation required for the bitmap on
      x86, the only architecture that uses alloc_remap is typically smaller than
      a cache line.  alloc_remap() pads out allocations to the cache size which
      would be a needless waste.
      
      Credit to Bob Picco for identifying the original problem and effecting a
      fix for the SECTION_BLOCKFLAGS_BITS calculation.  Credit to Andy Whitcroft
      for devising the best way of allocating the bitmaps only when required for
      the section.
      
      [wli@holomorphy.com: warning fix]
      Signed-off-by: default avatarBob Picco <bob.picco@hp.com>
      Signed-off-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: "Luck, Tony" <tony.luck@intel.com>
      Signed-off-by: default avatarWilliam Irwin <bill.irwin@oracle.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      5c0e3066
    • Mel Gorman's avatar
      Group high-order atomic allocations · e010487d
      Mel Gorman authored
      
      
      In rare cases, the kernel needs to allocate a high-order block of pages
      without sleeping.  For example, this is the case with e1000 cards configured
      to use jumbo frames.  Migrating or reclaiming pages in this situation is not
      an option.
      
      This patch groups these allocations together as much as possible by adding a
      new MIGRATE_TYPE.  The MIGRATE_HIGHATOMIC type are exactly what they sound
      like.  Care is taken that pages of other migrate types do not use the same
      blocks as high-order atomic allocations.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e010487d
    • Mel Gorman's avatar
      Group short-lived and reclaimable kernel allocations · e12ba74d
      Mel Gorman authored
      
      
      This patch marks a number of allocations that are either short-lived such as
      network buffers or are reclaimable such as inode allocations.  When something
      like updatedb is called, long-lived and unmovable kernel allocations tend to
      be spread throughout the address space which increases fragmentation.
      
      This patch groups these allocations together as much as possible by adding a
      new MIGRATE_TYPE.  The MIGRATE_RECLAIMABLE type is for allocations that can be
      reclaimed on demand, but not moved.  i.e.  they can be migrated by deleting
      them and re-reading the information from elsewhere.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Andy Whitcroft <apw@shadowen.org>
      Cc: Christoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e12ba74d
    • Mel Gorman's avatar
      Add a configure option to group pages by mobility · b92a6edd
      Mel Gorman authored
      
      
      The grouping mechanism has some memory overhead and a more complex allocation
      path.  This patch allows the strategy to be disabled for small memory systems
      or if it is known the workload is suffering because of the strategy.  It also
      acts to show where the page groupings strategy interacts with the standard
      buddy allocator.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarJoel Schopp <jschopp@austin.ibm.com>
      Cc: Andy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      b92a6edd
    • Mel Gorman's avatar
      Split the free lists for movable and unmovable allocations · b2a0ac88
      Mel Gorman authored
      
      
      This patch adds the core of the fragmentation reduction strategy.  It works by
      grouping pages together based on their ability to migrate or be reclaimed.
      Basically, it works by breaking the list in zone->free_area list into
      MIGRATE_TYPES number of lists.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      b2a0ac88
    • Mel Gorman's avatar
      Add a bitmap that is used to track flags affecting a block of pages · 835c134e
      Mel Gorman authored
      
      
      Here is the latest revision of the anti-fragmentation patches.  Of particular
      note in this version is special treatment of high-order atomic allocations.
      Care is taken to group them together and avoid grouping pages of other types
      near them.  Artifical tests imply that it works.  I'm trying to get the
      hardware together that would allow setting up of a "real" test.  If anyone
      already has a setup and test that can trigger the atomic-allocation problem,
      I'd appreciate a test of these patches and a report.  The second major change
      is that these patches will apply cleanly with patches that implement
      anti-fragmentation through zones.
      
      kernbench shows effectively no performance difference varying between -0.2%
      and +2% on a variety of test machines.  Success rates for huge page allocation
      are dramatically increased.  For example, on a ppc64 machine, the vanilla
      kernel was only able to allocate 1% of memory as a hugepage and this was due
      to a single hugepage reserved as min_free_kbytes.  With these patches applied,
      17% was allocatable as superpages.  With reclaim-related fixes from Andy
      Whitcroft, it was 40% and further reclaim-related improvements should increase
      this further.
      
      Changelog Since V28
      o Group high-order atomic allocations together
      o It is no longer required to set min_free_kbytes to 10% of memory. A value
        of 16384 in most cases will be sufficient
      o Now applied with zone-based anti-fragmentation
      o Fix incorrect VM_BUG_ON within buffered_rmqueue()
      o Reorder the stack so later patches do not back out work from earlier patches
      o Fix bug were journal pages were being treated as movable
      o Bias placement of non-movable pages to lower PFNs
      o More agressive clustering of reclaimable pages in reactions to workloads
        like updatedb that flood the size of inode caches
      
      Changelog Since V27
      
      o Renamed anti-fragmentation to Page Clustering. Anti-fragmentation was giving
        the mistaken impression that it was the 100% solution for high order
        allocations. Instead, it greatly increases the chances high-order
        allocations will succeed and lays the foundation for defragmentation and
        memory hot-remove to work properly
      o Redefine page groupings based on ability to migrate or reclaim instead of
        basing on reclaimability alone
      o Get rid of spurious inits
      o Per-cpu lists are no longer split up per-type. Instead the per-cpu list is
        searched for a page of the appropriate type
      o Added more explanation commentary
      o Fix up bug in pageblock code where bitmap was used before being initalised
      
      Changelog Since V26
      o Fix double init of lists in setup_pageset
      
      Changelog Since V25
      o Fix loop order of for_each_rclmtype_order so that order of loop matches args
      o gfpflags_to_rclmtype uses gfp_t instead of unsigned long
      o Rename get_pageblock_type() to get_page_rclmtype()
      o Fix alignment problem in move_freepages()
      o Add mechanism for assigning flags to blocks of pages instead of page->flags
      o On fallback, do not examine the preferred list of free pages a second time
      
      The purpose of these patches is to reduce external fragmentation by grouping
      pages of related types together.  When pages are migrated (or reclaimed under
      memory pressure), large contiguous pages will be freed.
      
      This patch works by categorising allocations by their ability to migrate;
      
      Movable - The pages may be moved with the page migration mechanism. These are
      	generally userspace pages.
      
      Reclaimable - These are allocations for some kernel caches that are
      	reclaimable or allocations that are known to be very short-lived.
      
      Unmovable - These are pages that are allocated by the kernel that
      	are not trivially reclaimed. For example, the memory allocated for a
      	loaded module would be in this category. By default, allocations are
      	considered to be of this type
      
      HighAtomic - These are high-order allocations belonging to callers that
      	cannot sleep or perform any IO. In practice, this is restricted to
      	jumbo frame allocation for network receive. It is assumed that the
      	allocations are short-lived
      
      Instead of having one MAX_ORDER-sized array of free lists in struct free_area,
      there is one for each type of reclaimability.  Once a 2^MAX_ORDER block of
      pages is split for a type of allocation, it is added to the free-lists for
      that type, in effect reserving it.  Hence, over time, pages of the different
      types can be clustered together.
      
      When the preferred freelists are expired, the largest possible block is taken
      from an alternative list.  Buddies that are split from that large block are
      placed on the preferred allocation-type freelists to mitigate fragmentation.
      
      This implementation gives best-effort for low fragmentation in all zones.
      Ideally, min_free_kbytes needs to be set to a value equal to 4 * (1 <<
      (MAX_ORDER-1)) pages in most cases.  This would be 16384 on x86 and x86_64 for
      example.
      
      Our tests show that about 60-70% of physical memory can be allocated on a
      desktop after a few days uptime.  In benchmarks and stress tests, we are
      finding that 80% of memory is available as contiguous blocks at the end of the
      test.  To compare, a standard kernel was getting < 1% of memory as large pages
      on a desktop and about 8-12% of memory as large pages at the end of stress
      tests.
      
      Following this email are 12 patches that implement thie page grouping feature.
       The first patch introduces a mechanism for storing flags related to a whole
      block of pages.  Then allocations are split between movable and all other
      allocations.  Following that are patches to deal with per-cpu pages and make
      the mechanism configurable.  The next patch moves free pages between lists
      when partially allocated blocks are used for pages of another migrate type.
      The second last patch groups reclaimable kernel allocations such as inode
      caches together.  The final patch related to groupings keeps high-order atomic
      allocations.
      
      The last two patches are more concerned with control of fragmentation.  The
      second last patch biases placement of non-movable allocations towards the
      start of memory.  This is with a view of supporting memory hot-remove of DIMMs
      with higher PFNs in the future.  The biasing could be enforced a lot heavier
      but it would cost.  The last patch agressively clusters reclaimable pages like
      inode caches together.
      
      The fragmentation reduction strategy needs to track if pages within a block
      can be moved or reclaimed so that pages are freed to the appropriate list.
      This patch adds a bitmap for flags affecting a whole a MAX_ORDER block of
      pages.
      
      In non-SPARSEMEM configurations, the bitmap is stored in the struct zone and
      allocated during initialisation.  SPARSEMEM statically allocates the bitmap in
      a struct mem_section so that bitmaps do not have to be resized during memory
      hotadd.  This wastes a small amount of memory per unused section (usually
      sizeof(unsigned long)) but the complexity of dynamically allocating the memory
      is quite high.
      
      Additional credit to Andy Whitcroft who reviewed up an earlier implementation
      of the mechanism an suggested how to make it a *lot* cleaner.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Andy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      835c134e
    • Christoph Lameter's avatar
      Memoryless nodes: Fix GFP_THISNODE behavior · 523b9458
      Christoph Lameter authored
      
      
      GFP_THISNODE checks that the zone selected is within the pgdat (node) of the
      first zone of a nodelist.  That only works if the node has memory.  A
      memoryless node will have its first node on another pgdat (node).
      
      GFP_THISNODE currently will return simply memory on the first pgdat.  Thus it
      is returning memory on other nodes.  GFP_THISNODE should fail if there is no
      local memory on a node.
      
      Add a new set of zonelists for each node that only contain the nodes that
      belong to the zones itself so that no fallback is possible.
      
      Then modify gfp_type to pickup the right zone based on the presence of
      __GFP_THISNODE.
      
      Drop the existing GFP_THISNODE checks from the page_allocators hot path.
      Signed-off-by: default avatarChristoph Lameter <clameter@sgi.com>
      Acked-by: default avatarNishanth Aravamudan <nacc@us.ibm.com>
      Tested-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Acked-by: default avatarBob Picco <bob.picco@hp.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Mel Gorman <mel@skynet.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      523b9458
    • Andy Whitcroft's avatar
      sparsemem: record when a section has a valid mem_map · 540557b9
      Andy Whitcroft authored
      
      
      We have flags to indicate whether a section actually has a valid mem_map
      associated with it.  This is never set and we rely solely on the present bit
      to indicate a section is valid.  By definition a section is not valid if it
      has no mem_map and there is a window during init where the present bit is set
      but there is no mem_map, during which pfn_valid() will return true
      incorrectly.
      
      Use the existing SECTION_HAS_MEM_MAP flag to indicate the presence of a valid
      mem_map.  Switch valid_section{,_nr} and pfn_valid() to this bit.  Add a new
      present_section{,_nr} and pfn_present() interfaces for those users who care to
      know that a section is going to be valid.
      
      [akpm@linux-foundation.org: coding-syle fixes]
      Signed-off-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Acked-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Christoph Lameter <clameter@sgi.com>
      Cc: "Luck, Tony" <tony.luck@intel.com>
      Cc: Andi Kleen <ak@suse.de>
      Cc: "David S. Miller" <davem@davemloft.net>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      540557b9
  10. 22 Aug, 2007 1 commit
    • Mel Gorman's avatar
      Apply memory policies to top two highest zones when highest zone is ZONE_MOVABLE · b377fd39
      Mel Gorman authored
      
      
      The NUMA layer only supports NUMA policies for the highest zone.  When
      ZONE_MOVABLE is configured with kernelcore=, the the highest zone becomes
      ZONE_MOVABLE.  The result is that policies are only applied to allocations
      like anonymous pages and page cache allocated from ZONE_MOVABLE when the
      zone is used.
      
      This patch applies policies to the two highest zones when the highest zone
      is ZONE_MOVABLE.  As ZONE_MOVABLE consists of pages from the highest "real"
      zone, it's always functionally equivalent.
      
      The patch has been tested on a variety of machines both NUMA and non-NUMA
      covering x86, x86_64 and ppc64.  No abnormal results were seen in
      kernbench, tbench, dbench or hackbench.  It passes regression tests from
      the numactl package with and without kernelcore= once numactl tests are
      patched to wait for vmstat counters to update.
      
      akpm: this is the nasty hack to fix NUMA mempolicies in the presence of
      ZONE_MOVABLE and kernelcore= in 2.6.23.  Christoph says "For .24 either merge
      the mobility or get the other solution that Mel is working on.  That solution
      would only use a single zonelist per node and filter on the fly.  That may
      help performance and also help to make memory policies work better."
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Tested-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Acked-by: default avatarChristoph Lameter <clameter@sgi.com>
      Cc: Andi Kleen <ak@suse.de>
      Cc: Paul Mundt <lethal@linux-sh.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      b377fd39
  11. 31 Jul, 2007 1 commit
  12. 17 Jul, 2007 2 commits
    • Andy Whitcroft's avatar
      Lumpy Reclaim V4 · 5ad333eb
      Andy Whitcroft authored
      
      
      When we are out of memory of a suitable size we enter reclaim.  The current
      reclaim algorithm targets pages in LRU order, which is great for fairness at
      order-0 but highly unsuitable if you desire pages at higher orders.  To get
      pages of higher order we must shoot down a very high proportion of memory;
      >95% in a lot of cases.
      
      This patch set adds a lumpy reclaim algorithm to the allocator.  It targets
      groups of pages at the specified order anchored at the end of the active and
      inactive lists.  This encourages groups of pages at the requested orders to
      move from active to inactive, and active to free lists.  This behaviour is
      only triggered out of direct reclaim when higher order pages have been
      requested.
      
      This patch set is particularly effective when utilised with an
      anti-fragmentation scheme which groups pages of similar reclaimability
      together.
      
      This patch set is based on Peter Zijlstra's lumpy reclaim V2 patch which forms
      the foundation.  Credit to Mel Gorman for sanitity checking.
      
      Mel said:
      
        The patches have an application with hugepage pool resizing.
      
        When lumpy-reclaim is used used with ZONE_MOVABLE, the hugepages pool can
        be resized with greater reliability.  Testing on a desktop machine with 2GB
        of RAM showed that growing the hugepage pool with ZONE_MOVABLE on it's own
        was very slow as the success rate was quite low.  Without lumpy-reclaim,
        each attempt to grow the pool by 100 pages would yield 1 or 2 hugepages.
        With lumpy-reclaim, getting 40 to 70 hugepages on each attempt was typical.
      
      [akpm@osdl.org: ia64 pfn_to_nid fixes and loop cleanup]
      [bunk@stusta.de: static declarations for internal functions]
      [a.p.zijlstra@chello.nl: initial lumpy V2 implementation]
      Signed-off-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Acked-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Acked-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Bob Picco <bob.picco@hp.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      5ad333eb
    • Mel Gorman's avatar
      Create the ZONE_MOVABLE zone · 2a1e274a
      Mel Gorman authored
      
      
      The following 8 patches against 2.6.20-mm2 create a zone called ZONE_MOVABLE
      that is only usable by allocations that specify both __GFP_HIGHMEM and
      __GFP_MOVABLE.  This has the effect of keeping all non-movable pages within a
      single memory partition while allowing movable allocations to be satisfied
      from either partition.  The patches may be applied with the list-based
      anti-fragmentation patches that groups pages together based on mobility.
      
      The size of the zone is determined by a kernelcore= parameter specified at
      boot-time.  This specifies how much memory is usable by non-movable
      allocations and the remainder is used for ZONE_MOVABLE.  Any range of pages
      within ZONE_MOVABLE can be released by migrating the pages or by reclaiming.
      
      When selecting a zone to take pages from for ZONE_MOVABLE, there are two
      things to consider.  First, only memory from the highest populated zone is
      used for ZONE_MOVABLE.  On the x86, this is probably going to be ZONE_HIGHMEM
      but it would be ZONE_DMA on ppc64 or possibly ZONE_DMA32 on x86_64.  Second,
      the amount of memory usable by the kernel will be spread evenly throughout
      NUMA nodes where possible.  If the nodes are not of equal size, the amount of
      memory usable by the kernel on some nodes may be greater than others.
      
      By default, the zone is not as useful for hugetlb allocations because they are
      pinned and non-migratable (currently at least).  A sysctl is provided that
      allows huge pages to be allocated from that zone.  This means that the huge
      page pool can be resized to the size of ZONE_MOVABLE during the lifetime of
      the system assuming that pages are not mlocked.  Despite huge pages being
      non-movable, we do not introduce additional external fragmentation of note as
      huge pages are always the largest contiguous block we care about.
      
      Credit goes to Andy Whitcroft for catching a large variety of problems during
      review of the patches.
      
      This patch creates an additional zone, ZONE_MOVABLE.  This zone is only usable
      by allocations which specify both __GFP_HIGHMEM and __GFP_MOVABLE.  Hot-added
      memory continues to be placed in their existing destination as there is no
      mechanism to redirect them to a specific zone.
      
      [y-goto@jp.fujitsu.com: Fix section mismatch of memory hotplug related code]
      [akpm@linux-foundation.org: various fixes]
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Andy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarYasunori Goto <y-goto@jp.fujitsu.com>
      Cc: William Lee Irwin III <wli@holomorphy.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      2a1e274a
  13. 16 Jul, 2007 1 commit
    • KAMEZAWA Hiroyuki's avatar
      change zonelist order: zonelist order selection logic · f0c0b2b8
      KAMEZAWA Hiroyuki authored
      
      
      Make zonelist creation policy selectable from sysctl/boot option v6.
      
      This patch makes NUMA's zonelist (of pgdat) order selectable.
      Available order are Default(automatic)/ Node-based / Zone-based.
      
      [Default Order]
      The kernel selects Node-based or Zone-based order automatically.
      
      [Node-based Order]
      This policy treats the locality of memory as the most important parameter.
      Zonelist order is created by each zone's locality. This means lower zones
      (ex. ZONE_DMA) can be used before higher zone (ex. ZONE_NORMAL) exhausion.
      IOW. ZONE_DMA will be in the middle of zonelist.
      current 2.6.21 kernel uses this.
      
      Pros.
       * A user can expect local memory as much as possible.
      Cons.
       * lower zone will be exhansted before higher zone. This may cause OOM_KILL.
      
      Maybe suitable if ZONE_DMA is relatively big and you never see OOM_KILL
      because of ZONE_DMA exhaution and you need the best locality.
      
      (example)
      assume 2 node NUMA. node(0) has ZONE_DMA/ZONE_NORMAL, node(1) has ZONE_NORMAL.
      
      *node(0)'s memory allocation order:
      
       node(0)'s NORMAL -> node(0)'s DMA -> node(1)'s NORMAL.
      
      *node(1)'s memory allocation order:
      
       node(1)'s NORMAL -> node(0)'s NORMAL -> node(0)'s DMA.
      
      [Zone-based order]
      This policy treats the zone type as the most important parameter.
      Zonelist order is created by zone-type order. This means lower zone
      never be used bofere higher zone exhaustion.
      IOW. ZONE_DMA will be always at the tail of zonelist.
      
      Pros.
       * OOM_KILL(bacause of lower zone) occurs only if the whole zones are exhausted.
      Cons.
       * memory locality may not be best.
      
      (example)
      assume 2 node NUMA. node(0) has ZONE_DMA/ZONE_NORMAL, node(1) has ZONE_NORMAL.
      
      *node(0)'s memory allocation order:
      
       node(0)'s NORMAL -> node(1)'s NORMAL -> node(0)'s DMA.
      
      *node(1)'s memory allocation order:
      
       node(1)'s NORMAL -> node(0)'s NORMAL -> node(0)'s DMA.
      
      bootoption "numa_zonelist_order=" and proc/sysctl is supporetd.
      
      command:
      %echo N > /proc/sys/vm/numa_zonelist_order
      
      Will rebuild zonelist in Node-based order.
      
      command:
      %echo Z > /proc/sys/vm/numa_zonelist_order
      
      Will rebuild zonelist in Zone-based order.
      
      Thanks to Lee Schermerhorn, he gives me much help and codes.
      
      [Lee.Schermerhorn@hp.com: add check_highest_zone to build_zonelists_in_zone_order]
      [akpm@linux-foundation.org: build fix]
      Signed-off-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: Christoph Lameter <clameter@sgi.com>
      Cc: Andi Kleen <ak@suse.de>
      Cc: "jesse.barnes@intel.com" <jesse.barnes@intel.com>
      Signed-off-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      f0c0b2b8
  14. 09 May, 2007 1 commit
    • Christoph Lameter's avatar
      Move remote node draining out of slab allocators · 4037d452
      Christoph Lameter authored
      
      
      Currently the slab allocators contain callbacks into the page allocator to
      perform the draining of pagesets on remote nodes.  This requires SLUB to have
      a whole subsystem in order to be compatible with SLAB.  Moving node draining
      out of the slab allocators avoids a section of code in SLUB.
      
      Move the node draining so that is is done when the vm statistics are updated.
      At that point we are already touching all the cachelines with the pagesets of
      a processor.
      
      Add a expire counter there.  If we have to update per zone or global vm
      statistics then assume that the pageset will require subsequent draining.
      
      The expire counter will be decremented on each vm stats update pass until it
      reaches zero.  Then we will drain one batch from the pageset.  The draining
      will cause vm counter updates which will then cause another expiration until
      the pcp is empty.  So we will drain a batch every 3 seconds.
      
      Note that remote node draining is a somewhat esoteric feature that is required
      on large NUMA systems because otherwise significant portions of system memory
      can become trapped in pcp queues.  The number of pcp is determined by the
      number of processors and nodes in a system.  A system with 4 processors and 2
      nodes has 8 pcps which is okay.  But a system with 1024 processors and 512
      nodes has 512k pcps with a high potential for large amount of memory being
      caught in them.
      Signed-off-by: default avatarChristoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      4037d452
  15. 07 May, 2007 1 commit