1. 16 Oct, 2007 26 commits
    • Mel Gorman's avatar
      Breakout page_order() to internal.h to avoid special knowledge of the buddy allocator · 48f13bf3
      Mel Gorman authored
      
      
      The statistics patch later needs to know what order a free page is on the free
      lists.  Rather than having special knowledge of page_private() when
      PageBuddy() is set, this patch places out page_order() in internal.h and adds
      a VM_BUG_ON to catch using it on non-PageBuddy pages.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarChristoph Lameter <clameter@sgi.com>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      48f13bf3
    • Adrian Bunk's avatar
      mm/page_alloc.c: make code static · 484f51f8
      Adrian Bunk authored
      
      
      This patch makes needlessly global code static.
      Signed-off-by: default avatarAdrian Bunk <bunk@stusta.de>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      484f51f8
    • Mel Gorman's avatar
      Print out statistics in relation to fragmentation avoidance to /proc/pagetypeinfo · 467c996c
      Mel Gorman authored
      
      
      This patch provides fragmentation avoidance statistics via /proc/pagetypeinfo.
       The information is collected only on request so there is no runtime overhead.
       The statistics are in three parts:
      
      The first part prints information on the size of blocks that pages are
      being grouped on and looks like
      
      Page block order: 10
      Pages per block:  1024
      
      The second part is a more detailed version of /proc/buddyinfo and looks like
      
      Free pages count per migrate type at order       0      1      2      3      4      5      6      7      8      9     10
      Node    0, zone      DMA, type    Unmovable      0      0      0      0      0      0      0      0      0      0      0
      Node    0, zone      DMA, type  Reclaimable      1      0      0      0      0      0      0      0      0      0      0
      Node    0, zone      DMA, type      Movable      0      0      0      0      0      0      0      0      0      0      0
      Node    0, zone      DMA, type      Reserve      0      4      4      0      0      0      0      1      0      1      0
      Node    0, zone   Normal, type    Unmovable    111      8      4      4      2      3      1      0      0      0      0
      Node    0, zone   Normal, type  Reclaimable    293     89      8      0      0      0      0      0      0      0      0
      Node    0, zone   Normal, type      Movable      1      6     13      9      7      6      3      0      0      0      0
      Node    0, zone   Normal, type      Reserve      0      0      0      0      0      0      0      0      0      0      4
      
      The third part looks like
      
      Number of blocks type     Unmovable  Reclaimable      Movable      Reserve
      Node 0, zone      DMA            0            1            2            1
      Node 0, zone   Normal            3           17           94            4
      
      To walk the zones within a node with interrupts disabled, walk_zones_in_node()
      is introduced and shared between /proc/buddyinfo, /proc/zoneinfo and
      /proc/pagetypeinfo to reduce code duplication.  It seems specific to what
      vmstat.c requires but could be broken out as a general utility function in
      mmzone.c if there were other other potential users.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Acked-by: default avatarChristoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      467c996c
    • Mel Gorman's avatar
      Do not depend on MAX_ORDER when grouping pages by mobility · d9c23400
      Mel Gorman authored
      
      
      Currently mobility grouping works at the MAX_ORDER_NR_PAGES level.  This makes
      sense for the majority of users where this is also the huge page size.
      However, on platforms like ia64 where the huge page size is runtime
      configurable it is desirable to group at a lower order.  On x86_64 and
      occasionally on x86, the hugepage size may not always be MAX_ORDER_NR_PAGES.
      
      This patch groups pages together based on the value of HUGETLB_PAGE_ORDER.  It
      uses a compile-time constant if possible and a variable where the huge page
      size is runtime configurable.
      
      It is assumed that grouping should be done at the lowest sensible order and
      that the user would not want to override this.  If this is not true,
      page_block order could be forced to a variable initialised via a boot-time
      kernel parameter.
      
      One potential issue with this patch is that IA64 now parses hugepagesz with
      early_param() instead of __setup().  __setup() is called after the memory
      allocator has been initialised and the pageblock bitmaps already setup.  In
      tests on one IA64 there did not seem to be any problem with using
      early_param() and in fact may be more correct as it guarantees the parameter
      is handled before the parsing of hugepages=.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Acked-by: default avatarChristoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d9c23400
    • Mel Gorman's avatar
      Fix calculation in move_freepages_block for counting pages · d100313f
      Mel Gorman authored
      
      
      move_freepages_block() returns the number of blocks moved.  This value is used
      to determine if a block of pages should be stolen for the exclusive use of a
      migrate type or not.  However, the value returned is being used correctly.
      This patch fixes the calculation to return the number of base pages that have
      been moved.
      
      This should be considered a fix to the patch
      move-free-pages-between-lists-on-steal.patch
      
      Credit to Andy Whitcroft for spotting the problem.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Acked-by: default avatarChristoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d100313f
    • Mel Gorman's avatar
      don't group high order atomic allocations · 64c5e135
      Mel Gorman authored
      
      
      Grouping high-order atomic allocations together was intended to allow
      bursty users of atomic allocations to work such as e1000 in situations
      where their preallocated buffers were depleted.  This did not work in at
      least one case with a wireless network adapter needing order-1 allocations
      frequently.  To resolve that, the free pages used for min_free_kbytes were
      moved to separate contiguous blocks with the patch
      bias-the-location-of-pages-freed-for-min_free_kbytes-in-the-same-max_order_nr_pages-blocks.
      
      It is felt that keeping the free pages in the same contiguous blocks should
      be sufficient for bursty short-lived high-order atomic allocations to
      succeed, maybe even with the e1000.  Even if there is a failure, increasing
      the value of min_free_kbytes will free pages as contiguous bloks in
      contrast to the standard buddy allocator which makes no attempt to keep the
      minimum number of free pages contiguous.
      
      This patch backs out grouping high order atomic allocations together to
      determine if it is really needed or not.  If a new report comes in about
      high-order atomic allocations failing, the feature can be reintroduced to
      determine if it fixes the problem or not.  As a side-effect, this patch
      reduces by 1 the number of bits required to track the mobility type of
      pages within a MAX_ORDER_NR_PAGES block.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      64c5e135
    • Mel Gorman's avatar
      remove PAGE_GROUP_BY_MOBILITY · ac0e5b7a
      Mel Gorman authored
      
      
      Grouping pages by mobility can be disabled at compile-time. This was
      considered undesirable by a number of people. However, in the current stack of
      patches, it is not a simple case of just dropping the configurable patch as it
      would cause merge conflicts.  This patch backs out the configuration option.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      ac0e5b7a
    • Mel Gorman's avatar
      Bias the location of pages freed for min_free_kbytes in the same MAX_ORDER_NR_PAGES blocks · 56fd56b8
      Mel Gorman authored
      
      
      The standard buddy allocator always favours the smallest block of pages.
      The effect of this is that the pages free to satisfy min_free_kbytes tends
      to be preserved since boot time at the same location of memory ffor a very
      long time and as a contiguous block.  When an administrator sets the
      reserve at 16384 at boot time, it tends to be the same MAX_ORDER blocks
      that remain free.  This allows the occasional high atomic allocation to
      succeed up until the point the blocks are split.  In practice, it is
      difficult to split these blocks but when they do split, the benefit of
      having min_free_kbytes for contiguous blocks disappears.  Additionally,
      increasing min_free_kbytes once the system has been running for some time
      has no guarantee of creating contiguous blocks.
      
      On the other hand, CONFIG_PAGE_GROUP_BY_MOBILITY favours splitting large
      blocks when there are no free pages of the appropriate type available.  A
      side-effect of this is that all blocks in memory tends to be used up and
      the contiguous free blocks from boot time are not preserved like in the
      vanilla allocator.  This can cause a problem if a new caller is unwilling
      to reclaim or does not reclaim for long enough.
      
      A failure scenario was found for a wireless network device allocating
      order-1 atomic allocations but the allocations were not intense or frequent
      enough for a whole block of pages to be preserved for MIGRATE_HIGHALLOC.
      This was reproduced on a desktop by booting with mem=256mb, forcing the
      driver to allocate at order-1, running a bittorrent client (downloading a
      debian ISO) and building a kernel with -j2.
      
      This patch addresses the problem on the desktop machine booted with
      mem=256mb.  It works by setting aside a reserve of MAX_ORDER_NR_PAGES
      blocks, the number of which depends on the value of min_free_kbytes.  These
      blocks are only fallen back to when there is no other free pages.  Then the
      smallest possible page is used just like the normal buddy allocator instead
      of the largest possible page to preserve contiguous pages The pages in free
      lists in the reserve blocks are never taken for another migrate type.  The
      results is that even if min_free_kbytes is set to a low value, contiguous
      blocks will be preserved in the MIGRATE_RESERVE blocks.
      
      This works better than the vanilla allocator because if min_free_kbytes is
      increased, a new reserve block will be chosen based on the location of
      reclaimable pages and the block will free up as contiguous pages.  In the
      vanilla allocator, no effort is made to target a block of pages to free as
      contiguous pages and min_free_kbytes pages are scattered randomly.
      
      This effect has been observed on the test machine.  min_free_kbytes was set
      initially low but it was kept as a contiguous free block within
      MIGRATE_RESERVE.  min_free_kbytes was then set to a higher value and over a
      period of time, the free blocks were within the reserve and coalescing.
      How long it takes to free up depends on how quickly LRU is rotating.
      Amusingly, this means that more activity will free the blocks faster.
      
      This mechanism potentially replaces MIGRATE_HIGHALLOC as it may be more
      effective than grouping contiguous free pages together.  It all depends on
      whether the number of active atomic high allocations exceeds
      min_free_kbytes or not.  If the number of active allocations exceeds
      min_free_kbytes, it's worth it but maybe in that situation, min_free_kbytes
      should be set higher.  Once there are no more reports of allocation
      failures, a patch will be submitted that backs out MIGRATE_HIGHALLOC and
      see if the reports stay missing.
      
      Credit to Mariusz Kozlowski for discovering the problem, describing the
      failure scenario and testing patches and scenarios.
      
      [akpm@linux-foundation.org: cleanups]
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      56fd56b8
    • Mel Gorman's avatar
      Be more agressive about stealing when MIGRATE_RECLAIMABLE allocations fallback · 46dafbca
      Mel Gorman authored
      
      
      MIGRATE_RECLAIMABLE allocations tend to be very bursty in nature like when
      updatedb starts.  It is likely this will occur in situations where MAX_ORDER
      blocks of pages are not free.  This means that updatedb can scatter
      MIGRATE_RECLAIMABLE pages throughout the address space.  This patch is more
      agressive about stealing blocks of pages for MIGRATE_RECLAIMABLE.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      46dafbca
    • Mel Gorman's avatar
      Bias the placement of kernel pages at lower PFNs · 5adc5be7
      Mel Gorman authored
      
      
      This patch chooses blocks with lower PFNs when placing kernel allocations.
      This is particularly important during fallback in low memory situations to
      stop unmovable pages being placed throughout the entire address space.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      5adc5be7
    • Mel Gorman's avatar
      Do not group pages by mobility type on low memory systems · 9ef9acb0
      Mel Gorman authored
      
      
      Grouping pages by mobility can only successfully operate when there are more
      MAX_ORDER_NR_PAGES areas than mobility types.  When there are insufficient
      areas, fallbacks cannot be avoided.  This has noticeable performance impacts
      on machines with small amounts of memory in comparison to MAX_ORDER_NR_PAGES.
      For example, on IA64 with a configuration including huge pages spans 1GiB with
      MAX_ORDER_NR_PAGES so would need at least 4GiB of RAM before grouping pages by
      mobility would be useful.  In comparison, an x86 would need 16MB.
      
      This patch checks the size of vm_total_pages in build_all_zonelists(). If
      there are not enough areas,  mobility is effectivly disabled by considering
      all allocations as the same type (UNMOVABLE).  This is achived via a
      __read_mostly flag.
      
      With this patch, performance is comparable to disabling grouping pages
      by mobility at compile-time on a test machine with insufficient memory.
      With this patch, it is reasonable to get rid of grouping pages by mobility
      a compile-time option.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      9ef9acb0
    • Mel Gorman's avatar
      Group high-order atomic allocations · e010487d
      Mel Gorman authored
      
      
      In rare cases, the kernel needs to allocate a high-order block of pages
      without sleeping.  For example, this is the case with e1000 cards configured
      to use jumbo frames.  Migrating or reclaiming pages in this situation is not
      an option.
      
      This patch groups these allocations together as much as possible by adding a
      new MIGRATE_TYPE.  The MIGRATE_HIGHATOMIC type are exactly what they sound
      like.  Care is taken that pages of other migrate types do not use the same
      blocks as high-order atomic allocations.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e010487d
    • Mel Gorman's avatar
      Group short-lived and reclaimable kernel allocations · e12ba74d
      Mel Gorman authored
      
      
      This patch marks a number of allocations that are either short-lived such as
      network buffers or are reclaimable such as inode allocations.  When something
      like updatedb is called, long-lived and unmovable kernel allocations tend to
      be spread throughout the address space which increases fragmentation.
      
      This patch groups these allocations together as much as possible by adding a
      new MIGRATE_TYPE.  The MIGRATE_RECLAIMABLE type is for allocations that can be
      reclaimed on demand, but not moved.  i.e.  they can be migrated by deleting
      them and re-reading the information from elsewhere.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Andy Whitcroft <apw@shadowen.org>
      Cc: Christoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e12ba74d
    • Mel Gorman's avatar
      Move free pages between lists on steal · c361be55
      Mel Gorman authored
      
      
      When a fallback occurs, there will be free pages for one allocation type
      stored on the list for another.  When a large steal occurs, this patch will
      move all the free pages within one list to the other.
      
      [y-goto@jp.fujitsu.com: fix BUG_ON check at move_freepages()]
      [apw@shadowen.org: Move to using pfn_valid_within()]
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Christoph Lameter <clameter@engr.sgi.com>
      Signed-off-by: default avatarYasunori Goto <y-goto@jp.fujitsu.com>
      Cc: Bjorn Helgaas <bjorn.helgaas@hp.com>
      Signed-off-by: default avatarAndy Whitcroft <andyw@uk.ibm.com>
      Cc: Bob Picco <bob.picco@hp.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      c361be55
    • Mel Gorman's avatar
      Drain per-cpu lists when high-order allocations fail · e2c55dc8
      Mel Gorman authored
      
      
      Per-cpu pages can accidentally cause fragmentation because they are free, but
      pinned pages in an otherwise contiguous block.  When this patch is applied,
      the per-cpu caches are drained after the direct-reclaim is entered if the
      requested order is greater than 0.  It simply reuses the code used by suspend
      and hotplug.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e2c55dc8
    • Mel Gorman's avatar
      Add a configure option to group pages by mobility · b92a6edd
      Mel Gorman authored
      
      
      The grouping mechanism has some memory overhead and a more complex allocation
      path.  This patch allows the strategy to be disabled for small memory systems
      or if it is known the workload is suffering because of the strategy.  It also
      acts to show where the page groupings strategy interacts with the standard
      buddy allocator.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarJoel Schopp <jschopp@austin.ibm.com>
      Cc: Andy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      b92a6edd
    • Mel Gorman's avatar
      Choose pages from the per-cpu list based on migration type · 535131e6
      Mel Gorman authored
      
      
      The freelists for each migrate type can slowly become polluted due to the
      per-cpu list.  Consider what happens when the following happens
      
      1. A 2^(MAX_ORDER-1) list is reserved for __GFP_MOVABLE pages
      2. An order-0 page is allocated from the newly reserved block
      3. The page is freed and placed on the per-cpu list
      4. alloc_page() is called with GFP_KERNEL as the gfp_mask
      5. The per-cpu list is used to satisfy the allocation
      
      This results in a kernel page is in the middle of a migratable region. This
      patch prevents this leak occuring by storing the MIGRATE_ type of the page in
      page->private. On allocate, a page will only be returned of the desired type,
      else more pages will be allocated. This may temporarily allow a per-cpu list
      to go over the pcp->high limit but it'll be corrected on the next free. Care
      is taken to preserve the hotness of pages recently freed.
      
      The additional code is not measurably slower for the workloads we've tested.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      535131e6
    • Mel Gorman's avatar
      Split the free lists for movable and unmovable allocations · b2a0ac88
      Mel Gorman authored
      
      
      This patch adds the core of the fragmentation reduction strategy.  It works by
      grouping pages together based on their ability to migrate or be reclaimed.
      Basically, it works by breaking the list in zone->free_area list into
      MIGRATE_TYPES number of lists.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      b2a0ac88
    • Mel Gorman's avatar
      Add a bitmap that is used to track flags affecting a block of pages · 835c134e
      Mel Gorman authored
      
      
      Here is the latest revision of the anti-fragmentation patches.  Of particular
      note in this version is special treatment of high-order atomic allocations.
      Care is taken to group them together and avoid grouping pages of other types
      near them.  Artifical tests imply that it works.  I'm trying to get the
      hardware together that would allow setting up of a "real" test.  If anyone
      already has a setup and test that can trigger the atomic-allocation problem,
      I'd appreciate a test of these patches and a report.  The second major change
      is that these patches will apply cleanly with patches that implement
      anti-fragmentation through zones.
      
      kernbench shows effectively no performance difference varying between -0.2%
      and +2% on a variety of test machines.  Success rates for huge page allocation
      are dramatically increased.  For example, on a ppc64 machine, the vanilla
      kernel was only able to allocate 1% of memory as a hugepage and this was due
      to a single hugepage reserved as min_free_kbytes.  With these patches applied,
      17% was allocatable as superpages.  With reclaim-related fixes from Andy
      Whitcroft, it was 40% and further reclaim-related improvements should increase
      this further.
      
      Changelog Since V28
      o Group high-order atomic allocations together
      o It is no longer required to set min_free_kbytes to 10% of memory. A value
        of 16384 in most cases will be sufficient
      o Now applied with zone-based anti-fragmentation
      o Fix incorrect VM_BUG_ON within buffered_rmqueue()
      o Reorder the stack so later patches do not back out work from earlier patches
      o Fix bug were journal pages were being treated as movable
      o Bias placement of non-movable pages to lower PFNs
      o More agressive clustering of reclaimable pages in reactions to workloads
        like updatedb that flood the size of inode caches
      
      Changelog Since V27
      
      o Renamed anti-fragmentation to Page Clustering. Anti-fragmentation was giving
        the mistaken impression that it was the 100% solution for high order
        allocations. Instead, it greatly increases the chances high-order
        allocations will succeed and lays the foundation for defragmentation and
        memory hot-remove to work properly
      o Redefine page groupings based on ability to migrate or reclaim instead of
        basing on reclaimability alone
      o Get rid of spurious inits
      o Per-cpu lists are no longer split up per-type. Instead the per-cpu list is
        searched for a page of the appropriate type
      o Added more explanation commentary
      o Fix up bug in pageblock code where bitmap was used before being initalised
      
      Changelog Since V26
      o Fix double init of lists in setup_pageset
      
      Changelog Since V25
      o Fix loop order of for_each_rclmtype_order so that order of loop matches args
      o gfpflags_to_rclmtype uses gfp_t instead of unsigned long
      o Rename get_pageblock_type() to get_page_rclmtype()
      o Fix alignment problem in move_freepages()
      o Add mechanism for assigning flags to blocks of pages instead of page->flags
      o On fallback, do not examine the preferred list of free pages a second time
      
      The purpose of these patches is to reduce external fragmentation by grouping
      pages of related types together.  When pages are migrated (or reclaimed under
      memory pressure), large contiguous pages will be freed.
      
      This patch works by categorising allocations by their ability to migrate;
      
      Movable - The pages may be moved with the page migration mechanism. These are
      	generally userspace pages.
      
      Reclaimable - These are allocations for some kernel caches that are
      	reclaimable or allocations that are known to be very short-lived.
      
      Unmovable - These are pages that are allocated by the kernel that
      	are not trivially reclaimed. For example, the memory allocated for a
      	loaded module would be in this category. By default, allocations are
      	considered to be of this type
      
      HighAtomic - These are high-order allocations belonging to callers that
      	cannot sleep or perform any IO. In practice, this is restricted to
      	jumbo frame allocation for network receive. It is assumed that the
      	allocations are short-lived
      
      Instead of having one MAX_ORDER-sized array of free lists in struct free_area,
      there is one for each type of reclaimability.  Once a 2^MAX_ORDER block of
      pages is split for a type of allocation, it is added to the free-lists for
      that type, in effect reserving it.  Hence, over time, pages of the different
      types can be clustered together.
      
      When the preferred freelists are expired, the largest possible block is taken
      from an alternative list.  Buddies that are split from that large block are
      placed on the preferred allocation-type freelists to mitigate fragmentation.
      
      This implementation gives best-effort for low fragmentation in all zones.
      Ideally, min_free_kbytes needs to be set to a value equal to 4 * (1 <<
      (MAX_ORDER-1)) pages in most cases.  This would be 16384 on x86 and x86_64 for
      example.
      
      Our tests show that about 60-70% of physical memory can be allocated on a
      desktop after a few days uptime.  In benchmarks and stress tests, we are
      finding that 80% of memory is available as contiguous blocks at the end of the
      test.  To compare, a standard kernel was getting < 1% of memory as large pages
      on a desktop and about 8-12% of memory as large pages at the end of stress
      tests.
      
      Following this email are 12 patches that implement thie page grouping feature.
       The first patch introduces a mechanism for storing flags related to a whole
      block of pages.  Then allocations are split between movable and all other
      allocations.  Following that are patches to deal with per-cpu pages and make
      the mechanism configurable.  The next patch moves free pages between lists
      when partially allocated blocks are used for pages of another migrate type.
      The second last patch groups reclaimable kernel allocations such as inode
      caches together.  The final patch related to groupings keeps high-order atomic
      allocations.
      
      The last two patches are more concerned with control of fragmentation.  The
      second last patch biases placement of non-movable allocations towards the
      start of memory.  This is with a view of supporting memory hot-remove of DIMMs
      with higher PFNs in the future.  The biasing could be enforced a lot heavier
      but it would cost.  The last patch agressively clusters reclaimable pages like
      inode caches together.
      
      The fragmentation reduction strategy needs to track if pages within a block
      can be moved or reclaimed so that pages are freed to the appropriate list.
      This patch adds a bitmap for flags affecting a whole a MAX_ORDER block of
      pages.
      
      In non-SPARSEMEM configurations, the bitmap is stored in the struct zone and
      allocated during initialisation.  SPARSEMEM statically allocates the bitmap in
      a struct mem_section so that bitmaps do not have to be resized during memory
      hotadd.  This wastes a small amount of memory per unused section (usually
      sizeof(unsigned long)) but the complexity of dynamically allocating the memory
      is quite high.
      
      Additional credit to Andy Whitcroft who reviewed up an earlier implementation
      of the mechanism an suggested how to make it a *lot* cleaner.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Andy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      835c134e
    • Lee Schermerhorn's avatar
      memoryless nodes: fixup uses of node_online_map in generic code · 37b07e41
      Lee Schermerhorn authored
      
      
      Here's a cut at fixing up uses of the online node map in generic code.
      
      mm/shmem.c:shmem_parse_mpol()
      
      	Ensure nodelist is subset of nodes with memory.
      	Use node_states[N_HIGH_MEMORY] as default for missing
      	nodelist for interleave policy.
      
      mm/shmem.c:shmem_fill_super()
      
      	initialize policy_nodes to node_states[N_HIGH_MEMORY]
      
      mm/page-writeback.c:highmem_dirtyable_memory()
      
      	sum over nodes with memory
      
      mm/page_alloc.c:zlc_setup()
      
      	allowednodes - use nodes with memory.
      
      mm/page_alloc.c:default_zonelist_order()
      
      	average over nodes with memory.
      
      mm/page_alloc.c:find_next_best_node()
      
      	skip nodes w/o memory.
      	N_HIGH_MEMORY state mask may not be initialized at this time,
      	unless we want to depend on early_calculate_totalpages() [see
      	below].  Will ZONE_MOVABLE ever be configurable?
      
      mm/page_alloc.c:find_zone_movable_pfns_for_nodes()
      
      	spread kernelcore over nodes with memory.
      
      	This required calling early_calculate_totalpages()
      	unconditionally, and populating N_HIGH_MEMORY node
      	state therein from nodes in the early_node_map[].
      	If we can depend on this, we can eliminate the
      	population of N_HIGH_MEMORY mask from __build_all_zonelists()
      	and use the N_HIGH_MEMORY mask in find_next_best_node().
      
      mm/mempolicy.c:mpol_check_policy()
      
      	Ensure nodes specified for policy are subset of
      	nodes with memory.
      
      [akpm@linux-foundation.org: fix warnings]
      Signed-off-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Acked-by: default avatarChristoph Lameter <clameter@sgi.com>
      Cc: Shaohua Li <shaohua.li@intel.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      37b07e41
    • Christoph Lameter's avatar
      Memoryless nodes: Fix GFP_THISNODE behavior · 523b9458
      Christoph Lameter authored
      
      
      GFP_THISNODE checks that the zone selected is within the pgdat (node) of the
      first zone of a nodelist.  That only works if the node has memory.  A
      memoryless node will have its first node on another pgdat (node).
      
      GFP_THISNODE currently will return simply memory on the first pgdat.  Thus it
      is returning memory on other nodes.  GFP_THISNODE should fail if there is no
      local memory on a node.
      
      Add a new set of zonelists for each node that only contain the nodes that
      belong to the zones itself so that no fallback is possible.
      
      Then modify gfp_type to pickup the right zone based on the presence of
      __GFP_THISNODE.
      
      Drop the existing GFP_THISNODE checks from the page_allocators hot path.
      Signed-off-by: default avatarChristoph Lameter <clameter@sgi.com>
      Acked-by: default avatarNishanth Aravamudan <nacc@us.ibm.com>
      Tested-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Acked-by: default avatarBob Picco <bob.picco@hp.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Mel Gorman <mel@skynet.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      523b9458
    • Christoph Lameter's avatar
      Memoryless nodes: drop one memoryless node boot warning · 633c0666
      Christoph Lameter authored
      
      
      get_pfn_range_for_nid() is called multiple times for each node at boot time.
      Each time, it will warn about nodes with no memory, resulting in boot messages
      like:
      
              Node 0 active with no memory
              Node 0 active with no memory
              Node 0 active with no memory
              Node 0 active with no memory
              Node 0 active with no memory
              Node 0 active with no memory
              On node 0 totalpages: 0
              Node 0 active with no memory
              Node 0 active with no memory
                DMA zone: 0 pages used for memmap
              Node 0 active with no memory
              Node 0 active with no memory
                Normal zone: 0 pages used for memmap
              Node 0 active with no memory
              Node 0 active with no memory
                Movable zone: 0 pages used for memmap
      
      and so on for each memoryless node.
      
      We already have the "On node N totalpages: ..." and other related messages, so
      drop the "Node N active with no memory" warnings.
      Signed-off-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: Bob Picco <bob.picco@hp.com>
      Cc: Nishanth Aravamudan <nacc@us.ibm.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Mel Gorman <mel@skynet.ie>
      Signed-off-by: default avatarChristoph Lameter <clameter@sgi.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      633c0666
    • Christoph Lameter's avatar
      Memoryless nodes: Add N_CPU node state · 37c0708d
      Christoph Lameter authored
      
      
      We need the check for a node with cpu in zone reclaim.  Zone reclaim will not
      allow remote zone reclaim if a node has a cpu.
      
      [Lee.Schermerhorn@hp.com: Move setup of N_CPU node state mask]
      Signed-off-by: default avatarChristoph Lameter <clameter@sgi.com>
      Tested-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Acked-by: default avatarBob Picco <bob.picco@hp.com>
      Cc: Nishanth Aravamudan <nacc@us.ibm.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Mel Gorman <mel@skynet.ie>
      Signed-off-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      37c0708d
    • Christoph Lameter's avatar
      Memoryless nodes: introduce mask of nodes with memory · 7ea1530a
      Christoph Lameter authored
      
      
      It is necessary to know if nodes have memory since we have recently begun to
      add support for memoryless nodes.  For that purpose we introduce a two new
      node states: N_HIGH_MEMORY and N_NORMAL_MEMORY.
      
      A node has its bit in N_HIGH_MEMORY set if it has any memory regardless of the
      type of mmemory.  If a node has memory then it has at least one zone defined
      in its pgdat structure that is located in the pgdat itself.
      
      A node has its bit in N_NORMAL_MEMORY set if it has a lower zone than
      ZONE_HIGHMEM.  This means it is possible to allocate memory that is not
      subject to kmap.
      
      N_HIGH_MEMORY and N_NORMAL_MEMORY can then be used in various places to insure
      that we do the right thing when we encounter a memoryless node.
      
      [akpm@linux-foundation.org: build fix]
      [Lee.Schermerhorn@hp.com: update N_HIGH_MEMORY node state for memory hotadd]
      [y-goto@jp.fujitsu.com: Fix memory hotplug + sparsemem build]
      Signed-off-by: default avatarLee Schermerhorn <Lee.Schermerhorn@hp.com>
      Signed-off-by: default avatarNishanth Aravamudan <nacc@us.ibm.com>
      Signed-off-by: default avatarChristoph Lameter <clameter@sgi.com>
      Acked-by: default avatarBob Picco <bob.picco@hp.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Mel Gorman <mel@skynet.ie>
      Signed-off-by: default avatarYasunori Goto <y-goto@jp.fujitsu.com>
      Signed-off-by: default avatarPaul Mundt <lethal@linux-sh.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      7ea1530a
    • Christoph Lameter's avatar
      Memoryless nodes: Generic management of nodemasks for various purposes · 13808910
      Christoph Lameter authored
      
      
      Why do we need to support memoryless nodes?
      
      KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
      
      > For fujitsu, problem is called "empty" node.
      >
      > When ACPI's SRAT table includes "possible nodes", ia64 bootstrap(acpi_numa_init)
      > creates nodes, which includes no memory, no cpu.
      >
      > I tried to remove empty-node in past, but that was denied.
      > It was because we can hot-add cpu to the empty node.
      > (node-hotplug triggered by cpu is not implemented now. and it will be ugly.)
      >
      >
      > For HP, (Lee can comment on this later), they have memory-less-node.
      > As far as I hear, HP's machine can have following configration.
      >
      > (example)
      > Node0: CPU0   memory AAA MB
      > Node1: CPU1   memory AAA MB
      > Node2: CPU2   memory AAA MB
      > Node3: CPU3   memory AAA MB
      > Node4: Memory XXX GB
      >
      > AAA is very small value (below 16MB)  and will be omitted by ia64 bootstrap.
      > After boot, only Node 4 has valid memory (but have no cpu.)
      >
      > Maybe this is memory-interleave by firmware config.
      
      Christoph Lameter <clameter@sgi.com> wrote:
      
      > Future SGI platforms (actually also current one can have but nothing like
      > that is deployed to my knowledge) have nodes with only cpus. Current SGI
      > platforms have nodes with just I/O that we so far cannot manage in the
      > core. So the arch code maps them to the nearest memory node.
      
      Lee Schermerhorn <Lee.Schermerhorn@hp.com> wrote:
      
      > For the HP platforms, we can configure each cell with from 0% to 100%
      > "cell local memory".  When we configure with <100% CLM, the "missing
      > percentages" are interleaved by hardware on a cache-line granularity to
      > improve bandwidth at the expense of latency for numa-challenged
      > applications [and OSes, but not our problem ;-)].  When we boot Linux on
      > such a config, all of the real nodes have no memory--it all resides in a
      > single interleaved pseudo-node.
      >
      > When we boot Linux on a 100% CLM configuration [== NUMA], we still have
      > the interleaved pseudo-node.  It contains a few hundred MB stolen from
      > the real nodes to contain the DMA zone.  [Interleaved memory resides at
      > phys addr 0].  The memoryless-nodes patches, along with the zoneorder
      > patches, support this config as well.
      >
      > Also, when we boot a NUMA config with the "mem=" command line,
      > specifying less memory than actually exists, Linux takes the excluded
      > memory "off the top" rather than distributing it across the nodes.  This
      > can result in memoryless nodes, as well.
      >
      
      This patch:
      
      Preparation for memoryless node patches.
      
      Provide a generic way to keep nodemasks describing various characteristics of
      NUMA nodes.
      
      Remove the node_online_map and the node_possible map and realize the same
      functionality using two nodes stats: N_POSSIBLE and N_ONLINE.
      
      [Lee.Schermerhorn@hp.com: Initialize N_*_MEMORY and N_CPU masks for non-NUMA config]
      Signed-off-by: default avatarChristoph Lameter <clameter@sgi.com>
      Tested-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Acked-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Acked-by: default avatarBob Picco <bob.picco@hp.com>
      Cc: Nishanth Aravamudan <nacc@us.ibm.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Mel Gorman <mel@skynet.ie>
      Signed-off-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: "Serge E. Hallyn" <serge@hallyn.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      13808910
    • Jesper Juhl's avatar
      mm: no need to cast vmalloc() return value in zone_wait_table_init() · 8691f3a7
      Jesper Juhl authored
      
      
      vmalloc() returns a void pointer, so there's no need to cast its
      return value in mm/page_alloc.c::zone_wait_table_init().
      Signed-off-by: default avatarJesper Juhl <jesper.juhl@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      8691f3a7
  2. 31 Aug, 2007 1 commit
  3. 22 Aug, 2007 1 commit
    • Mel Gorman's avatar
      Apply memory policies to top two highest zones when highest zone is ZONE_MOVABLE · b377fd39
      Mel Gorman authored
      
      
      The NUMA layer only supports NUMA policies for the highest zone.  When
      ZONE_MOVABLE is configured with kernelcore=, the the highest zone becomes
      ZONE_MOVABLE.  The result is that policies are only applied to allocations
      like anonymous pages and page cache allocated from ZONE_MOVABLE when the
      zone is used.
      
      This patch applies policies to the two highest zones when the highest zone
      is ZONE_MOVABLE.  As ZONE_MOVABLE consists of pages from the highest "real"
      zone, it's always functionally equivalent.
      
      The patch has been tested on a variety of machines both NUMA and non-NUMA
      covering x86, x86_64 and ppc64.  No abnormal results were seen in
      kernbench, tbench, dbench or hackbench.  It passes regression tests from
      the numactl package with and without kernelcore= once numactl tests are
      patched to wait for vmstat counters to update.
      
      akpm: this is the nasty hack to fix NUMA mempolicies in the presence of
      ZONE_MOVABLE and kernelcore= in 2.6.23.  Christoph says "For .24 either merge
      the mobility or get the other solution that Mel is working on.  That solution
      would only use a single zonelist per node and filter on the fly.  That may
      help performance and also help to make memory policies work better."
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Tested-by: default avatarLee Schermerhorn <lee.schermerhorn@hp.com>
      Acked-by: default avatarChristoph Lameter <clameter@sgi.com>
      Cc: Andi Kleen <ak@suse.de>
      Cc: Paul Mundt <lethal@linux-sh.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      b377fd39
  4. 31 Jul, 2007 1 commit
    • Mel Gorman's avatar
      Do not trigger OOM-killer for high-order allocation failures · a8bbf72a
      Mel Gorman authored
      
      
      out_of_memory() may be called when an allocation is failing and the direct
      reclaim is not making any progress.  This does not take into account the
      requested order of the allocation.  If the request if for an order larger
      than PAGE_ALLOC_COSTLY_ORDER, it is reasonable to fail the allocation
      because the kernel makes no guarantees about those allocations succeeding.
      
      This false OOM situation can occur if a user is trying to grow the hugepage
      pool in a script like;
      
      #!/bin/bash
      REQUIRED=$1
      echo 1 > /proc/sys/vm/hugepages_treat_as_movable
      echo $REQUIRED > /proc/sys/vm/nr_hugepages
      ACTUAL=`cat /proc/sys/vm/nr_hugepages`
      while [ $REQUIRED -ne $ACTUAL ]; do
      	echo Huge page pool at $ACTUAL growing to $REQUIRED
      	echo $REQUIRED > /proc/sys/vm/nr_hugepages
      	ACTUAL=`cat /proc/sys/vm/nr_hugepages`
      	sleep 1
      done
      
      This is a reasonable scenario when ZONE_MOVABLE is in use but triggers OOM
      easily on 2.6.23-rc1. This patch will fail an allocation for an order above
      PAGE_ALLOC_COSTLY_ORDER instead of killing processes and retrying.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Cc: David Rientjes <rientjes@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      a8bbf72a
  5. 29 Jul, 2007 1 commit
    • Rafael J. Wysocki's avatar
      Introduce CONFIG_SUSPEND for suspend-to-Ram and standby · 296699de
      Rafael J. Wysocki authored
      
      
      Introduce CONFIG_SUSPEND representing the ability to enter system sleep
      states, such as the ACPI S3 state, and allow the user to choose SUSPEND
      and HIBERNATION independently of each other.
      
      Make HOTPLUG_CPU be selected automatically if SUSPEND or HIBERNATION has
      been chosen and the kernel is intended for SMP systems.
      
      Also, introduce CONFIG_PM_SLEEP which is automatically selected if
      CONFIG_SUSPEND or CONFIG_HIBERNATION is set and use it to select the
      code needed for both suspend and hibernation.
      
      The top-level power management headers and the ACPI code related to
      suspend and hibernation are modified to use the new definitions (the
      changes in drivers/acpi/sleep/main.c are, mostly, moving code to reduce
      the number of ifdefs).
      
      There are many other files in which CONFIG_PM can be replaced with
      CONFIG_PM_SLEEP or even with CONFIG_SUSPEND, but they can be updated in
      the future.
      Signed-off-by: default avatarRafael J. Wysocki <rjw@sisk.pl>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      296699de
  6. 26 Jul, 2007 1 commit
    • Mel Gorman's avatar
      Allow nodes to exist that only contain ZONE_MOVABLE · b5445f95
      Mel Gorman authored
      
      
      With the introduction of kernelcore=, a configurable zone is created on
      request.  In some cases, this value will be small enough that some nodes
      contain only ZONE_MOVABLE.  On some NUMA configurations when this occurs,
      arch-independent zone-sizing will get the size of the memory holes within
      the node incorrect.  The value of present_pages goes negative and the boot
      fails.
      
      This patch fixes the bug in the calculation of the size of the hole.  The
      test case is to boot test a NUMA machine with a low value of kernelcore=
      before and after the patch is applied.  While this bug exists in early
      kernel it cannot be triggered in practice.
      
      This patch has been boot-tested on a variety machines with and without
      kernelcore= set.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      b5445f95
  7. 20 Jul, 2007 1 commit
  8. 19 Jul, 2007 2 commits
    • Fengguang Wu's avatar
      mm: share PG_readahead and PG_reclaim · fe3cba17
      Fengguang Wu authored
      
      
      Share the same page flag bit for PG_readahead and PG_reclaim.
      
      One is used only on file reads, another is only for emergency writes.  One
      is used mostly for fresh/young pages, another is for old pages.
      
      Combinations of possible interactions are:
      
      a) clear PG_reclaim => implicit clear of PG_readahead
      	it will delay an asynchronous readahead into a synchronous one
      	it actually does _good_ for readahead:
      		the pages will be reclaimed soon, it's readahead thrashing!
      		in this case, synchronous readahead makes more sense.
      
      b) clear PG_readahead => implicit clear of PG_reclaim
      	one(and only one) page will not be reclaimed in time
      	it can be avoided by checking PageWriteback(page) in readahead first
      
      c) set PG_reclaim => implicit set of PG_readahead
      	will confuse readahead and make it restart the size rampup process
      	it's a trivial problem, and can mostly be avoided by checking
      	PageWriteback(page) first in readahead
      
      d) set PG_readahead => implicit set of PG_reclaim
      	PG_readahead will never be set on already cached pages.
      	PG_reclaim will always be cleared on dirtying a page.
      	so not a problem.
      
      In summary,
      	a)   we get better behavior
      	b,d) possible interactions can be avoided
      	c)   racy condition exists that might affect readahead, but the chance
      	     is _really_ low, and the hurt on readahead is trivial.
      
      Compound pages also use PG_reclaim, but for now they do not interact with
      reclaim/readahead code.
      Signed-off-by: default avatarFengguang Wu <wfg@mail.ustc.edu.cn>
      Cc: Rusty Russell <rusty@rustcorp.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      fe3cba17
    • Fengguang Wu's avatar
      readahead: introduce PG_readahead · d77c2d7c
      Fengguang Wu authored
      
      
      Introduce a new page flag: PG_readahead.
      
      It acts as a look-ahead mark, which tells the page reader: Hey, it's time to
      invoke the read-ahead logic.  For the sake of I/O pipelining, don't wait until
      it runs out of cached pages!
      Signed-off-by: default avatarFengguang Wu <wfg@mail.ustc.edu.cn>
      Cc: Steven Pratt <slpratt@austin.ibm.com>
      Cc: Ram Pai <linuxram@us.ibm.com>
      Cc: Rusty Russell <rusty@rustcorp.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d77c2d7c
  9. 17 Jul, 2007 5 commits
    • Meelap Shah's avatar
      knfsd: nfsd4: vary maximum delegation limit based on RAM size · c2f1a551
      Meelap Shah authored
      
      
      Our original NFSv4 delegation policy was to give out a read delegation on any
      open when it was possible to.
      
      Since the lifetime of a delegation isn't limited to that of an open, a client
      may quite reasonably hang on to a delegation as long as it has the inode
      cached.  This becomes an obvious problem the first time a client's inode cache
      approaches the size of the server's total memory.
      
      Our first quick solution was to add a hard-coded limit.  This patch makes a
      mild incremental improvement by varying that limit according to the server's
      total memory size, allowing at most 4 delegations per megabyte of RAM.
      
      My quick back-of-the-envelope calculation finds that in the worst case (where
      every delegation is for a different inode), a delegation could take about
      1.5K, which would make the worst case usage about 6% of memory.  The new limit
      works out to be about the same as the old on a 1-gig server.
      
      [akpm@linux-foundation.org: Don't needlessly bloat vmlinux]
      [akpm@linux-foundation.org: Make it right for highmem machines]
      Signed-off-by: default avatar"J. Bruce Fields" <bfields@citi.umich.edu>
      Signed-off-by: default avatarNeil Brown <neilb@suse.de>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      c2f1a551
    • Andy Whitcroft's avatar
      Lumpy Reclaim V4 · 5ad333eb
      Andy Whitcroft authored
      
      
      When we are out of memory of a suitable size we enter reclaim.  The current
      reclaim algorithm targets pages in LRU order, which is great for fairness at
      order-0 but highly unsuitable if you desire pages at higher orders.  To get
      pages of higher order we must shoot down a very high proportion of memory;
      >95% in a lot of cases.
      
      This patch set adds a lumpy reclaim algorithm to the allocator.  It targets
      groups of pages at the specified order anchored at the end of the active and
      inactive lists.  This encourages groups of pages at the requested orders to
      move from active to inactive, and active to free lists.  This behaviour is
      only triggered out of direct reclaim when higher order pages have been
      requested.
      
      This patch set is particularly effective when utilised with an
      anti-fragmentation scheme which groups pages of similar reclaimability
      together.
      
      This patch set is based on Peter Zijlstra's lumpy reclaim V2 patch which forms
      the foundation.  Credit to Mel Gorman for sanitity checking.
      
      Mel said:
      
        The patches have an application with hugepage pool resizing.
      
        When lumpy-reclaim is used used with ZONE_MOVABLE, the hugepages pool can
        be resized with greater reliability.  Testing on a desktop machine with 2GB
        of RAM showed that growing the hugepage pool with ZONE_MOVABLE on it's own
        was very slow as the success rate was quite low.  Without lumpy-reclaim,
        each attempt to grow the pool by 100 pages would yield 1 or 2 hugepages.
        With lumpy-reclaim, getting 40 to 70 hugepages on each attempt was typical.
      
      [akpm@osdl.org: ia64 pfn_to_nid fixes and loop cleanup]
      [bunk@stusta.de: static declarations for internal functions]
      [a.p.zijlstra@chello.nl: initial lumpy V2 implementation]
      Signed-off-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Acked-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Acked-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Bob Picco <bob.picco@hp.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      5ad333eb
    • Mel Gorman's avatar
      Add a movablecore= parameter for sizing ZONE_MOVABLE · 7e63efef
      Mel Gorman authored
      
      
      This patch adds a new parameter for sizing ZONE_MOVABLE called
      movablecore=.  While kernelcore= is used to specify the minimum amount of
      memory that must be available for all allocation types, movablecore= is
      used to specify the minimum amount of memory that is used for migratable
      allocations.  The amount of memory used for migratable allocations
      determines how large the huge page pool could be dynamically resized to at
      runtime for example.
      
      How movablecore is actually handled is that the total number of pages in
      the system is calculated and a value is set for kernelcore that is
      
      kernelcore == totalpages - movablecore
      
      Both kernelcore= and movablecore= can be safely specified at the same time.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      7e63efef
    • Mel Gorman's avatar
      handle kernelcore=: generic · ed7ed365
      Mel Gorman authored
      
      
      This patch adds the kernelcore= parameter for x86.
      
      Once all patches are applied, a new command-line parameter exist and a new
      sysctl.  This patch adds the necessary documentation.
      
      From: Yasunori Goto <y-goto@jp.fujitsu.com>
      
        When "kernelcore" boot option is specified, kernel can't boot up on ia64
        because of an infinite loop.  In addition, the parsing code can be handled
        in an architecture-independent manner.
      
        This patch uses common code to handle the kernelcore= parameter.  It is
        only available to architectures that support arch-independent zone-sizing
        (i.e.  define CONFIG_ARCH_POPULATES_NODE_MAP).  Other architectures will
        ignore the boot parameter.
      
      [bunk@stusta.de: make cmdline_parse_kernelcore() static]
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Signed-off-by: default avatarYasunori Goto <y-goto@jp.fujitsu.com>
      Acked-by: default avatarAndy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      ed7ed365
    • Mel Gorman's avatar
      Create the ZONE_MOVABLE zone · 2a1e274a
      Mel Gorman authored
      
      
      The following 8 patches against 2.6.20-mm2 create a zone called ZONE_MOVABLE
      that is only usable by allocations that specify both __GFP_HIGHMEM and
      __GFP_MOVABLE.  This has the effect of keeping all non-movable pages within a
      single memory partition while allowing movable allocations to be satisfied
      from either partition.  The patches may be applied with the list-based
      anti-fragmentation patches that groups pages together based on mobility.
      
      The size of the zone is determined by a kernelcore= parameter specified at
      boot-time.  This specifies how much memory is usable by non-movable
      allocations and the remainder is used for ZONE_MOVABLE.  Any range of pages
      within ZONE_MOVABLE can be released by migrating the pages or by reclaiming.
      
      When selecting a zone to take pages from for ZONE_MOVABLE, there are two
      things to consider.  First, only memory from the highest populated zone is
      used for ZONE_MOVABLE.  On the x86, this is probably going to be ZONE_HIGHMEM
      but it would be ZONE_DMA on ppc64 or possibly ZONE_DMA32 on x86_64.  Second,
      the amount of memory usable by the kernel will be spread evenly throughout
      NUMA nodes where possible.  If the nodes are not of equal size, the amount of
      memory usable by the kernel on some nodes may be greater than others.
      
      By default, the zone is not as useful for hugetlb allocations because they are
      pinned and non-migratable (currently at least).  A sysctl is provided that
      allows huge pages to be allocated from that zone.  This means that the huge
      page pool can be resized to the size of ZONE_MOVABLE during the lifetime of
      the system assuming that pages are not mlocked.  Despite huge pages being
      non-movable, we do not introduce additional external fragmentation of note as
      huge pages are always the largest contiguous block we care about.
      
      Credit goes to Andy Whitcroft for catching a large variety of problems during
      review of the patches.
      
      This patch creates an additional zone, ZONE_MOVABLE.  This zone is only usable
      by allocations which specify both __GFP_HIGHMEM and __GFP_MOVABLE.  Hot-added
      memory continues to be placed in their existing destination as there is no
      mechanism to redirect them to a specific zone.
      
      [y-goto@jp.fujitsu.com: Fix section mismatch of memory hotplug related code]
      [akpm@linux-foundation.org: various fixes]
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Andy Whitcroft <apw@shadowen.org>
      Signed-off-by: default avatarYasunori Goto <y-goto@jp.fujitsu.com>
      Cc: William Lee Irwin III <wli@holomorphy.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      2a1e274a
  10. 16 Jul, 2007 1 commit