All new accounts created on Gitlab now require administrator approval. If you invite any collaborators, please let Flux staff know so they can approve the accounts.

  1. 22 Sep, 2009 40 commits
    • KAMEZAWA Hiroyuki's avatar
      kcore: /proc/kcore should use vread · 73d7c33e
      KAMEZAWA Hiroyuki authored
      /proc/kcore has its own routine to access vmallc area.  It can be replaced
      with vread().  And by this, /proc/kcore can do safe access to vmalloc
      area.
      Signed-off-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: WANG Cong <xiyou.wangcong@gmail.com>
      Cc: Mike Smith <scgtrp@gmail.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      73d7c33e
    • KAMEZAWA Hiroyuki's avatar
      kcore: fix vread/vwrite to be aware of holes · d0107eb0
      KAMEZAWA Hiroyuki authored
      vread/vwrite access vmalloc area without checking there is a page or not.
      In most case, this works well.
      
      In old ages, the caller of get_vm_ara() is only IOREMAP and there is no
      memory hole within vm_struct's [addr...addr + size - PAGE_SIZE] (
      -PAGE_SIZE is for a guard page.)
      
      After per-cpu-alloc patch, it uses get_vm_area() for reserve continuous
      virtual address but remap _later_.  There tend to be a hole in valid
      vmalloc area in vm_struct lists.  Then, skip the hole (not mapped page) is
      necessary.  This patch updates vread/vwrite() for avoiding memory hole.
      
      Routines which access vmalloc area without knowing for which addr is used
      are
        - /proc/kcore
        - /dev/kmem
      
      kcore checks IOREMAP, /dev/kmem doesn't.  After this patch, IOREMAP is
      checked and /dev/kmem will avoid to read/write it.  Fixes to /proc/kcore
      will be in the next patch in series.
      Signed-off-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: WANG Cong <xiyou.wangcong@gmail.com>
      Cc: Mike Smith <scgtrp@gmail.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d0107eb0
    • KAMEZAWA Hiroyuki's avatar
      vmalloc: unmap vmalloc area after hiding it · dd32c279
      KAMEZAWA Hiroyuki authored
      vmap area should be purged after vm_struct is removed from the list
      because vread/vwrite etc...believes the range is valid while it's on
      vm_struct list.
      Signed-off-by: default avatarKAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Reviewed-by: default avatarWANG Cong <xiyou.wangcong@gmail.com>
      Cc: Mike Smith <scgtrp@gmail.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      dd32c279
    • Mel Gorman's avatar
      page-allocator: change migratetype for all pageblocks within a high-order page... · 2f66a68f
      Mel Gorman authored
      page-allocator: change migratetype for all pageblocks within a high-order page during __rmqueue_fallback
      
      When there are no pages of a target migratetype free, the page allocator
      selects a high-order block of another migratetype to allocate from.  When
      the order of the page taken is greater than pageblock_order, all
      pageblocks within that high-order page should change migratetype so that
      pages are later freed to the correct free-lists.
      
      The current behaviour is that pageblocks change migratetype if the order
      being split matches the pageblock_order.  When pageblock_order <
      MAX_ORDER-1, ownership is not changing correct and pages are being later
      freed to the incorrect list and this impacts fragmentation avoidance.
      
      This patch changes all pageblocks within the high-order page being split
      to the correct migratetype.  Without the patch, allocation success rates
      for hugepages under stress were about 59% of physical memory on x86-64.
      With the patch applied, this goes up to 65%.
      Signed-off-by: default avatarMel Gorman <mel@csn.ul.ie>
      Cc: Andy Whitcroft <apw@shadowen.org>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      2f66a68f
    • Benjamin Herrenschmidt's avatar
      mm: kmem_cache_create(): make it easier to catch NULL cache names · fe1ff49d
      Benjamin Herrenschmidt authored
      Right now, if you inadvertently pass NULL to kmem_cache_create() at boot
      time, it crashes much later after boot somewhere deep inside sysfs which
      makes it very non obvious to figure out what's going on.
      Signed-off-by: default avatarBenjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Pekka Enberg <penberg@cs.helsinki.fi>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      fe1ff49d
    • Moussa A. Ba's avatar
      pagemap clear_refs: modify to specify anon or mapped vma clearing · 398499d5
      Moussa A. Ba authored
      The patch makes the clear_refs more versatile in adding the option to
      select anonymous pages or file backed pages for clearing.  This addition
      has a measurable impact on user space application performance as it
      decreases the number of pagewalks in scenarios where one is only
      interested in a specific type of page (anonymous or file mapped).
      
      The patch adds anonymous and file backed filters to the clear_refs interface.
      
      echo 1 > /proc/PID/clear_refs resets the bits on all pages
      echo 2 > /proc/PID/clear_refs resets the bits on anonymous pages only
      echo 3 > /proc/PID/clear_refs resets the bits on file backed pages only
      
      Any other value is ignored
      Signed-off-by: default avatarMoussa A. Ba <moussa.a.ba@gmail.com>
      Signed-off-by: default avatarJared E. Hulbert <jaredeh@gmail.com>
      Acked-by: default avatarDavid Rientjes <rientjes@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      398499d5
    • Hugh Dickins's avatar
      ksm: mremap use err from ksm_madvise · 7103ad32
      Hugh Dickins authored
      mremap move's use of ksm_madvise() was assuming -ENOMEM on failure,
      because ksm_madvise used to say -EAGAIN for that; but ksm_madvise now says
      -ENOMEM (letting madvise convert that to -EAGAIN), and can also say
      -ERESTARTSYS when signalled: so pass the error from ksm_madvise.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      7103ad32
    • Hugh Dickins's avatar
      ksm: unmerge is an origin of OOMs · 35451bee
      Hugh Dickins authored
      Just as the swapoff system call allocates many pages of RAM to various
      processes, perhaps triggering OOM, so "echo 2 >/sys/kernel/mm/ksm/run"
      (unmerge) is liable to allocate many pages of RAM to various processes,
      perhaps triggering OOM; and each is normally run from a modest admin
      process (swapoff or shell), easily repeated until it succeeds.
      
      So treat unmerge_and_remove_all_rmap_items() in the same way that we treat
      try_to_unuse(): generalize PF_SWAPOFF to PF_OOM_ORIGIN, and bracket both
      with that, to ask the OOM killer to kill them first, to prevent them from
      spawning more and more OOM kills.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      35451bee
    • Hugh Dickins's avatar
      ksm: clean up obsolete references · a913e182
      Hugh Dickins authored
      A few cleanups, given the munlock fix: the comment on ksm_test_exit() no
      longer applies, and it can be made private to ksm.c; there's no more
      reference to mmu_gather or tlb.h, and mmap.c doesn't need ksm.h.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      a913e182
    • Hugh Dickins's avatar
      ksm: remove VM_MERGEABLE_FLAGS · 8314c4f2
      Hugh Dickins authored
      KSM originally stood for Kernel Shared Memory: but the kernel has long
      supported shared memory, and VM_SHARED and VM_MAYSHARE vmas, and KSM is
      something else.  So we switched to saying "merge" instead of "share".
      
      But Chris Wright points out that this is confusing where mmap.c merges
      adjacent vmas: most especially in the name VM_MERGEABLE_FLAGS, used by
      is_mergeable_vma() to let vmas be merged despite flags being different.
      
      Call it VMA_MERGE_DESPITE_FLAGS?  Perhaps, but at present it consists
      only of VM_CAN_NONLINEAR: so for now it's clearer on all sides to use
      that directly, with a comment on it in is_mergeable_vma().
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      8314c4f2
    • Hugh Dickins's avatar
      ksm: add some documentation · 7701c9c0
      Hugh Dickins authored
      Add Documentation/vm/ksm.txt: how to use the Kernel Samepage Merging feature
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: Michael Kerrisk <mtk.manpages@googlemail.com>
      Cc: Randy Dunlap <randy.dunlap@oracle.com>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      7701c9c0
    • Hugh Dickins's avatar
      ksm: sysfs and defaults · 2ffd8679
      Hugh Dickins authored
      At present KSM is just a waste of space if you don't have CONFIG_SYSFS=y
      to provide the /sys/kernel/mm/ksm files to tune and activate it.
      
      Make KSM depend on SYSFS?  Could do, but it might be better to provide
      some defaults so that KSM works out-of-the-box, ready for testers to
      madvise MADV_MERGEABLE, even without SYSFS.
      
      Though anyone serious is likely to want to retune the numbers to their
      taste once they have experience; and whether these settings ever reach
      2.6.32 can be discussed along the way.
      
      Save 1kB from tiny kernels by #ifdef'ing the SYSFS side of it.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      2ffd8679
    • Andrea Arcangeli's avatar
      ksm: fix deadlock with munlock in exit_mmap · 1c2fb7a4
      Andrea Arcangeli authored
      Rawhide users have reported hang at startup when cryptsetup is run: the
      same problem can be simply reproduced by running a program int main() {
      mlockall(MCL_CURRENT | MCL_FUTURE); return 0; }
      
      The problem is that exit_mmap() applies munlock_vma_pages_all() to
      clean up VM_LOCKED areas, and its current implementation (stupidly)
      tries to fault in absent pages, for example where PROT_NONE prevented
      them being faulted in when mlocking.  Whereas the "ksm: fix oom
      deadlock" patch, knowing there's a race by which KSM might try to fault
      in pages after exit_mmap() had finally zapped the range, backs out of
      such faults doing nothing when its ksm_test_exit() notices mm_users 0.
      
      So revert that part of "ksm: fix oom deadlock" which moved the
      ksm_exit() call from before exit_mmap() to the middle of exit_mmap();
      and remove those ksm_test_exit() checks from the page fault paths, so
      allowing the munlocking to proceed without interference.
      
      ksm_exit, if there are rmap_items still chained on this mm slot, takes
      mmap_sem write side: so preventing KSM from working on an mm while
      exit_mmap runs.  And KSM will bail out as soon as it notices that
      mm_users is already zero, thanks to its internal ksm_test_exit checks.
      So that when a task is killed by OOM killer or the user, KSM will not
      indefinitely prevent it from running exit_mmap to release its memory.
      
      This does break a part of what "ksm: fix oom deadlock" was trying to
      achieve.  When unmerging KSM (echo 2 >/sys/kernel/mm/ksm), and even
      when ksmd itself has to cancel a KSM page, it is possible that the
      first OOM-kill victim would be the KSM process being faulted: then its
      memory won't be freed until a second victim has been selected (freeing
      memory for the unmerging fault to complete).
      
      But the OOM killer is already liable to kill a second victim once the
      intended victim's p->mm goes to NULL: so there's not much point in
      rejecting this KSM patch before fixing that OOM behaviour.  It is very
      much more important to allow KSM users to boot up, than to haggle over
      an unlikely and poorly supported OOM case.
      
      We also intend to fix munlocking to not fault pages: at which point
      this patch _could_ be reverted; though that would be controversial, so
      we hope to find a better solution.
      Signed-off-by: default avatarAndrea Arcangeli <aarcange@redhat.com>
      Acked-by: default avatarJustin M. Forbes <jforbes@redhat.com>
      Acked-for-now-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: Izik Eidus <ieidus@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      1c2fb7a4
    • Hugh Dickins's avatar
      ksm: fix oom deadlock · 9ba69294
      Hugh Dickins authored
      There's a now-obvious deadlock in KSM's out-of-memory handling:
      imagine ksmd or KSM_RUN_UNMERGE handling, holding ksm_thread_mutex,
      trying to allocate a page to break KSM in an mm which becomes the
      OOM victim (quite likely in the unmerge case): it's killed and goes
      to exit, and hangs there waiting to acquire ksm_thread_mutex.
      
      Clearly we must not require ksm_thread_mutex in __ksm_exit, simple
      though that made everything else: perhaps use mmap_sem somehow?
      And part of the answer lies in the comments on unmerge_ksm_pages:
      __ksm_exit should also leave all the rmap_item removal to ksmd.
      
      But there's a fundamental problem, that KSM relies upon mmap_sem to
      guarantee the consistency of the mm it's dealing with, yet exit_mmap
      tears down an mm without taking mmap_sem.  And bumping mm_users won't
      help at all, that just ensures that the pages the OOM killer assumes
      are on their way to being freed will not be freed.
      
      The best answer seems to be, to move the ksm_exit callout from just
      before exit_mmap, to the middle of exit_mmap: after the mm's pages
      have been freed (if the mmu_gather is flushed), but before its page
      tables and vma structures have been freed; and down_write,up_write
      mmap_sem there to serialize with KSM's own reliance on mmap_sem.
      
      But KSM then needs to be careful, whenever it downs mmap_sem, to
      check that the mm is not already exiting: there's a danger of using
      find_vma on a layout that's being torn apart, or writing into page
      tables which have been freed for reuse; and even do_anonymous_page
      and __do_fault need to check they're not being called by break_ksm
      to reinstate a pte after zap_pte_range has zapped that page table.
      
      Though it might be clearer to add an exiting flag, set while holding
      mmap_sem in __ksm_exit, that wouldn't cover the issue of reinstating
      a zapped pte.  All we need is to check whether mm_users is 0 - but
      must remember that ksmd may detect that before __ksm_exit is reached.
      So, ksm_test_exit(mm) added to comment such checks on mm->mm_users.
      
      __ksm_exit now has to leave clearing up the rmap_items to ksmd,
      that needs ksm_thread_mutex; but shift the exiting mm just after the
      ksm_scan cursor so that it will soon be dealt with.  __ksm_enter raise
      mm_count to hold the mm_struct, ksmd's exit processing (exactly like
      its processing when it finds all VM_MERGEABLEs unmapped) mmdrop it,
      similar procedure for KSM_RUN_UNMERGE (which has stopped ksmd).
      
      But also give __ksm_exit a fast path: when there's no complication
      (no rmap_items attached to mm and it's not at the ksm_scan cursor),
      it can safely do all the exiting work itself.  This is not just an
      optimization: when ksmd is not running, the raised mm_count would
      otherwise leak mm_structs.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      9ba69294
    • Hugh Dickins's avatar
      ksm: distribute remove_mm_from_lists · cd551f97
      Hugh Dickins authored
      Do some housekeeping in ksm.c, to help make the next patch easier
      to understand: remove the function remove_mm_from_lists, distributing
      its code to its callsites scan_get_next_rmap_item and __ksm_exit.
      
      That turns out to be a win in scan_get_next_rmap_item: move its
      remove_trailing_rmap_items and cursor advancement up, and it becomes
      simpler than before.  __ksm_exit becomes messier, but will change
      again; and moving its remove_trailing_rmap_items up lets us strengthen
      the unstable tree item's age condition in remove_rmap_item_from_tree.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      cd551f97
    • Hugh Dickins's avatar
      ksm: fix endless loop on oom · d952b791
      Hugh Dickins authored
      break_ksm has been looping endlessly ignoring VM_FAULT_OOM: that should
      only be a problem for ksmd when a memory control group imposes limits
      (normally the OOM killer will kill others with an mm until it succeeds);
      but in general (especially for MADV_UNMERGEABLE and KSM_RUN_UNMERGE) we
      do need to route the error (or kill) back to the caller (or sighandling).
      
      Test signal_pending in unmerge_ksm_pages, which could be a lengthy
      procedure if it has to spill into swap: returning -ERESTARTSYS so that
      trivial signals will restart but fatals will terminate (is that right?
      we do different things in different places in mm, none exactly this).
      
      unmerge_and_remove_all_rmap_items was forgetting to lock when going
      down the mm_list: fix that.  Whether it's successful or not, reset
      ksm_scan cursor to head; but only if it's successful, reset seqnr
      (shown in full_scans) - page counts will have gone down to zero.
      
      This patch leaves a significant OOM deadlock, but it's a good step
      on the way, and that deadlock is fixed in a subsequent patch.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d952b791
    • Hugh Dickins's avatar
      ksm: five little cleanups · 81464e30
      Hugh Dickins authored
      1. We don't use __break_cow entry point now: merge it into break_cow.
      2. remove_all_slot_rmap_items is just a special case of
         remove_trailing_rmap_items: use the latter instead.
      3. Extend comment on unmerge_ksm_pages and rmap_items.
      4. try_to_merge_two_pages should use try_to_merge_with_ksm_page
         instead of duplicating its code; and so swap them around.
      5. Comment on cmp_and_merge_page described last year's: update it.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      81464e30
    • Hugh Dickins's avatar
      ksm: keep quiet while list empty · 6e158384
      Hugh Dickins authored
      ksm_scan_thread already sleeps in wait_event_interruptible until setting
      ksm_run activates it; but if there's nothing on its list to look at, i.e.
      nobody has yet said madvise MADV_MERGEABLE, it's a shame to be clocking
      up system time and full_scans: ksmd_should_run added to check that too.
      
      And move the mutex_lock out around it: the new counts showed that when
      ksm_run is stopped, a little work often got done afterwards, because it
      had been read before taking the mutex.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      6e158384
    • Hugh Dickins's avatar
      ksm: break cow once unshared · 26465d3e
      Hugh Dickins authored
      We kept agreeing not to bother about the unswappable shared KSM pages
      which later become unshared by others: observation suggests they're not
      a significant proportion.  But they are disadvantageous, and it is easier
      to break COW to replace them by swappable pages, than offer statistics
      to show that they don't matter; then we can stop worrying about them.
      
      Doing this in ksm_do_scan, they don't go through cmp_and_merge_page on
      this pass: give them a good chance of getting into the unstable tree
      on the next pass, or back into the stable, by computing checksum now.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      26465d3e
    • Hugh Dickins's avatar
      ksm: pages_unshared and pages_volatile · 473b0ce4
      Hugh Dickins authored
      The pages_shared and pages_sharing counts give a good picture of how
      successful KSM is at sharing; but no clue to how much wasted work it's
      doing to get there.  Add pages_unshared (count of unique pages waiting
      in the unstable tree, hoping to find a mate) and pages_volatile.
      
      pages_volatile is harder to define.  It includes those pages changing
      too fast to get into the unstable tree, but also whatever other edge
      conditions prevent a page getting into the trees: a high value may
      deserve investigation.  Don't try to calculate it from the various
      conditions: it's the total of rmap_items less those accounted for.
      
      Also show full_scans: the number of completed scans of everything
      registered in the mm list.
      
      The locking for all these counts is simply ksm_thread_mutex.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Acked-by: default avatarAndrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      473b0ce4
    • Hugh Dickins's avatar
      ksm: move pages_sharing updates · e178dfde
      Hugh Dickins authored
      The pages_shared count is incremented and decremented when adding a node
      to and removing a node from the stable tree: easy to understand.  But the
      pages_sharing count was hard to follow, being adjusted in various places:
      increment and decrement it when adding to and removing from the stable tree.
      
      And the pages_sharing variable used to include the pages_shared, then those
      were subtracted when shown in the pages_sharing sysfs file: now keep it as
      an exclusive count of leaves hanging off the stable tree nodes, throughout.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Acked-by: default avatarAndrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      e178dfde
    • Hugh Dickins's avatar
      ksm: rename kernel_pages_allocated · b4028260
      Hugh Dickins authored
      We're not implementing swapping of KSM pages in its first release;
      but when that follows, "kernel_pages_allocated" will be a very poor
      name for the sysfs file showing number of nodes in the stable tree:
      rename that to "pages_shared" throughout.
      
      But we already have a "pages_shared", counting those page slots
      sharing the shared pages: first rename that to... "pages_sharing".
      
      What will become of "max_kernel_pages" when the pages shared can
      be swapped?  I guess it will just be removed, so keep that name.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Acked-by: default avatarIzik Eidus <ieidus@redhat.com>
      Acked-by: default avatarAndrea Arcangeli <aarcange@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      b4028260
    • Izik Eidus's avatar
      ksm: change ksm nice level to be 5 · 339aa624
      Izik Eidus authored
      ksm should try not to disturb other tasks as much as possible.
      Signed-off-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Chris Wright <chrisw@redhat.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Balbir Singh <balbir@in.ibm.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: Avi Kivity <avi@redhat.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      339aa624
    • Izik Eidus's avatar
      ksm: change copyright message · 36b2528d
      Izik Eidus authored
      Adding Hugh Dickins into the authors list.
      Signed-off-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Chris Wright <chrisw@redhat.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Balbir Singh <balbir@in.ibm.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: Avi Kivity <avi@redhat.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      36b2528d
    • Hugh Dickins's avatar
      ksm: prevent mremap move poisoning · 1ff82995
      Hugh Dickins authored
      KSM's scan allows for user pages to be COWed or unmapped at any time,
      without requiring any notification.  But its stable tree does assume that
      when it finds a KSM page where it placed a KSM page, then it is the same
      KSM page that it placed there.
      
      mremap move could break that assumption: if an area containing a KSM page
      was unmapped, then an area containing a different KSM page was moved with
      mremap into the place of the original, before KSM's scan came around to
      notice.  That could then poison a node of the stable tree, so that memcmps
      would "lie" and upset the ordering of the tree.
      
      Probably noone will ever need mremap move on a VM_MERGEABLE area; except
      that prohibiting it would make trouble for schemes in which we try making
      everything VM_MERGEABLE e.g.  for testing: an mremap which normally works
      would then fail mysteriously.
      
      There's no need to go to any trouble, such as re-sorting KSM's list of
      rmap_items to match the new layout: simply unmerge the area to COW all its
      KSM pages before moving, but leave VM_MERGEABLE on so that they're
      remerged later.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Signed-off-by: default avatarChris Wright <chrisw@redhat.com>
      Signed-off-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Balbir Singh <balbir@in.ibm.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: Avi Kivity <avi@redhat.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      1ff82995
    • Izik Eidus's avatar
      ksm: Kernel SamePage Merging · 31dbd01f
      Izik Eidus authored
      Ksm is code that allows merging of identical pages between one or more
      applications, in a way invisible to the applications that use it.  Pages
      that are merged are marked as read-only, then COWed when any application
      tries to change them.
      
      Whereas fork() allows sharing anonymous pages between parent and child,
      ksm can share anonymous pages between unrelated processes.
      
      Ksm works by walking over the memory pages of the applications it scans,
      in order to find identical pages.  It uses two sorted data structures,
      called the stable and unstable trees, to locate identical pages in an
      effective way.
      
      When ksm finds two identical pages, it marks them as readonly and merges
      them into a single page.  After the pages have been marked as readonly and
      merged into one, Linux treats them as normal copy-on-write pages, copying
      to a fresh anonymous page if write access is required later.
      
      Ksm scans and merges anonymous pages only in those memory areas that have
      been registered with it by madvise(addr, length, MADV_MERGEABLE).
      
      The ksm scanner is controlled by sysfs files in /sys/kernel/mm/ksm/:
      
      max_kernel_pages - the maximum number of unswappable kernel pages
                         which may be allocated by ksm (0 for unlimited).
      
      kernel_pages_allocated - how many ksm pages are currently allocated,
                               sharing identical content between different
                               processes (pages unswappable in this release).
      
      pages_shared - how many pages have been saved by sharing with ksm pages
                     (kernel_pages_allocated being excluded from this count).
      
      pages_to_scan - how many pages ksm should scan before sleeping.
      
      sleep_millisecs - how many milliseconds ksm should sleep between scans.
      
      run - write 0 to disable ksm, read 0 while ksm is disabled (default),
            write 1 to run ksm, read 1 while ksm is running,
            write 2 to disable ksm and unmerge all its pages.
      
      Includes contributions by Andrea Arcangeli Chris Wright and Hugh Dickins.
      
      [hugh.dickins@tiscali.co.uk: fix rare page leak]
      Signed-off-by: default avatarIzik Eidus <ieidus@redhat.com>
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Signed-off-by: default avatarChris Wright <chrisw@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Balbir Singh <balbir@in.ibm.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: Avi Kivity <avi@redhat.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      31dbd01f
    • Hugh Dickins's avatar
      ksm: identify PageKsm pages · 9a840895
      Hugh Dickins authored
      KSM will need to identify its kernel merged pages unambiguously, and
      /proc/kpageflags will probably like to do so too.
      
      Since KSM will only be substituting anonymous pages, statistics are best
      preserved by making a PageKsm page a special PageAnon page: one with no
      anon_vma.
      
      But KSM then needs its own page_add_ksm_rmap() - keep it in ksm.h near
      PageKsm; and do_wp_page() must COW them, unlike singly mapped PageAnons.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Signed-off-by: default avatarChris Wright <chrisw@redhat.com>
      Signed-off-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Balbir Singh <balbir@in.ibm.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: Avi Kivity <avi@redhat.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      9a840895
    • Hugh Dickins's avatar
      ksm: no debug in page_dup_rmap() · 21333b2b
      Hugh Dickins authored
      page_dup_rmap(), used on each mapped page when forking, was originally
      just an inline atomic_inc of mapcount.  2.6.22 added CONFIG_DEBUG_VM
      out-of-line checks to it, which would need to be ever-so-slightly
      complicated to allow for the PageKsm() we're about to define.
      
      But I think these checks never caught anything.  And if it's coding errors
      we're worried about, such checks should be in page_remove_rmap() too, not
      just when forking; whereas if it's pagetable corruption we're worried
      about, then they shouldn't be limited to CONFIG_DEBUG_VM.
      
      Oh, just revert page_dup_rmap() to an inline atomic_inc of mapcount.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Signed-off-by: default avatarChris Wright <chrisw@redhat.com>
      Signed-off-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Nick Piggin <npiggin@suse.de>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Balbir Singh <balbir@in.ibm.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: Avi Kivity <avi@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      21333b2b
    • Hugh Dickins's avatar
      ksm: the mm interface to ksm · f8af4da3
      Hugh Dickins authored
      This patch presents the mm interface to a dummy version of ksm.c, for
      better scrutiny of that interface: the real ksm.c follows later.
      
      When CONFIG_KSM is not set, madvise(2) reject MADV_MERGEABLE and
      MADV_UNMERGEABLE with EINVAL, since that seems more helpful than
      pretending that they can be serviced.  But when CONFIG_KSM=y, accept them
      even if KSM is not currently running, and even on areas which KSM will not
      touch (e.g.  hugetlb or shared file or special driver mappings).
      
      Like other madvices, report ENOMEM despite success if any area in the
      range is unmapped, and use EAGAIN to report out of memory.
      
      Define vma flag VM_MERGEABLE to identify an area on which KSM may try
      merging pages: leave it to ksm_madvise() to decide whether to set it.
      Define mm flag MMF_VM_MERGEABLE to identify an mm which might contain
      VM_MERGEABLE areas, to minimize callouts when forking or exiting.
      
      Based upon earlier patches by Chris Wright and Izik Eidus.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Signed-off-by: default avatarChris Wright <chrisw@redhat.com>
      Signed-off-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Michael Kerrisk <mtk.manpages@gmail.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Balbir Singh <balbir@in.ibm.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: Avi Kivity <avi@redhat.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      f8af4da3
    • Hugh Dickins's avatar
      ksm: define MADV_MERGEABLE and MADV_UNMERGEABLE · d19f3524
      Hugh Dickins authored
      The out-of-tree KSM used ioctls on fds cloned from /dev/ksm to register a
      memory area for merging: we prefer now to use an madvise(2) interface.
      
      This patch just defines MADV_MERGEABLE (to tell KSM it may merge pages in
      this area found identical to pages in other mergeable areas) and
      MADV_UNMERGEABLE (to undo that).
      
      Most architectures use asm-generic, but alpha, mips, parisc, xtensa need
      their own definitions: included here for mmotm convenience, but we'll
      probably want to split this and feed pieces to arch maintainers.
      
      Based upon earlier patches by Chris Wright and Izik Eidus.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Signed-off-by: default avatarChris Wright <chrisw@redhat.com>
      Signed-off-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Michael Kerrisk <mtk.manpages@gmail.com>
      Cc: Richard Henderson <rth@twiddle.net>
      Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
      Cc: Ralf Baechle <ralf@linux-mips.org>
      Cc: Kyle McMartin <kyle@mcmartin.ca>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Chris Zankel <chris@zankel.net>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Balbir Singh <balbir@in.ibm.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: Avi Kivity <avi@redhat.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      d19f3524
    • Hugh Dickins's avatar
      ksm: first tidy up madvise_vma() · 3866ea90
      Hugh Dickins authored
      madvise.c has several levels of switch statements, what to do in which?
      Move MADV_DOFORK code down from madvise_vma() to madvise_behavior(), so
      madvise_vma() can be a simple router, to madvise_behavior() by default.
      
      vma->vm_flags is an unsigned long so use the same type for new_flags.  Add
      missing comment lines to describe MADV_DONTFORK and MADV_DOFORK.
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Signed-off-by: default avatarChris Wright <chrisw@redhat.com>
      Signed-off-by: default avatarIzik Eidus <ieidus@redhat.com>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Balbir Singh <balbir@in.ibm.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: Avi Kivity <avi@redhat.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      3866ea90
    • Izik Eidus's avatar
      ksm: add mmu_notifier set_pte_at_notify() · 828502d3
      Izik Eidus authored
      KSM is a linux driver that allows dynamicly sharing identical memory pages
      between one or more processes.
      
      Unlike tradtional page sharing that is made at the allocation of the
      memory, ksm do it dynamicly after the memory was created.  Memory is
      periodically scanned; identical pages are identified and merged.
      
      The sharing is made in a transparent way to the processes that use it.
      
      Ksm is highly important for hypervisors (kvm), where in production
      enviorments there might be many copys of the same data data among the host
      memory.  This kind of data can be: similar kernels, librarys, cache, and
      so on.
      
      Even that ksm was wrote for kvm, any userspace application that want to
      use it to share its data can try it.
      
      Ksm may be useful for any application that might have similar (page
      aligment) data strctures among the memory, ksm will find this data merge
      it to one copy, and even if it will be changed and thereforew copy on
      writed, ksm will merge it again as soon as it will be identical again.
      
      Another reason to consider using ksm is the fact that it might simplify
      alot the userspace code of application that want to use shared private
      data, instead that the application will mange shared area, ksm will do
      this for the application, and even write to this data will be allowed
      without any synchinization acts from the application.
      
      Ksm was designed to be a loadable module that doesn't change the VM code
      of linux.
      
      This patch:
      
      The set_pte_at_notify() macro allows setting a pte in the shadow page
      table directly, instead of flushing the shadow page table entry and then
      getting vmexit to set it.  It uses a new change_pte() callback to do so.
      
      set_pte_at_notify() is an optimization for kvm, and other users of
      mmu_notifiers, for COW pages.  It is useful for kvm when ksm is used,
      because it allows kvm not to have to receive vmexit and only then map the
      ksm page into the shadow page table, but instead map it directly at the
      same time as Linux maps the page into the host page table.
      
      Users of mmu_notifiers who don't implement new mmu_notifier_change_pte()
      callback will just receive the mmu_notifier_invalidate_page() callback.
      Signed-off-by: default avatarIzik Eidus <ieidus@redhat.com>
      Signed-off-by: default avatarChris Wright <chrisw@redhat.com>
      Signed-off-by: default avatarHugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Cc: Balbir Singh <balbir@in.ibm.com>
      Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
      Cc: Avi Kivity <avi@redhat.com>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      828502d3
    • Johannes Weiner's avatar
      mm: perform non-atomic test-clear of PG_mlocked on free · 451ea25d
      Johannes Weiner authored
      By the time PG_mlocked is cleared in the page freeing path, nobody else is
      looking at our page->flags anymore.
      
      It is thus safe to make the test-and-clear non-atomic and thereby removing
      an unnecessary and expensive operation from a hotpath.
      Signed-off-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Reviewed-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Reviewed-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      Cc: Mel Gorman <mel@csn.ul.ie>
      Cc: Nick Piggin <nickpiggin@yahoo.com.au>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      451ea25d
    • Figo.zhang's avatar
      vmalloc.c: fix double error checking · bf88c8c8
      Figo.zhang authored
      There is no need for double error checking.
      Signed-off-by: default avatarFigo.zhang <figo1802@gmail.com>
      Acked-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      bf88c8c8
    • Akinobu Mita's avatar
      mm: add gfp mask checking for __get_free_pages() · 945a1113
      Akinobu Mita authored
      __get_free_pages() with __GFP_HIGHMEM is not safe because the return
      address cannot represent a highmem page.  get_zeroed_page() already has
      such a debug checking.
      Signed-off-by: default avatarAkinobu Mita <akinobu.mita@gmail.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      945a1113
    • KOSAKI Motohiro's avatar
      vmscan: kill unnecessary prefetch · a26f5320
      KOSAKI Motohiro authored
      The pages in the list passed move_active_pages_to_lru() are already
      touched by shrink_active_list().  IOW the prefetch in
      move_active_pages_to_lru() don't populate any cache.  it's pointless.
      
      This patch remove it.
      Signed-off-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Reviewed-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Minchan Kim <minchan.kim@gmail.com>
      Cc: Mel Gorman <mel@csn.ul.ie>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      a26f5320
    • KOSAKI Motohiro's avatar
      vmscan: kill unnecessary page flag test · 74a1c48f
      KOSAKI Motohiro authored
      The page_lru() already evaluate PageActive() and PageSwapBacked().  We
      don't need to re-evaluate it.
      Signed-off-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Reviewed-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Minchan Kim <minchan.kim@gmail.com>
      Cc: Mel Gorman <mel@csn.ul.ie>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      74a1c48f
    • KOSAKI Motohiro's avatar
      vmscan: move ClearPageActive from move_active_pages() to shrink_active_list() · 5205e56e
      KOSAKI Motohiro authored
      The move_active_pages_to_lru() function is called under irq disabled and
      ClearPageActive() doesn't need irq disabling.
      
      Then, this patch move it into shrink_active_list().
      Signed-off-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Reviewed-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Minchan Kim <minchan.kim@gmail.com>
      Cc: Mel Gorman <mel@csn.ul.ie>
      Cc: Wu Fengguang <fengguang.wu@intel.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      5205e56e
    • Minchan Kim's avatar
      vmscan: don't attempt to reclaim anon page in lumpy reclaim when no swap space is available · de2e7567
      Minchan Kim authored
      The VM already avoids attempting to reclaim anon pages in various places,
      But it doesn't avoid it for lumpy reclaim.
      
      It shuffles lru list unnecessary so that it is pointless.
      
      [akpm@linux-foundation.org: cleanup]
      Signed-off-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Reviewed-by: default avatarKOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Cc: Mel Gorman <mel@csn.ul.ie>
      Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      de2e7567
    • Wu Fengguang's avatar
      mm: count only reclaimable lru pages · adea02a1
      Wu Fengguang authored
      global_lru_pages() / zone_lru_pages() can be used in two ways:
      - to estimate max reclaimable pages in determine_dirtyable_memory()
      - to calculate the slab scan ratio
      
      When swap is full or not present, the anon lru lists are not reclaimable
      and also won't be scanned.  So the anon pages shall not be counted in both
      usage scenarios.  Also rename to _reclaimable_pages: now they are counting
      the possibly reclaimable lru pages.
      
      It can greatly (and correctly) increase the slab scan rate under high
      memory pressure (when most file pages have been reclaimed and swap is
      full/absent), thus reduce false OOM kills.
      Acked-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Reviewed-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Reviewed-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Signed-off-by: default avatarWu Fengguang <fengguang.wu@intel.com>
      Acked-by: default avatarJohannes Weiner <hannes@cmpxchg.org>
      Reviewed-by: default avatarMinchan Kim <minchan.kim@gmail.com>
      Reviewed-by: default avatarJesse Barnes <jbarnes@virtuousgeek.org>
      Cc: David Howells <dhowells@redhat.com>
      Cc: "Li, Ming Chun" <macli@brc.ubc.ca>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      adea02a1