1. 25 Mar, 2015 1 commit
    • Mel Gorman's avatar
      mm: numa: preserve PTE write permissions across a NUMA hinting fault · b191f9b1
      Mel Gorman authored
      Protecting a PTE to trap a NUMA hinting fault clears the writable bit
      and further faults are needed after trapping a NUMA hinting fault to set
      the writable bit again.  This patch preserves the writable bit when
      trapping NUMA hinting faults.  The impact is obvious from the number of
      minor faults trapped during the basis balancing benchmark and the system
      CPU usage;
                                                   4.0.0-rc4             4.0.0-rc4
                                                    baseline              preserve
        Time System-NUMA01                  107.13 (  0.00%)      103.13 (  3.73%)
        Time System-NUMA01_THEADLOCAL       131.87 (  0.00%)       83.30 ( 36.83%)
        Time System-NUMA02                    8.95 (  0.00%)       10.72 (-19.78%)
        Time System-NUMA02_SMT                4.57 (  0.00%)        3.99 ( 12.69%)
        Time Elapsed-NUMA01                 515.78 (  0.00%)      517.26 ( -0.29%)
        Time Elapsed-NUMA01_THEADLOCAL      384.10 (  0.00%)      384.31 ( -0.05%)
        Time Elapsed-NUMA02                  48.86 (  0.00%)       48.78 (  0.16%)
        Time Elapsed-NUMA02_SMT              47.98 (  0.00%)       48.12 ( -0.29%)
                     4.0.0-rc4   4.0.0-rc4
                      baseline    preserve
        User          44383.95    43971.89
        System          252.61      201.24
        Elapsed         998.68     1000.94
        Minor Faults   2597249     1981230
        Major Faults       365         364
      There is a similar drop in system CPU usage using Dave Chinner's xfsrepair
                                            4.0.0-rc4             4.0.0-rc4
                                             baseline              preserve
        Amean    real-xfsrepair      454.14 (  0.00%)      442.36 (  2.60%)
        Amean    syst-xfsrepair      277.20 (  0.00%)      204.68 ( 26.16%)
      The patch looks hacky but the alternatives looked worse.  The tidest was
      to rewalk the page tables after a hinting fault but it was more complex
      than this approach and the performance was worse.  It's not generally
      safe to just mark the page writable during the fault if it's a write
      fault as it may have been read-only for COW so that approach was
      Signed-off-by: default avatarMel Gorman <mgorman@suse.de>
      Reported-by: default avatarDave Chinner <david@fromorbit.com>
      Tested-by: default avatarDave Chinner <david@fromorbit.com>
      Cc: Ingo Molnar <mingo@kernel.org>
      Cc: Aneesh Kumar <aneesh.kumar@linux.vnet.ibm.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
  2. 12 Feb, 2015 4 commits
  3. 10 Feb, 2015 1 commit
  4. 13 Oct, 2014 1 commit
    • Peter Feiner's avatar
      mm: softdirty: enable write notifications on VMAs after VM_SOFTDIRTY cleared · 64e45507
      Peter Feiner authored
      For VMAs that don't want write notifications, PTEs created for read faults
      have their write bit set.  If the read fault happens after VM_SOFTDIRTY is
      cleared, then the PTE's softdirty bit will remain clear after subsequent
      Here's a simple code snippet to demonstrate the bug:
        char* m = mmap(NULL, getpagesize(), PROT_READ | PROT_WRITE,
                       MAP_ANONYMOUS | MAP_SHARED, -1, 0);
        system("echo 4 > /proc/$PPID/clear_refs"); /* clear VM_SOFTDIRTY */
        assert(*m == '\0');     /* new PTE allows write access */
        *m = 'x';               /* should dirty the page */
        assert(soft_dirty(x));  /* fails */
      With this patch, write notifications are enabled when VM_SOFTDIRTY is
      cleared.  Furthermore, to avoid unnecessary faults, write notifications
      are disabled when VM_SOFTDIRTY is set.
      As a side effect of enabling and disabling write notifications with
      care, this patch fixes a bug in mprotect where vm_page_prot bits set by
      drivers were zapped on mprotect.  An analogous bug was fixed in mmap by
      commit c9d0bf24 ("mm: uncached vma support with writenotify").
      Signed-off-by: default avatarPeter Feiner <pfeiner@google.com>
      Reported-by: default avatarPeter Feiner <pfeiner@google.com>
      Suggested-by: default avatarKirill A. Shutemov <kirill.shutemov@linux.intel.com>
      Cc: Cyrill Gorcunov <gorcunov@openvz.org>
      Cc: Pavel Emelyanov <xemul@parallels.com>
      Cc: Jamie Liu <jamieliu@google.com>
      Cc: Hugh Dickins <hughd@google.com>
      Cc: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
      Cc: Bjorn Helgaas <bhelgaas@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
  5. 07 Apr, 2014 3 commits
  6. 16 Feb, 2014 2 commits
  7. 21 Jan, 2014 1 commit
  8. 18 Dec, 2013 3 commits
    • Rik van Riel's avatar
      mm: fix TLB flush race between migration, and change_protection_range · 20841405
      Rik van Riel authored
      There are a few subtle races, between change_protection_range (used by
      mprotect and change_prot_numa) on one side, and NUMA page migration and
      compaction on the other side.
      The basic race is that there is a time window between when the PTE gets
      made non-present (PROT_NONE or NUMA), and the TLB is flushed.
      During that time, a CPU may continue writing to the page.
      This is fine most of the time, however compaction or the NUMA migration
      code may come in, and migrate the page away.
      When that happens, the CPU may continue writing, through the cached
      translation, to what is no longer the current memory location of the
      This only affects x86, which has a somewhat optimistic pte_accessible.
      All other architectures appear to be safe, and will either always flush,
      or flush whenever there is a valid mapping, even with no permissions
      The basic race looks like this:
      CPU A			CPU B			CPU C
      						load TLB entry
      make entry PTE/PMD_NUMA
      			fault on entry
      						read/write old page
      			start migrating page
      			change PTE/PMD to new page
      						read/write old page [*]
      flush TLB
      						reload TLB from new entry
      						read/write new page
      						lose data
      [*] the old page may belong to a new user at this point!
      The obvious fix is to flush remote TLB entries, by making sure that
      pte_accessible aware of the fact that PROT_NONE and PROT_NUMA memory may
      still be accessible if there is a TLB flush pending for the mm.
      This should fix both NUMA migration and compaction.
      [mgorman@suse.de: fix build]
      Signed-off-by: default avatarRik van Riel <riel@redhat.com>
      Signed-off-by: default avatarMel Gorman <mgorman@suse.de>
      Cc: Alex Thorlton <athorlton@sgi.com>
      Cc: <stable@vger.kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    • Mel Gorman's avatar
      mm: numa: clear numa hinting information on mprotect · 1667918b
      Mel Gorman authored
      On a protection change it is no longer clear if the page should be still
      accessible.  This patch clears the NUMA hinting fault bits on a
      protection change.
      Signed-off-by: default avatarMel Gorman <mgorman@suse.de>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Cc: Alex Thorlton <athorlton@sgi.com>
      Cc: <stable@vger.kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    • Mel Gorman's avatar
      mm: numa: do not clear PTE for pte_numa update · 0c5f83c2
      Mel Gorman authored
      The TLB must be flushed if the PTE is updated but change_pte_range is
      clearing the PTE while marking PTEs pte_numa without necessarily
      flushing the TLB if it reinserts the same entry.  Without the flush,
      it's conceivable that two processors have different TLBs for the same
      virtual address and at the very least it would generate spurious faults.
      This patch only unmaps the pages in change_pte_range for a full
      protection change.
      [riel@redhat.com: write pte_numa pte back to the page tables]
      Signed-off-by: default avatarMel Gorman <mgorman@suse.de>
      Signed-off-by: default avatarRik van Riel <riel@redhat.com>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Cc: Alex Thorlton <athorlton@sgi.com>
      Cc: Chegu Vinod <chegu_vinod@hp.com>
      Cc: <stable@vger.kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
  9. 12 Nov, 2013 1 commit
    • Mel Gorman's avatar
      mm: numa: return the number of base pages altered by protection changes · 72403b4a
      Mel Gorman authored
      Commit 0255d491 ("mm: Account for a THP NUMA hinting update as one
      PTE update") was added to account for the number of PTE updates when
      marking pages prot_numa.  task_numa_work was using the old return value
      to track how much address space had been updated.  Altering the return
      value causes the scanner to do more work than it is configured or
      documented to in a single unit of work.
      This patch reverts that commit and accounts for the number of THP
      updates separately in vmstat.  It is up to the administrator to
      interpret the pair of values correctly.  This is a straight-forward
      operation and likely to only be of interest when actively debugging NUMA
      balancing problems.
      The impact of this patch is that the NUMA PTE scanner will scan slower
      when THP is enabled and workloads may converge slower as a result.  On
      the flip size system CPU usage should be lower than recent tests
      reported.  This is an illustrative example of a short single JVM specjbb
                             3.12.0                3.12.0
                            vanilla      acctupdates
      TPut 1      26143.00 (  0.00%)     25747.00 ( -1.51%)
      TPut 7     185257.00 (  0.00%)    183202.00 ( -1.11%)
      TPut 13    329760.00 (  0.00%)    346577.00 (  5.10%)
      TPut 19    442502.00 (  0.00%)    460146.00 (  3.99%)
      TPut 25    540634.00 (  0.00%)    549053.00 (  1.56%)
      TPut 31    512098.00 (  0.00%)    519611.00 (  1.47%)
      TPut 37    461276.00 (  0.00%)    474973.00 (  2.97%)
      TPut 43    403089.00 (  0.00%)    414172.00 (  2.75%)
                    3.12.0      3.12.0
      User         5169.64     5184.14
      System        100.45       80.02
      Elapsed       252.75      251.85
      Performance is similar but note the reduction in system CPU time.  While
      this showed a performance gain, it will not be universal but at least
      it'll be behaving as documented.  The vmstats are obviously different but
      here is an obvious interpretation of them from mmtests.
                                      3.12.0      3.12.0
      NUMA page range updates        1408326    11043064
      NUMA huge PMD updates                0       21040
      NUMA PTE updates               1408326      291624
      "NUMA page range updates" == nr_pte_updates and is the value returned to
      the NUMA pte scanner.  NUMA huge PMD updates were the number of THP
      updates which in combination can be used to calculate how many ptes were
      updated from userspace.
      Signed-off-by: default avatarMel Gorman <mgorman@suse.de>
      Reported-by: default avatarAlex Thorlton <athorlton@sgi.com>
      Reviewed-by: default avatarRik van Riel <riel@redhat.com>
      Cc: <stable@vger.kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
  10. 29 Oct, 2013 1 commit
  11. 16 Oct, 2013 1 commit
    • Cyrill Gorcunov's avatar
      mm: migration: do not lose soft dirty bit if page is in migration state · c3d16e16
      Cyrill Gorcunov authored
      If page migration is turned on in config and the page is migrating, we
      may lose the soft dirty bit.  If fork and mprotect are called on
      migrating pages (once migration is complete) pages do not obtain the
      soft dirty bit in the correspond pte entries.  Fix it adding an
      appropriate test on swap entries.
      Signed-off-by: default avatarCyrill Gorcunov <gorcunov@openvz.org>
      Cc: Pavel Emelyanov <xemul@parallels.com>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Matt Mackall <mpm@selenic.com>
      Cc: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com>
      Cc: Marcelo Tosatti <mtosatti@redhat.com>
      Cc: KOSAKI Motohiro <kosaki.motohiro@gmail.com>
      Cc: Stephen Rothwell <sfr@canb.auug.org.au>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com>
      Cc: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
      Cc: Mel Gorman <mel@csn.ul.ie>
      Cc: <stable@vger.kernel.org>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
  12. 09 Oct, 2013 8 commits
  13. 18 Dec, 2012 1 commit
  14. 12 Dec, 2012 1 commit
  15. 11 Dec, 2012 4 commits
    • Mel Gorman's avatar
      mm: numa: Migrate pages handled during a pmd_numa hinting fault · 9532fec1
      Mel Gorman authored
      To say that the PMD handling code was incorrectly transferred from autonuma
      is an understatement. The intention was to handle a PMDs worth of pages
      in the same fault and effectively batch the taking of the PTL and page
      migration. The copied version instead has the impact of clearing a number
      of pte_numa PTE entries and whether any page migration takes place depends
      on racing. This just happens to work in some cases.
      This patch handles pte_numa faults in batch when a pmd_numa fault is
      handled. The pages are migrated if they are currently misplaced.
      Essentially this is making an assumption that NUMA locality is
      on a PMD boundary but that could be addressed by only setting
      pmd_numa if all the pages within that PMD are on the same node
      if necessary.
      Signed-off-by: default avatarMel Gorman <mgorman@suse.de>
    • Mel Gorman's avatar
      mm: mempolicy: Implement change_prot_numa() in terms of change_protection() · 4b10e7d5
      Mel Gorman authored
      This patch converts change_prot_numa() to use change_protection(). As
      pte_numa and friends check the PTE bits directly it is necessary for
      change_protection() to use pmd_mknuma(). Hence the required
      modifications to change_protection() are a little clumsy but the
      end result is that most of the numa page table helpers are just one or
      two instructions.
      Signed-off-by: default avatarMel Gorman <mgorman@suse.de>
    • Ingo Molnar's avatar
      mm: Optimize the TLB flush of sys_mprotect() and change_protection() users · 1233d588
      Ingo Molnar authored
      Reuse the NUMA code's 'modified page protections' count that
      change_protection() computes and skip the TLB flush if there's
      no changes to a range that sys_mprotect() modifies.
      Given that mprotect() already optimizes the same-flags case
      I expected this optimization to dominantly trigger on
      CONFIG_NUMA_BALANCING=y kernels - but even with that feature
      disabled it triggers rather often.
      There's two reasons for that:
      While sys_mprotect() already optimizes the same-flag case:
              if (newflags == oldflags) {
                      *pprev = vma;
                      return 0;
      and this test works in many cases, but it is too sharp in some
      others, where it differentiates between protection values that the
      underlying PTE format makes no distinction about, such as
      PROT_EXEC == PROT_READ on x86.
      Even where the pte format over vma flag changes necessiates a
      modification of the pagetables, there might be no pagetables
      yet to modify: they might not be instantiated yet.
      During a regular desktop bootup this optimization hits a couple
      of hundred times. During a Java test I measured thousands of
      So this optimization improves sys_mprotect() in general, not just
      CONFIG_NUMA_BALANCING=y kernels.
      [ We could further increase the efficiency of this optimization if
        change_pte_range() and change_huge_pmd() was a bit smarter about
        recognizing exact-same-value protection masks - when the hardware
        can do that safely. This would probably further speed up mprotect(). ]
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Mel Gorman <mgorman@suse.de>
      Cc: Hugh Dickins <hughd@google.com>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
    • Peter Zijlstra's avatar
      mm: Count the number of pages affected in change_protection() · 7da4d641
      Peter Zijlstra authored
      This will be used for three kinds of purposes:
       - to optimize mprotect()
       - to speed up working set scanning for working set areas that
         have not been touched
       - to more accurately scan per real working set
      No change in functionality from this patch.
      Suggested-by: default avatarIngo Molnar <mingo@kernel.org>
      Signed-off-by: default avatarPeter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Linus Torvalds <torvalds@linux-foundation.org>
      Cc: Andrew Morton <akpm@linux-foundation.org>
      Cc: Andrea Arcangeli <aarcange@redhat.com>
      Cc: Rik van Riel <riel@redhat.com>
      Cc: Mel Gorman <mgorman@suse.de>
      Cc: Hugh Dickins <hughd@google.com>
      Cc: Thomas Gleixner <tglx@linutronix.de>
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
  16. 21 Mar, 2012 1 commit
  17. 06 Mar, 2012 1 commit
    • Linus Torvalds's avatar
      vm: avoid using find_vma_prev() unnecessarily · 097d5910
      Linus Torvalds authored
      Several users of "find_vma_prev()" were not in fact interested in the
      previous vma if there was no primary vma to be found either.  And in
      those cases, we're much better off just using the regular "find_vma()",
      and then "prev" can be looked up by just checking vma->vm_prev.
      The find_vma_prev() semantics are fairly subtle (see Mikulas' recent
      commit 83cd904d: "mm: fix find_vma_prev"), and the whole "return
      prev by reference" means that it generates worse code too.
      Thus this "let's avoid using this inconvenient and clearly too subtle
      interface when we don't really have to" patch.
      Cc: Mikulas Patocka <mpatocka@redhat.com>
      Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
  18. 13 Feb, 2012 1 commit
  19. 13 Jan, 2011 3 commits
  20. 09 Nov, 2010 1 commit