1. 14 Jan, 2016 3 commits
    • Jerome Marchand's avatar
      mm, procfs: breakdown RSS for anon, shmem and file in /proc/pid/status · 8cee852e
      Jerome Marchand authored
      
      
      There are several shortcomings with the accounting of shared memory
      (SysV shm, shared anonymous mapping, mapping of a tmpfs file).  The
      values in /proc/<pid>/status and <...>/statm don't allow to distinguish
      between shmem memory and a shared mapping to a regular file, even though
      theirs implication on memory usage are quite different: during reclaim,
      file mapping can be dropped or written back on disk, while shmem needs a
      place in swap.
      
      Also, to distinguish the memory occupied by anonymous and file mappings,
      one has to read the /proc/pid/statm file, which has a field for the file
      mappings (again, including shmem) and total memory occupied by these
      mappings (i.e.  equivalent to VmRSS in the <...>/status file.  Getting
      the value for anonymous mappings only is thus not exactly user-friendly
      (the statm file is intended to be rather efficiently machine-readable).
      
      To address both of these shortcomings, this patch adds a breakdown of
      VmRSS in /proc/<pid>/status via new fields RssAnon, RssFile and
      RssShmem, making use of the previous preparatory patch.  These fields
      tell the user the memory occupied by private anonymous pages, mapped
      regular files and shmem, respectively.  Other existing fields in /status
      and /statm files are left without change.  The /statm file can be
      extended in the future, if there's a need for that.
      
      Example (part of) /proc/pid/status output including the new Rss* fields:
      
      VmPeak:  2001008 kB
      VmSize:  2001004 kB
      VmLck:         0 kB
      VmPin:         0 kB
      VmHWM:      5108 kB
      VmRSS:      5108 kB
      RssAnon:              92 kB
      RssFile:            1324 kB
      RssShmem:           3692 kB
      VmData:      192 kB
      VmStk:       136 kB
      VmExe:         4 kB
      VmLib:      1784 kB
      VmPTE:      3928 kB
      VmPMD:        20 kB
      VmSwap:        0 kB
      HugetlbPages:          0 kB
      
      [vbabka@suse.cz: forward-porting, tweak changelog]
      Signed-off-by: default avatarJerome Marchand <jmarchan@redhat.com>
      Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Acked-by: default avatarKonstantin Khlebnikov <khlebnikov@yandex-team.ru>
      Acked-by: default avatarMichal Hocko <mhocko@suse.com>
      Acked-by: default avatarHugh Dickins <hughd@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      8cee852e
    • Vlastimil Babka's avatar
      mm, proc: account for shmem swap in /proc/pid/smaps · c261e7d9
      Vlastimil Babka authored
      
      
      Currently, /proc/pid/smaps will always show "Swap: 0 kB" for
      shmem-backed mappings, even if the mapped portion does contain pages
      that were swapped out.  This is because unlike private anonymous
      mappings, shmem does not change pte to swap entry, but pte_none when
      swapping the page out.  In the smaps page walk, such page thus looks
      like it was never faulted in.
      
      This patch changes smaps_pte_entry() to determine the swap status for
      such pte_none entries for shmem mappings, similarly to how
      mincore_page() does it.  Swapped out shmem pages are thus accounted for.
      For private mappings of tmpfs files that COWed some of the pages, swaped
      out status of the original shmem pages is naturally ignored.  If some of
      the private copies was also swapped out, they are accounted via their
      page table swap entries, so the resulting reported swap usage is then a
      sum of both swapped out private copies, and swapped out shmem pages that
      were not COWed.  No double accounting can thus happen.
      
      The accounting is arguably still not as precise as for private anonymous
      mappings, since now we will count also pages that the process in
      question never accessed, but another process populated them and then let
      them become swapped out.  I believe it is still less confusing and
      subtle than not showing any swap usage by shmem mappings at all.
      Swapped out counter might of interest of users who would like to prevent
      from future swapins during performance critical operation and pre-fault
      them at their convenience.  Especially for larger swapped out regions
      the cost of swapin is much higher than a fresh page allocation.  So a
      differentiation between pte_none vs.  swapped out is important for those
      usecases.
      
      One downside of this patch is that it makes /proc/pid/smaps more
      expensive for shmem mappings, as we consult the radix tree for each
      pte_none entry, so the overal complexity is O(n*log(n)).  I have
      measured this on a process that creates a 2GB mapping and dirties single
      pages with a stride of 2MB, and time how long does it take to cat
      /proc/pid/smaps of this process 100 times.
      
      Private anonymous mapping:
      
      real    0m0.949s
      user    0m0.116s
      sys     0m0.348s
      
      Mapping of a /dev/shm/file:
      
      real    0m3.831s
      user    0m0.180s
      sys     0m3.212s
      
      The difference is rather substantial, so the next patch will reduce the
      cost for shared or read-only mappings.
      
      In a less controlled experiment, I've gathered pids of processes on my
      desktop that have either '/dev/shm/*' or 'SYSV*' in smaps.  This
      included the Chrome browser and some KDE processes.  Again, I've run cat
      /proc/pid/smaps on each 100 times.
      
      Before this patch:
      
      real    0m9.050s
      user    0m0.518s
      sys     0m8.066s
      
      After this patch:
      
      real    0m9.221s
      user    0m0.541s
      sys     0m8.187s
      
      This suggests low impact on average systems.
      
      Note that this patch doesn't attempt to adjust the SwapPss field for
      shmem mappings, which would need extra work to determine who else could
      have the pages mapped.  Thus the value stays zero except for COWed
      swapped out pages in a shmem mapping, which are accounted as usual.
      Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Acked-by: default avatarKonstantin Khlebnikov <khlebnikov@yandex-team.ru>
      Acked-by: default avatarJerome Marchand <jmarchan@redhat.com>
      Acked-by: default avatarMichal Hocko <mhocko@suse.com>
      Cc: Hugh Dickins <hughd@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      c261e7d9
    • Vlastimil Babka's avatar
      mm, documentation: clarify /proc/pid/status VmSwap limitations for shmem · bf9683d6
      Vlastimil Babka authored
      This series is based on Jerome Marchand's [1] so let me quote the first
      paragraph from there:
      
      There are several shortcomings with the accounting of shared memory
      (sysV shm, shared anonymous mapping, mapping to a tmpfs file).  The
      values in /proc/<pid>/status and statm don't allow to distinguish
      between shmem memory and a shared mapping to a regular file, even though
      their implications on memory usage are quite different: at reclaim, file
      mapping can be dropped or written back on disk while shmem needs a place
      in swap.  As for shmem pages that are swapped-out or in swap cache, they
      aren't accounted at all.
      
      The original motivation for myself is that a customer found (IMHO
      rightfully) confusing that e.g.  top output for process swap usage is
      unreliable with respect to swapped out shmem pages, which are not
      accounted for.
      
      The fundamental difference between private anonymous and shmem pages is
      that the latter has PTE's converted to pte_none, and not swapents.  As
      such, they are not accounted to the number of swapents visible e.g.  in
      /proc/pid/status VmSwap row.  It might be theoretically possible to use
      swapents when swapping out shmem (without extra cost, as one has to
      change all mappers anyway), and on swap in only convert the swapent for
      the faulting process, leaving swapents in other processes until they
      also fault (so again no extra cost).  But I don't know how many
      assumptions this would break, and it would be too disruptive change for
      a relatively small benefit.
      
      Instead, my approach is to document the limitation of VmSwap, and
      provide means to determine the swap usage for shmem areas for those who
      are interested and willing to pay the price, using /proc/pid/smaps.
      Because outside of ipcs, I don't think it's possible to currently to
      determine the usage at all.  The previous patchset [1] did introduce new
      shmem-specific fields into smaps output, and functions to determine the
      values.  I take a simpler approach, noting that smaps output already has
      a "Swap: X kB" line, where currently X == 0 always for shmem areas.  I
      think we can just consider this a bug and provide the proper value by
      consulting the radix tree, as e.g.  mincore_page() does.  In the patch
      changelog I explain why this is also not perfect (and cannot be without
      swapents), but still arguably much better than showing a 0.
      
      The last two patches are adapted from Jerome's patchset and provide a
      VmRSS breakdown to RssAnon, RssFile and RssShm in /proc/pid/status.
      Hugh noted that this is a welcome addition, and I agree that it might
      help e.g.  debugging process memory usage at albeit non-zero, but still
      rather low cost of extra per-mm counter and some page flag checks.
      
      [1] http://lwn.net/Articles/611966/
      
      
      
      This patch (of 6):
      
      The documentation for /proc/pid/status does not mention that the value
      of VmSwap counts only swapped out anonymous private pages, and not
      swapped out pages of the underlying shmem objects (for shmem mappings).
      This is not obvious, so document this limitation.
      Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
      Acked-by: default avatarKonstantin Khlebnikov <khlebnikov@yandex-team.ru>
      Acked-by: default avatarMichal Hocko <mhocko@suse.com>
      Acked-by: default avatarJerome Marchand <jmarchan@redhat.com>
      Acked-by: default avatarHugh Dickins <hughd@google.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      bf9683d6
  2. 04 Jan, 2016 1 commit
    • Pantelis Antoniou's avatar
      configfs: implement binary attributes · 03607ace
      Pantelis Antoniou authored
      
      
      ConfigFS lacked binary attributes up until now. This patch
      introduces support for binary attributes in a somewhat similar
      manner of sysfs binary attributes albeit with changes that
      fit the configfs usage model.
      
      Problems that configfs binary attributes fix are everything that
      requires a binary blob as part of the configuration of a resource,
      such as bitstream loading for FPGAs, DTBs for dynamically created
      devices etc.
      
      Look at Documentation/filesystems/configfs/configfs.txt for internals
      and howto use them.
      
      This patch is against linux-next as of today that contains
      Christoph's configfs rework.
      Signed-off-by: default avatarPantelis Antoniou <pantelis.antoniou@konsulko.com>
      [hch: folded a fix from Geert Uytterhoeven <geert+renesas@glider.be>]
      [hch: a few tiny updates based on review feedback]
      Signed-off-by: default avatarChristoph Hellwig <hch@lst.de>
      03607ace
  3. 30 Dec, 2015 1 commit
  4. 16 Dec, 2015 1 commit
  5. 08 Dec, 2015 2 commits
    • Al Viro's avatar
      replace ->follow_link() with new method that could stay in RCU mode · 6b255391
      Al Viro authored
      
      
      new method: ->get_link(); replacement of ->follow_link().  The differences
      are:
      	* inode and dentry are passed separately
      	* might be called both in RCU and non-RCU mode;
      the former is indicated by passing it a NULL dentry.
      	* when called that way it isn't allowed to block
      and should return ERR_PTR(-ECHILD) if it needs to be called
      in non-RCU mode.
      
      It's a flagday change - the old method is gone, all in-tree instances
      converted.  Conversion isn't hard; said that, so far very few instances
      do not immediately bail out when called in RCU mode.  That'll change
      in the next commits.
      Signed-off-by: default avatarAl Viro <viro@zeniv.linux.org.uk>
      6b255391
    • Al Viro's avatar
      don't put symlink bodies in pagecache into highmem · 21fc61c7
      Al Viro authored
      
      
      kmap() in page_follow_link_light() needed to go - allowing to hold
      an arbitrary number of kmaps for long is a great way to deadlocking
      the system.
      
      new helper (inode_nohighmem(inode)) needs to be used for pagecache
      symlinks inodes; done for all in-tree cases.  page_follow_link_light()
      instrumented to yell about anything missed.
      Signed-off-by: default avatarAl Viro <viro@zeniv.linux.org.uk>
      21fc61c7
  6. 04 Dec, 2015 1 commit
  7. 11 Nov, 2015 1 commit
  8. 09 Nov, 2015 1 commit
    • Ross Zwisler's avatar
      coredump: add DAX filtering for ELF coredumps · 5037835c
      Ross Zwisler authored
      
      
      Add two new flags to the existing coredump mechanism for ELF files to
      allow us to explicitly filter DAX mappings.  This is desirable because
      DAX mappings, like hugetlb mappings, have the potential to be very
      large.
      
      Update the coredump_filter documentation in
      Documentation/filesystems/proc.txt so that it addresses the new DAX
      coredump flags.  Also update the documented default value of
      coredump_filter to be consistent with the core(5) man page.  The
      documentation being updated talks about bit 4, Dump ELF headers, which
      is enabled if CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is turned on in the
      kernel config.  This kernel config option defaults to "y" if both ELF
      binaries and coredump are enabled.
      Signed-off-by: default avatarRoss Zwisler <ross.zwisler@linux.intel.com>
      Acked-by: default avatarJeff Moyer <jmoyer@redhat.com>
      Signed-off-by: default avatarDan Williams <dan.j.williams@intel.com>
      5037835c
  9. 05 Nov, 2015 4 commits
  10. 02 Nov, 2015 1 commit
  11. 29 Oct, 2015 1 commit
  12. 18 Oct, 2015 1 commit
    • Li RongQing's avatar
      ipconfig: send Client-identifier in DHCP requests · 26fb342c
      Li RongQing authored
      
      
      A dhcp server may provide parameters to a client from a pool of IP
      addresses and using a shared rootfs, or provide a specific set of
      parameters for a specific client, usually using the MAC address to
      identify each client individually. The dhcp protocol also specifies
      a client-id field which can be used to determine the correct
      parameters to supply when no MAC address is available. There is
      currently no way to tell the kernel to supply a specific client-id,
      only the userspace dhcp clients support this feature, but this can
      not be used when the network is needed before userspace is available
      such as when the root filesystem is on NFS.
      
      This patch is to be able to do something like "ip=dhcp,client_id_type,
      client_id_value", as a kernel parameter to enable the kernel to
      identify itself to the server.
      Signed-off-by: default avatarLi RongQing <roy.qing.li@gmail.com>
      Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
      26fb342c
  13. 13 Oct, 2015 1 commit
  14. 09 Oct, 2015 1 commit
  15. 04 Oct, 2015 1 commit
  16. 01 Oct, 2015 1 commit
    • Ingo Molnar's avatar
      fs/proc, core/debug: Don't expose absolute kernel addresses via wchan · b2f73922
      Ingo Molnar authored
      
      
      So the /proc/PID/stat 'wchan' field (the 30th field, which contains
      the absolute kernel address of the kernel function a task is blocked in)
      leaks absolute kernel addresses to unprivileged user-space:
      
              seq_put_decimal_ull(m, ' ', wchan);
      
      The absolute address might also leak via /proc/PID/wchan as well, if
      KALLSYMS is turned off or if the symbol lookup fails for some reason:
      
      static int proc_pid_wchan(struct seq_file *m, struct pid_namespace *ns,
                                struct pid *pid, struct task_struct *task)
      {
              unsigned long wchan;
              char symname[KSYM_NAME_LEN];
      
              wchan = get_wchan(task);
      
              if (lookup_symbol_name(wchan, symname) < 0) {
                      if (!ptrace_may_access(task, PTRACE_MODE_READ))
                              return 0;
                      seq_printf(m, "%lu", wchan);
              } else {
                      seq_printf(m, "%s", symname);
              }
      
              return 0;
      }
      
      This isn't ideal, because for example it trivially leaks the KASLR offset
      to any local attacker:
      
        fomalhaut:~> printf "%016lx\n" $(cat /proc/$$/stat | cut -d' ' -f35)
        ffffffff8123b380
      
      Most real-life uses of wchan are symbolic:
      
        ps -eo pid:10,tid:10,wchan:30,comm
      
      and procps uses /proc/PID/wchan, not the absolute address in /proc/PID/stat:
      
        triton:~/tip> strace -f ps -eo pid:10,tid:10,wchan:30,comm 2>&1 | grep wchan | tail -1
        open("/proc/30833/wchan", O_RDONLY)     = 6
      
      There's one compatibility quirk here: procps relies on whether the
      absolute value is non-zero - and we can provide that functionality
      by outputing "0" or "1" depending on whether the task is blocked
      (whether there's a wchan address).
      
      These days there appears to be very little legitimate reason
      user-space would be interested in  the absolute address. The
      absolute address is mostly historic: from the days when we
      didn't have kallsyms and user-space procps had to do the
      decoding itself via the System.map.
      
      So this patch sets all numeric output to "0" or "1" and keeps only
      symbolic output, in /proc/PID/wchan.
      
      ( The absolute sleep address can generally still be profiled via
        perf, by tasks with sufficient privileges. )
      Reviewed-by: default avatarThomas Gleixner <tglx@linutronix.de>
      Acked-by: default avatarKees Cook <keescook@chromium.org>
      Acked-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      Cc: <stable@vger.kernel.org>
      Cc: Al Viro <viro@zeniv.linux.org.uk>
      Cc: Alexander Potapenko <glider@google.com>
      Cc: Andrey Konovalov <andreyknvl@google.com>
      Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
      Cc: Andy Lutomirski <luto@amacapital.net>
      Cc: Andy Lutomirski <luto@kernel.org>
      Cc: Borislav Petkov <bp@alien8.de>
      Cc: Denys Vlasenko <dvlasenk@redhat.com>
      Cc: Dmitry Vyukov <dvyukov@google.com>
      Cc: Kostya Serebryany <kcc@google.com>
      Cc: Mike Galbraith <efault@gmx.de>
      Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
      Cc: Peter Zijlstra <peterz@infradead.org>
      Cc: Sasha Levin <sasha.levin@oracle.com>
      Cc: kasan-dev <kasan-dev@googlegroups.com>
      Cc: linux-kernel@vger.kernel.org
      Link: http://lkml.kernel.org/r/20150930135917.GA3285@gmail.com
      
      Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
      b2f73922
  17. 13 Sep, 2015 3 commits
  18. 08 Sep, 2015 2 commits
    • Minchan Kim's avatar
      mm: /proc/pid/smaps:: show proportional swap share of the mapping · 8334b962
      Minchan Kim authored
      
      
      We want to know per-process workingset size for smart memory management
      on userland and we use swap(ex, zram) heavily to maximize memory
      efficiency so workingset includes swap as well as RSS.
      
      On such system, if there are lots of shared anonymous pages, it's really
      hard to figure out exactly how many each process consumes memory(ie, rss
      + wap) if the system has lots of shared anonymous memory(e.g, android).
      
      This patch introduces SwapPss field on /proc/<pid>/smaps so we can get
      more exact workingset size per process.
      
      Bongkyu tested it. Result is below.
      
      1. 50M used swap
      SwapTotal: 461976 kB
      SwapFree: 411192 kB
      
      $ adb shell cat /proc/*/smaps | grep "SwapPss:" | awk '{sum += $2} END {print sum}';
      48236
      $ adb shell cat /proc/*/smaps | grep "Swap:" | awk '{sum += $2} END {print sum}';
      141184
      
      2. 240M used swap
      SwapTotal: 461976 kB
      SwapFree: 216808 kB
      
      $ adb shell cat /proc/*/smaps | grep "SwapPss:" | awk '{sum += $2} END {print sum}';
      230315
      $ adb shell cat /proc/*/smaps | grep "Swap:" | awk '{sum += $2} END {print sum}';
      1387744
      
      [akpm@linux-foundation.org: simplify kunmap_atomic() call]
      Signed-off-by: default avatarMinchan Kim <minchan@kernel.org>
      Reported-by: default avatarBongkyu Kim <bongkyu.kim@lge.com>
      Tested-by: default avatarBongkyu Kim <bongkyu.kim@lge.com>
      Cc: Hugh Dickins <hughd@google.com>
      Cc: Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com>
      Cc: Jonathan Corbet <corbet@lwn.net>
      Cc: Jerome Marchand <jmarchan@redhat.com>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      8334b962
    • Matthew Wilcox's avatar
      dax: add huge page fault support · 844f35db
      Matthew Wilcox authored
      
      
      This is the support code for DAX-enabled filesystems to allow them to
      provide huge pages in response to faults.
      Signed-off-by: default avatarMatthew Wilcox <willy@linux.intel.com>
      Cc: Hillf Danton <dhillf@gmail.com>
      Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
      Cc: Theodore Ts'o <tytso@mit.edu>
      Cc: Jan Kara <jack@suse.cz>
      Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
      Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
      844f35db
  19. 20 Aug, 2015 1 commit
  20. 05 Aug, 2015 1 commit
  21. 04 Aug, 2015 1 commit
  22. 24 Jul, 2015 1 commit
  23. 23 Jul, 2015 1 commit
    • Jan Kara's avatar
      fs: Remove ext3 filesystem driver · c290ea01
      Jan Kara authored
      
      
      The functionality of ext3 is fully supported by ext4 driver. Major
      distributions (SUSE, RedHat) already use ext4 driver to handle ext3
      filesystems for quite some time. There is some ugliness in mm resulting
      from jbd cleaning buffers in a dirty page without cleaning page dirty
      bit and also support for buffer bouncing in the block layer when stable
      pages are required is there only because of jbd. So let's remove the
      ext3 driver. This saves us some 28k lines of duplicated code.
      Acked-by: default avatarTheodore Ts'o <tytso@mit.edu>
      Signed-off-by: default avatarJan Kara <jack@suse.cz>
      c290ea01
  24. 20 Jul, 2015 1 commit
  25. 09 Jul, 2015 1 commit
  26. 04 Jul, 2015 1 commit
  27. 01 Jul, 2015 1 commit
  28. 08 Jun, 2015 1 commit
  29. 04 Jun, 2015 1 commit
  30. 31 May, 2015 1 commit
  31. 18 May, 2015 1 commit