Commit 4aed2fd8 authored by Linus Torvalds's avatar Linus Torvalds

Merge branch 'perf-core-for-linus' of...

Merge branch 'perf-core-for-linus' of git://

* 'perf-core-for-linus' of git:// (162 commits)
  tracing/kprobes: unregister_trace_probe needs to be called under mutex
  perf: expose event__process function
  perf events: Fix mmap offset determination
  perf, powerpc: fsl_emb: Restore setting perf_sample_data.period
  perf, powerpc: Convert the FSL driver to use local64_t
  perf tools: Don't keep unreferenced maps when unmaps are detected
  perf session: Invalidate last_match when removing threads from rb_tree
  perf session: Free the ref_reloc_sym memory at the right place
  x86,mmiotrace: Add support for tracing STOS instruction
  perf, sched migration: Librarize task states and event headers helpers
  perf, sched migration: Librarize the GUI class
  perf, sched migration: Make the GUI class client agnostic
  perf, sched migration: Make it vertically scrollable
  perf, sched migration: Parameterize cpu height and spacing
  perf, sched migration: Fix key bindings
  perf, sched migration: Ignore unhandled task states
  perf, sched migration: Handle ignored migrate out events
  perf: New migration tool overview
  tracing: Drop cpparg() macro
  perf: Use tracepoint_synchronize_unregister() to flush any pending tracepoint call

Fix up trivial conflicts in Makefile and drivers/cpufreq/cpufreq.c
parents 3a3527b6 fc9ea5a1
What: /sys/kernel/debug/kmemtrace/
Date: July 2008
Contact: Eduard - Gabriel Munteanu <>
In kmemtrace-enabled kernels, the following files are created:
cpu<n> (0400) Per-CPU tracing data, see below. (binary)
total_overruns (0400) Total number of bytes which were dropped from
cpu<n> files because of full buffer condition,
non-binary. (text)
abi_version (0400) Kernel's kmemtrace ABI version. (text)
Each per-CPU file should be read according to the relay interface. That is,
the reader should set affinity to that specific CPU and, as currently done by
the userspace application (though there are other methods), use poll() with
an infinite timeout before every read(). Otherwise, erroneous data may be
read. The binary data has the following _core_ format:
Event ID (1 byte) Unsigned integer, one of:
0 - represents an allocation (KMEMTRACE_EVENT_ALLOC)
1 - represents a freeing of previously allocated memory
Type ID (1 byte) Unsigned integer, one of:
0 - this is a kmalloc() / kfree()
1 - this is a kmem_cache_alloc() / kmem_cache_free()
2 - this is a __get_free_pages() et al.
Event size (2 bytes) Unsigned integer representing the
size of this event. Used to extend
kmemtrace. Discard the bytes you
don't know about.
Sequence number (4 bytes) Signed integer used to reorder data
logged on SMP machines. Wraparound
must be taken into account, although
it is unlikely.
Caller address (8 bytes) Return address to the caller.
Pointer to mem (8 bytes) Pointer to target memory area. Can be
NULL, but not all such calls might be
In case of KMEMTRACE_EVENT_ALLOC events, the next fields follow:
Requested bytes (8 bytes) Total number of requested bytes,
unsigned, must not be zero.
Allocated bytes (8 bytes) Total number of actually allocated
bytes, unsigned, must not be lower
than requested bytes.
Requested flags (4 bytes) GFP flags supplied by the caller.
Target CPU (4 bytes) Signed integer, valid for event id 1.
If equal to -1, target CPU is the same
as origin CPU, but the reverse might
not be true.
The data is made available in the same endianness the machine has.
Other event ids and type ids may be defined and added. Other fields may be
added by increasing event size, but see below for details.
Every modification to the ABI, including new id definitions, are followed
by bumping the ABI version by one.
Adding new data to the packet (features) is done at the end of the mandatory
Feature size (2 byte)
Feature ID (1 byte)
Feature data (Feature size - 3 bytes)
kmemtrace-user - git://
......@@ -1816,6 +1816,8 @@ and is between 256 and 4096 characters. It is defined in the file
nousb [USB] Disable the USB subsystem
nowatchdog [KNL] Disable the lockup detector.
nowb [ARM]
nox2apic [X86-64,APIC] Do not enable x2APIC mode.
......@@ -13,6 +13,9 @@ Note that this focuses on architecture implementation details only. If you
want more explanation of a feature in terms of common code, review the common
ftrace.txt file.
Ideally, everyone who wishes to retain performance while supporting tracing in
their kernel should make it all the way to dynamic ftrace support.
......@@ -215,7 +218,7 @@ An arch may pass in a unique value (frame pointer) to both the entering and
exiting of a function. On exit, the value is compared and if it does not
match, then it will panic the kernel. This is largely a sanity check for bad
code generation with gcc. If gcc for your port sanely updates the frame
pointer under different opitmization levels, then ignore this option.
pointer under different optimization levels, then ignore this option.
However, adding support for it isn't terribly difficult. In your assembly code
that calls prepare_ftrace_return(), pass the frame pointer as the 3rd argument.
......@@ -234,7 +237,7 @@ If you can't trace NMI functions, then skip this option.
You need very few things to get the syscalls tracing in an arch.
......@@ -250,12 +253,152 @@ You need very few things to get the syscalls tracing in an arch.
See scripts/ for more info.
See scripts/ for more info. Just fill in the arch-specific
details for how to locate the addresses of mcount call sites via objdump.
This option doesn't make much sense without also implementing dynamic ftrace.
scroll your reader back up if you got over eager.
Once those are out of the way, you will need to implement:
- asm/ftrace.h:
- ftrace_call_adjust()
- struct dyn_arch_ftrace{}
- asm code:
- mcount() (new stub)
- ftrace_caller()
- ftrace_call()
- ftrace_stub()
- C code:
- ftrace_dyn_arch_init()
- ftrace_make_nop()
- ftrace_make_call()
- ftrace_update_ftrace_func()
First you will need to fill out some arch details in your asm/ftrace.h.
Define MCOUNT_ADDR as the address of your mcount symbol similar to:
#define MCOUNT_ADDR ((unsigned long)mcount)
Since no one else will have a decl for that function, you will need to:
extern void mcount(void);
You will also need the helper function ftrace_call_adjust(). Most people
will be able to stub it out like so:
static inline unsigned long ftrace_call_adjust(unsigned long addr)
return addr;
<details to be filled>
Lastly you will need the custom dyn_arch_ftrace structure. If you need
some extra state when runtime patching arbitrary call sites, this is the
place. For now though, create an empty struct:
struct dyn_arch_ftrace {
/* No extra data needed */
With the header out of the way, we can fill out the assembly code. While we
did already create a mcount() function earlier, dynamic ftrace only wants a
stub function. This is because the mcount() will only be used during boot
and then all references to it will be patched out never to return. Instead,
the guts of the old mcount() will be used to create a new ftrace_caller()
function. Because the two are hard to merge, it will most likely be a lot
easier to have two separate definitions split up by #ifdefs. Same goes for
the ftrace_stub() as that will now be inlined in ftrace_caller().
Before we get confused anymore, let's check out some pseudo code so you can
implement your own stuff in assembly:
void mcount(void)
void ftrace_caller(void)
/* implement HAVE_FUNCTION_TRACE_MCOUNT_TEST if you desire */
/* save all state needed by the ABI (see paragraph above) */
unsigned long frompc = ...;
unsigned long selfpc = <return address> - MCOUNT_INSN_SIZE;
ftrace_stub(frompc, selfpc);
/* restore all state needed by the ABI */
This might look a little odd at first, but keep in mind that we will be runtime
patching multiple things. First, only functions that we actually want to trace
will be patched to call ftrace_caller(). Second, since we only have one tracer
active at a time, we will patch the ftrace_caller() function itself to call the
specific tracer in question. That is the point of the ftrace_call label.
With that in mind, let's move on to the C code that will actually be doing the
runtime patching. You'll need a little knowledge of your arch's opcodes in
order to make it through the next section.
Every arch has an init callback function. If you need to do something early on
to initialize some state, this is the time to do that. Otherwise, this simple
function below should be sufficient for most people:
int __init ftrace_dyn_arch_init(void *data)
/* return value is done indirectly via data */
*(unsigned long *)data = 0;
return 0;
There are two functions that are used to do runtime patching of arbitrary
functions. The first is used to turn the mcount call site into a nop (which
is what helps us retain runtime performance when not tracing). The second is
used to turn the mcount call site into a call to an arbitrary location (but
typically that is ftracer_caller()). See the general function definition in
linux/ftrace.h for the functions:
The rec->ip value is the address of the mcount call site that was collected
by the scripts/ during build time.
The last function is used to do runtime patching of the active tracer. This
will be modifying the assembly code at the location of the ftrace_call symbol
inside of the ftrace_caller() function. So you should have sufficient padding
at that location to support the new function calls you'll be inserting. Some
people will be using a "call" type instruction while others will be using a
"branch" type instruction. Specifically, the function is:
The function grapher needs a few tweaks in order to work with dynamic ftrace.
Basically, you will need to:
- update:
- ftrace_caller()
- ftrace_graph_call()
- ftrace_graph_caller()
- implement:
- ftrace_enable_ftrace_graph_caller()
- ftrace_disable_ftrace_graph_caller()
<details to be filled>
Quick notes:
- add a nop stub after the ftrace_call location named ftrace_graph_call;
stub needs to be large enough to support a call to ftrace_graph_caller()
- update ftrace_graph_caller() to work with being called by the new
ftrace_caller() since some semantics may have changed
- ftrace_enable_ftrace_graph_caller() will runtime patch the
ftrace_graph_call location with a call to ftrace_graph_caller()
- ftrace_disable_ftrace_graph_caller() will runtime patch the
ftrace_graph_call location with nops
kmemtrace - Kernel Memory Tracer
by Eduard - Gabriel Munteanu
I. Introduction
kmemtrace helps kernel developers figure out two things:
1) how different allocators (SLAB, SLUB etc.) perform
2) how kernel code allocates memory and how much
To do this, we trace every allocation and export information to the userspace
through the relay interface. We export things such as the number of requested
bytes, the number of bytes actually allocated (i.e. including internal
fragmentation), whether this is a slab allocation or a plain kmalloc() and so
The actual analysis is performed by a userspace tool (see section III for
details on where to get it from). It logs the data exported by the kernel,
processes it and (as of writing this) can provide the following information:
- the total amount of memory allocated and fragmentation per call-site
- the amount of memory allocated and fragmentation per allocation
- total memory allocated and fragmentation in the collected dataset
- number of cross-CPU allocation and frees (makes sense in NUMA environments)
Moreover, it can potentially find inconsistent and erroneous behavior in
kernel code, such as using slab free functions on kmalloc'ed memory or
allocating less memory than requested (but not truly failed allocations).
kmemtrace also makes provisions for tracing on some arch and analysing the
data on another.
II. Design and goals
kmemtrace was designed to handle rather large amounts of data. Thus, it uses
the relay interface to export whatever is logged to userspace, which then
stores it. Analysis and reporting is done asynchronously, that is, after the
data is collected and stored. By design, it allows one to log and analyse
on different machines and different arches.
As of writing this, the ABI is not considered stable, though it might not
change much. However, no guarantees are made about compatibility yet. When
deemed stable, the ABI should still allow easy extension while maintaining
backward compatibility. This is described further in Documentation/ABI.
Summary of design goals:
- allow logging and analysis to be done across different machines
- be fast and anticipate usage in high-load environments (*)
- be reasonably extensible
- make it possible for GNU/Linux distributions to have kmemtrace
included in their repositories
(*) - one of the reasons Pekka Enberg's original userspace data analysis
tool's code was rewritten from Perl to C (although this is more than a
simple conversion)
III. Quick usage guide
1) Get a kernel that supports kmemtrace and build it accordingly (i.e. enable
2) Get the userspace tool and build it:
$ git clone git:// # current repository
$ cd kmemtrace-user/
$ ./
$ ./configure
$ make
3) Boot the kmemtrace-enabled kernel if you haven't, preferably in the
'single' runlevel (so that relay buffers don't fill up easily), and run
# '$' does not mean user, but root here.
$ mount -t debugfs none /sys/kernel/debug
$ mount -t proc none /proc
$ cd path/to/kmemtrace-user/
$ ./kmemtraced
Wait a bit, then stop it with CTRL+C.
$ cat /sys/kernel/debug/kmemtrace/total_overruns # Check if we didn't
# overrun, should
# be zero.
$ (Optionally) [Run kmemtrace_check separately on each cpu[0-9]*.out file to
check its correctness]
$ ./kmemtrace-report
Now you should have a nice and short summary of how the allocator performs.
IV. FAQ and known issues
Q: 'cat /sys/kernel/debug/kmemtrace/total_overruns' is non-zero, how do I fix
this? Should I worry?
A: If it's non-zero, this affects kmemtrace's accuracy, depending on how
large the number is. You can fix it by supplying a higher
'kmemtrace.subbufs=N' kernel parameter.
Q: kmemtrace_check reports errors, how do I fix this? Should I worry?
A: This is a bug and should be reported. It can occur for a variety of
- possible bugs in relay code
- possible misuse of relay by kmemtrace
- timestamps being collected unorderly
Or you may fix it yourself and send us a patch.
Q: kmemtrace_report shows many errors, how do I fix this? Should I worry?
A: This is a known issue and I'm working on it. These might be true errors
in kernel code, which may have inconsistent behavior (e.g. allocating memory
with kmem_cache_alloc() and freeing it with kfree()). Pekka Enberg pointed
out this behavior may work with SLAB, but may fail with other allocators.
It may also be due to lack of tracing in some unusual allocator functions.
We don't want bug reports regarding this issue yet.
V. See also
......@@ -42,7 +42,7 @@ Synopsis of kprobe_events
+|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(**)
NAME=FETCHARG : Set NAME as the argument name of FETCHARG.
FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
(u8/u16/u32/u64/s8/s16/s32/s64) are supported.
(u8/u16/u32/u64/s8/s16/s32/s64) and string are supported.
(*) only for return probe.
(**) this is useful for fetching a field of data structures.
......@@ -3403,13 +3403,6 @@ F: include/linux/kmemleak.h
F: mm/kmemleak.c
F: mm/kmemleak-test.c
M: Eduard - Gabriel Munteanu <>
S: Maintained
F: Documentation/trace/kmemtrace.txt
F: include/linux/kmemtrace.h
F: kernel/trace/kmemtrace.c
M: Ananth N Mavinakayanahalli <>
M: Anil S Keshavamurthy <>
......@@ -5685,7 +5678,7 @@ TRACING
M: Steven Rostedt <>
M: Frederic Weisbecker <>
M: Ingo Molnar <>
T: git git:// tracing/core
T: git git:// perf/core
S: Maintained
F: Documentation/trace/ftrace.txt
F: arch/*/*/*/ftrace.h
......@@ -420,7 +420,7 @@ endif
no-dot-config-targets := clean mrproper distclean \
cscope TAGS tags help %docs check% coccicheck \
include/linux/version.h headers_% \
kernelversion %src-pkg
config-targets := 0
mixed-targets := 0
......@@ -1168,6 +1168,8 @@ distclean: mrproper
# rpm target kept for backward compatibility
package-dir := $(srctree)/scripts/package
%src-pkg: FORCE
$(Q)$(MAKE) $(build)=$(package-dir) $@
%pkg: include/config/kernel.release FORCE
$(Q)$(MAKE) $(build)=$(package-dir) $@
rpm: include/config/kernel.release FORCE
......@@ -151,4 +151,11 @@ config HAVE_MIXED_BREAKPOINTS_REGS
System hardware can generate an NMI using the perf event
subsystem. Also has support for calculating CPU cycle events
to determine how many clock cycles in a given period.
source "kernel/gcov/Kconfig"
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
......@@ -164,20 +164,20 @@ armpmu_event_set_period(struct perf_event *event,
struct hw_perf_event *hwc,
int idx)
s64 left = atomic64_read(&hwc->period_left);
s64 left = local64_read(&hwc->period_left);
s64 period = hwc->sample_period;
int ret = 0;
if (unlikely(left <= -period)) {
left = period;
atomic64_set(&hwc->period_left, left);
local64_set(&hwc->period_left, left);
hwc->last_period = period;
ret = 1;
if (unlikely(left <= 0)) {
left += period;
atomic64_set(&hwc->period_left, left);
local64_set(&hwc->period_left, left);
hwc->last_period = period;
ret = 1;
......@@ -185,7 +185,7 @@ armpmu_event_set_period(struct perf_event *event,
if (left > (s64)armpmu->max_period)
left = armpmu->max_period;
atomic64_set(&hwc->prev_count, (u64)-left);
local64_set(&hwc->prev_count, (u64)-left);
armpmu->write_counter(idx, (u64)(-left) & 0xffffffff);
......@@ -204,18 +204,18 @@ armpmu_event_update(struct perf_event *event,
u64 delta;
prev_raw_count = atomic64_read(&hwc->prev_count);
prev_raw_count = local64_read(&hwc->prev_count);
new_raw_count = armpmu->read_counter(idx);
if (atomic64_cmpxchg(&hwc->prev_count, prev_raw_count,
if (local64_cmpxchg(&hwc->prev_count, prev_raw_count,
new_raw_count) != prev_raw_count)
goto again;
delta = (new_raw_count << shift) - (prev_raw_count << shift);
delta >>= shift;
atomic64_add(delta, &event->count);
atomic64_sub(delta, &hwc->period_left);
local64_add(delta, &event->count);
local64_sub(delta, &hwc->period_left);
return new_raw_count;
......@@ -478,7 +478,7 @@ __hw_perf_event_init(struct perf_event *event)
if (!hwc->sample_period) {
hwc->sample_period = armpmu->max_period;
hwc->last_period = hwc->sample_period;
atomic64_set(&hwc->period_left, hwc->sample_period);
local64_set(&hwc->period_left, hwc->sample_period);
err = 0;
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
#include <asm-generic/local64.h>
......@@ -21,3 +21,15 @@
#include <asm/perf_event_fsl_emb.h>
#include <asm/ptrace.h>
#include <asm/reg.h>
#define perf_arch_fetch_caller_regs(regs, __ip) \
do { \
(regs)->nip = __ip; \
(regs)->gpr[1] = *(unsigned long *)__get_SP(); \
asm volatile("mfmsr %0" : "=r" ((regs)->msr)); \
} while (0)
......@@ -127,29 +127,3 @@ _GLOBAL(__setup_cpu_power7)
/* place holder */
* Get a minimal set of registers for our caller's nth caller.
* r3 = regs pointer, r5 = n.
* We only get R1 (stack pointer), NIP (next instruction pointer)
* and LR (link register). These are all we can get in the
* general case without doing complicated stack unwinding, but
* fortunately they are enough to do a stack backtrace, which
* is all we need them for.
mr r6,r1
cmpwi r5,0
mflr r4
ble 2f
mtctr r5
1: PPC_LL r6,0(r6)
bdnz 1b
2: PPC_LL r7,0(r6)
......@@ -410,15 +410,15 @@ static void power_pmu_read(struct perf_event *event)
* Therefore we treat them like NMIs.
do {
prev = atomic64_read(&event->hw.prev_count);
prev = local64_read(&event->hw.prev_count);
val = read_pmc(event->hw.idx);
} while (atomic64_cmpxchg(&event->hw.prev_count, prev, val) != prev);
} while (local64_cmpxchg(&event->hw.prev_count, prev, val) != prev);
/* The counters are only 32 bits wide */
delta = (val - prev) & 0xfffffffful;
atomic64_add(delta, &event->count);
atomic64_sub(delta, &event->hw.period_left);
local64_add(delta, &event->count);
local64_sub(delta, &event->hw.period_left);
......@@ -444,10 +444,10 @@ static void freeze_limited_counters(struct cpu_hw_events *cpuhw,
if (!event->hw.idx)
val = (event->hw.idx == 5) ? pmc5 : pmc6;
prev = atomic64_read(&event->hw.prev_count);
prev = local64_read(&event->hw.prev_count);
event->hw.idx = 0;
delta = (val - prev) & 0xfffffffful;
atomic64_add(delta, &event->count);
local64_add(delta, &event->count);
......@@ -462,7 +462,7 @@ static void thaw_limited_counters(struct cpu_hw_events *cpuhw,
event = cpuhw->limited_counter[i];
event->hw.idx = cpuhw->limited_hwidx[i];
val = (event->hw.idx == 5) ? pmc5 : pmc6;
atomic64_set(&event->hw.prev_count, val);
local64_set(&event->hw.prev_count, val);
......@@ -666,11 +666,11 @@ void hw_perf_enable(void)
val = 0;
if (event->hw.sample_period) {
left = atomic64_read(&event->hw.period_left);
left = local64_read(&event->hw.period_left);
if (left < 0x80000000L)
val = 0x80000000L - left;
atomic64_set(&event->hw.prev_count, val);
local64_set(&event->hw.prev_count, val);
event->hw.idx = idx;
write_pmc(idx, val);
......@@ -754,7 +754,7 @@ static int power_pmu_enable(struct perf_event *event)
* skip the schedulability test here, it will be peformed
* at commit time(->commit_txn) as a whole
if (cpuhw->group_flag & PERF_EVENT_TXN_STARTED)
if (cpuhw->group_flag & PERF_EVENT_TXN)
goto nocheck;
if (check_excludes(cpuhw->event, cpuhw->flags, n0, 1))
......@@ -845,8 +845,8 @@ static void power_pmu_unthrottle(struct perf_event *event)