Skip to content
  • Masami Hiramatsu's avatar
    perf probe: Support @BUILDID or @FILE suffix for SDT events · a598180a
    Masami Hiramatsu authored
    
    
    Support @BUILDID or @FILE suffix for SDT events. This allows perf to add
    probes on SDTs/pre-cached events on given FILE or the file which has
    given BUILDID (also, this complements BUILDID.)
    
    For example, both gcc and libstdc++ has same SDTs as below.  If you
    would like to add a probe on sdt_libstdcxx:catch on gcc, you can do as
    below.
    
      ----
      # perf list sdt | tail -n 6
        sdt_libstdcxx:catch@/usr/bin/gcc(0cc207fc4b27)     [SDT event]
        sdt_libstdcxx:catch@/usr/lib64/libstdc++.so.6.0.20(91c7a88fdf49)
        sdt_libstdcxx:rethrow@/usr/bin/gcc(0cc207fc4b27)   [SDT event]
        sdt_libstdcxx:rethrow@/usr/lib64/libstdc++.so.6.0.20(91c7a88fdf49)
        sdt_libstdcxx:throw@/usr/bin/gcc(0cc207fc4b27)     [SDT event]
        sdt_libstdcxx:throw@/usr/lib64/libstdc++.so.6.0.20(91c7a88fdf49)
      # perf probe -a %sdt_libstdcxx:catch@0cc
      Added new event:
        sdt_libstdcxx:catch  (on %catch in /usr/bin/gcc)
    
      You can now use it in all perf tools, such as:
    
      	perf record -e sdt_libstdcxx:catch -aR sleep 1
      ----
    
    Committer note:
    
    Doing the full sequence of steps to get the results above:
    
    With a clean build-id cache:
    
      [root@jouet ~]# rm -rf ~/.debug/
      [root@jouet ~]# perf list sdt
    
      List of pre-defined events (to be used in -e):
    
      [root@jouet ~]#
    
    No events whatsoever, then, we can add all events in gcc to the build-id
    cache, doing a --add + --dry-run:
    
      [root@jouet ~]# perf probe --dry-run --cache -x /usr/bin/gcc --add %sdt_libstdcxx:\*
      Added new events:
        sdt_libstdcxx:throw  (on %* in /usr/bin/gcc)
        sdt_libstdcxx:rethrow (on %* in /usr/bin/gcc)
        sdt_libstdcxx:catch  (on %* in /usr/bin/gcc)
    
      You can now use it in all perf tools, such as:
    
    	perf record -e sdt_libstdcxx:catch -aR sleep 1
    
      [root@jouet ~]#
    
    It really didn't add any events, it just cached them:
    
      [root@jouet ~]# perf probe -l
      [root@jouet ~]#
    
    We can see that it was cached as:
    
      [root@jouet ~]# ls -la ~/.debug/usr/bin/gcc/9a0730e2bcc6d2a2003d21ac46807e8ee6bcb7c2/
      total 976
      drwxr-xr-x. 2 root root   4096 Jul 13 21:47 .
      drwxr-xr-x. 3 root root   4096 Jul 13 21:47 ..
      -rwxr-xr-x. 4 root root 985912 Jun 22 18:52 elf
      -rw-r--r--. 1 root root    303 Jul 13 21:47 probes
      [root@jouet ~]# file ~/.debug/usr/bin/gcc/9a0730e2bcc6d2a2003d21ac46807e8ee6bcb7c2/elf
      /root/.debug/usr/bin/gcc/9a0730e2bcc6d2a2003d21ac46807e8ee6bcb7c2/elf: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=9a0730e2bcc6d2a2003d21ac46807e8ee6bcb7c2, stripped
      [root@jouet ~]# cat ~/.debug/usr/bin/gcc/9a0730e2bcc6d2a2003d21ac46807e8ee6bcb7c2/probes
      %sdt_libstdcxx:throw=throw
      p:sdt_libstdcxx/throw /usr/bin/gcc:0x71ffd
      %sdt_libstdcxx:rethrow=rethrow
      p:sdt_libstdcxx/rethrow /usr/bin/gcc:0x720b8
      %sdt_libstdcxx:catch=catch
      p:sdt_libstdcxx/catch /usr/bin/gcc:0x7307f
      %sdt_libgcc:unwind=unwind
      p:sdt_libgcc/unwind /usr/bin/gcc:0x7eec0
      #sdt_libstdcxx:*=%*
      [root@jouet ~]#
    
    Ok, now we can use 'perf probe' to refer to those cached entries as:
    
      Humm, nope, doing as above we end up with:
    
      [root@jouet ~]# perf probe -a %sdt_libstdcxx:catch
      Semantic error :* is bad for event name -it must follow C symbol-naming rule.
        Error: Failed to add events.
      [root@jouet ~]#
    
    But it worked at some point, lets try not using --dry-run:
    
    Resetting everything:
    
      # rm -rf ~/.debug/
      # perf probe -d *:*
      # perf probe -l
      # perf list sdt
    
        List of pre-defined events (to be used in -e):
    
      #
    
    Ok, now it cached everything, even things we haven't asked it to
    (sdt_libgcc:unwind):
    
      [root@jouet ~]# perf probe -x /usr/bin/gcc --add %sdt_libstdcxx:\*
      Added new events:
        sdt_libstdcxx:throw  (on %* in /usr/bin/gcc)
        sdt_libstdcxx:rethrow (on %* in /usr/bin/gcc)
        sdt_libstdcxx:catch  (on %* in /usr/bin/gcc)
    
      You can now use it in all perf tools, such as:
    
    	perf record -e sdt_libstdcxx:catch -aR sleep 1
    
      [root@jouet ~]# perf list sdt
    
      List of pre-defined events (to be used in -e):
    
        sdt_libgcc:unwind                                  [SDT event]
        sdt_libstdcxx:catch                                [SDT event]
        sdt_libstdcxx:rethrow                              [SDT event]
        sdt_libstdcxx:throw                                [SDT event]
      [root@jouet ~]#
    
    And we have the events in place:
    
      [root@jouet ~]# perf probe -l
        sdt_libstdcxx:catch  (on execute_cfa_program+1551@../../../libgcc/unwind-dw2.c in /usr/bin/gcc)
        sdt_libstdcxx:rethrow (on d_print_subexpr+280@libsupc++/cp-demangle.c in /usr/bin/gcc)
        sdt_libstdcxx:throw  (on d_print_subexpr+93@libsupc++/cp-demangle.c in /usr/bin/gcc)
      [root@jouet ~]#
    
    And trying to use them at least has 'perf trace --event sdt*:*' working.
    
    Then, if we try to add the ones in libstdc++:
    
      [root@jouet ~]# perf probe -x /usr/lib64/libstdc++.so.6 -a %sdt_libstdcxx:\*
      Error: event "catch" already exists.
       Hint: Remove existing event by 'perf probe -d'
             or force duplicates by 'perf probe -f'
             or set 'force=yes' in BPF source.
        Error: Failed to add events.
      [root@jouet ~]#
    
    Doesn't work, dups, but at least this served to, unbeknownst to the user, add
    the SDT probes in /usr/lib64/libstdc++.so.6!
    
      [root@jouet ~]# perf list sdt
    
      List of pre-defined events (to be used in -e):
    
        sdt_libgcc:unwind                                  [SDT event]
        sdt_libstdcxx:catch@/usr/bin/gcc(9a0730e2bcc6)     [SDT event]
        sdt_libstdcxx:catch@/usr/lib64/libstdc++.so.6.0.22(ef2b7066559a) [SDT event]
        sdt_libstdcxx:rethrow@/usr/bin/gcc(9a0730e2bcc6)   [SDT event]
        sdt_libstdcxx:rethrow@/usr/lib64/libstdc++.so.6.0.22(ef2b7066559a) [SDT event]
        sdt_libstdcxx:throw@/usr/bin/gcc(9a0730e2bcc6)     [SDT event]
        sdt_libstdcxx:throw@/usr/lib64/libstdc++.so.6.0.22(ef2b7066559a) [SDT event]
      [root@jouet ~]#
    
    Now we should be able to get to the original cset comment, if we remove all
    SDTs events in place, not from the cache, from the kernel, where it was set up as:
    
      [root@jouet ~]# ls -la /sys/kernel/debug/tracing/events/sdt_libstdcxx/
      total 0
      drwxr-xr-x.  5 root root 0 Jul 13 22:00 .
      drwxr-xr-x. 80 root root 0 Jul 13 21:56 ..
      drwxr-xr-x.  2 root root 0 Jul 13 22:00 catch
      -rw-r--r--.  1 root root 0 Jul 13 22:00 enable
      -rw-r--r--.  1 root root 0 Jul 13 22:00 filter
      drwxr-xr-x.  2 root root 0 Jul 13 22:00 rethrow
      drwxr-xr-x.  2 root root 0 Jul 13 22:00 throw
      [root@jouet ~]#
    
      [root@jouet ~]# head -2 /sys/kernel/debug/tracing/events/sdt_libstdcxx/throw/format
      name: throw
      ID: 2059
      [root@jouet ~]#
    
    Now to remove it:
    
      [root@jouet ~]# perf probe -d sdt_libstdc*:*
      Removed event: sdt_libstdcxx:catch
      Removed event: sdt_libstdcxx:rethrow
      Removed event: sdt_libstdcxx:throw
      [root@jouet ~]#
    
    Which caused:
    
      [root@jouet ~]# ls -la /sys/kernel/debug/tracing/events/sdt_libstdcxx/
      ls: cannot access '/sys/kernel/debug/tracing/events/sdt_libstdcxx/': No such file or directory
      [root@jouet ~]#
    
    Ok, now we can do:
    
      [root@jouet ~]# perf list sdt_libstdcxx:catch
    
      List of pre-defined events (to be used in -e):
    
        sdt_libstdcxx:catch@/usr/bin/gcc(9a0730e2bcc6)     [SDT event]
        sdt_libstdcxx:catch@/usr/lib64/libstdc++.so.6.0.22(ef2b7066559a) [SDT event]
      [root@jouet ~]#
    
    So, these are not really 'pre-defined events', i.e. we can't use them with
    'perf record --event':
    
      [root@jouet ~]# perf record --event sdt_libstdcxx:catch*
      event syntax error: 'sdt_libstdcxx:catch*'
                           \___ unknown tracepoint
    
      Error:	File /sys/kernel/debug/tracing/events/sdt_libstdcxx/catch* not found.
      Hint:	Perhaps this kernel misses some CONFIG_ setting to enable this feature?.
    <SNIP>
      [root@jouet ~]#
    
    To have it really pre-defined we must use perf probe to get its definition from
    the cache and set it up in the kernel, creating the tracepoint to _then_ use it
    with 'perf record --event':
    
      [root@jouet ~]# perf probe -a sdt_libstdcxx:catch
      Semantic error :There is non-digit char in line number.
      <SNIP>
    
    Oops, there is another gotcha here, we need that pesky '%' character:
    
      [root@jouet ~]# perf probe -a %sdt_libstdcxx:catch
      Added new events:
        sdt_libstdcxx:catch  (on %catch in /usr/bin/gcc)
        sdt_libstdcxx:catch_1 (on %catch in /usr/lib64/libstdc++.so.6.0.22)
    
      You can now use it in all perf tools, such as:
    
    	perf record -e sdt_libstdcxx:catch_1 -aR sleep 1
    
      [root@jouet ~]#
    
    But then we added _two_ events, one with the name we expected, the other one
    with a _ added, when doing the analysis we need to pay attention to who maps to
    who.
    
    And here is where we get to the point of this patch, which is to be able to
    disambiguate those definitions for 'catch' in the build-id cache, but first we need
    remove those events we just added:
    
    [root@jouet ~]# perf probe -d %sdt_libstdcxx:catch
    
    Oops, that didn't remove anything, we need to _remove_ that % char in this case:
    
      [root@jouet ~]# perf probe -d sdt_libstdcxx:catch
      Removed event: sdt_libstdcxx:catch
    
    And we need to remove the other event added, i.e. I forgot to add a * at the end:
    
      [root@jouet ~]# perf probe -d sdt_libstdcxx:catch*
      Removed event: sdt_libstdcxx:catch_1
      [root@jouet ~]#
    
    Ok, disambiguating it using what is in this patch:
    
      [root@jouet ~]# perf list sdt_libstdcxx:catch
    
      List of pre-defined events (to be used in -e):
    
        sdt_libstdcxx:catch@/usr/bin/gcc(9a0730e2bcc6)     [SDT event]
        sdt_libstdcxx:catch@/usr/lib64/libstdc++.so.6.0.22(ef2b7066559a) [SDT event]
      [root@jouet ~]#
      [root@jouet ~]# perf probe -a %sdt_libstdcxx:catch@9a07
      Added new event:
        sdt_libstdcxx:catch  (on %catch in /usr/bin/gcc)
    
      You can now use it in all perf tools, such as:
    
    	perf record -e sdt_libstdcxx:catch -aR sleep 1
    
      [root@jouet ~]# perf probe -l
        sdt_libstdcxx:catch  (on execute_cfa_program+1551@../../../libgcc/unwind-dw2.c in /usr/bin/gcc)
      [root@jouet ~]#
    
    Yeah, it works! But we need to try and simplify this :-)
    
    Update: Some aspects of this simplification take place in the following
            patches.
    
    Signed-off-by: default avatarMasami Hiramatsu <mhiramat@kernel.org>
    Tested-by: default avatarArnaldo Carvalho de Melo <acme@redhat.com>
    Cc: Ananth N Mavinakayanahalli <ananth@linux.vnet.ibm.com>
    Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
    Cc: Hemant Kumar <hemant@linux.vnet.ibm.com>
    Cc: Namhyung Kim <namhyung@kernel.org>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Link: http://lkml.kernel.org/r/146831793746.17065.13065062753978236612.stgit@devbox
    
    
    Signed-off-by: default avatarArnaldo Carvalho de Melo <acme@redhat.com>
    a598180a