This repository has been archived by the owner on Jan 23, 2023. It is now read-only.
Initial perf jitdump implementation #26897
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I looked at the missing symbols. It turns out when we grow the code page, EDIT: When we grow the code pages, the kernel is coalescing the pages and sending an anonymous mmap event for the coalesced region. |
janvorli
reviewed
Sep 27, 2019
noahfalk
reviewed
Sep 30, 2019
Writer should always write the same value, independent of endianness.
|
Status: The draft functional implementation is complete. Testing with perf built from: I captured a fairly simple app that printed hello world + a counter 100K times. I was able to see and annotate R2R and JIT generated method (disassembly), some for multiple compilation tiers. The vast majority of symbols were resolved. There were a handful of unknown addresses. I think we have missed logging of one type of stub. The call stacks looked believable and complete. Unwindinfo didn't seem critically important for 'fp' based perf callstacks. They are probably needed for perf dwarf call stack support. Future
|
sdmaclea
changed the title
|
I have addressed all feedback and finished my to do list. This should be ready for review. /cc @brianrob |
jkotas
reviewed
Oct 4, 2019
janvorli
reviewed
Oct 4, 2019
src/inc/perfjitdump.h
Outdated
| // Finish the jitdump file | ||
| static void Finish(); | ||
| }; | ||
| int |
There was a problem hiding this comment.
These should go to pal.h where all other PAL function declarations are located.
| @@ -274,6 +281,11 @@ void PerfMap::LogPreCompiledMethod(MethodDesc * pMethod, PCODE pCode) | |||
| { | |||
| LIMITED_METHOD_CONTRACT; | |||
|
|
|||
| if (s_Current == nullptr) | |||
There was a problem hiding this comment.
It seems that if one of these logging functions races with the PerfMap::Destroy, we have a potential to crash in the code below.
There was a problem hiding this comment.
This doesn't look like an issue with this function (since it is just an early out), but for other preexisting functions it is potentially an issue.
I'll open a separate PR to fix this.
fengguang
pushed a commit
to 0day-ci/linux
that referenced
this pull request
Nov 1, 2019
While a JIT is jitting code it will eventually need to commit more pages and change these pages to executable permissions. Typically the JIT will want these colocated to minimize branch displacements. The kernel will coalesce these anonymous mapping with identical permissions before sending an MMAP event for the new pages. This means the mmap event for the new pages will include the older pages. These anonymous mmap events will obscure the jitdump injected pseudo events. This means that the jitdump generated symbols, machine code, debugging info, and unwind info will no longer be used. Observations: When a process emits a jit dump marker and a jitdump file, the perf-xxx.map file represents inferior information which has been superceded by the jitdump jit-xxx.dump file. Further the '//anon*' mmap events are only required for the legacy perf-xxx.map mapping. When attaching to an existing process, the synthetic anon map events are given a time stamp of -1. These should not obscure the jitdump events which have an actual time. Summary: Use thread->priv to store whether a jitdump file has been processed During "perf inject --jit", discard "//anon*" mmap events for any pid which has sucessfully processed a jitdump file. Committer testing: // jitdump case perf record <app with jitdump> perf inject --jit --input perf.data --output perfjit.data // verify mmap "//anon" events present initially perf script --input perf.data --show-mmap-events | grep '//anon' // verify mmap "//anon" events removed perf script --input perfjit.data --show-mmap-events | grep '//anon' // no jitdump case perf record <app without jitdump> perf inject --jit --input perf.data --output perfjit.data // verify mmap "//anon" events present initially perf script --input perf.data --show-mmap-events | grep '//anon' // verify mmap "//anon" events not removed perf script --input perfjit.data --show-mmap-events | grep '//anon' Repro: This issue was discovered while testing the initial CoreCLR jitdump implementation. dotnet/coreclr#26897. Cc: Peter Zijlstra <peterz@infradead.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Stephane Eranian <eranian@google.com> Cc: linux-kernel@vger.kernel.org Signed-off-by: Steve MacLean <Steve.MacLean@Microsoft.com>
fengguang
pushed a commit
to 0day-ci/linux
that referenced
this pull request
Nov 2, 2019
While a JIT is jitting code it will eventually need to commit more pages and change these pages to executable permissions. Typically the JIT will want these colocated to minimize branch displacements. The kernel will coalesce these anonymous mapping with identical permissions before sending an MMAP event for the new pages. This means the mmap event for the new pages will include the older pages. These anonymous mmap events will obscure the jitdump injected pseudo events. This means that the jitdump generated symbols, machine code, debugging info, and unwind info will no longer be used. Observations: When a process emits a jit dump marker and a jitdump file, the perf-xxx.map file represents inferior information which has been superceded by the jitdump jit-xxx.dump file. Further the '//anon*' mmap events are only required for the legacy perf-xxx.map mapping. When attaching to an existing process, the synthetic anon map events are given a time stamp of -1. These should not obscure the jitdump events which have an actual time. Summary: Use thread->priv to store whether a jitdump file has been processed During "perf inject --jit", discard "//anon*" mmap events for any pid which has sucessfully processed a jitdump file. Committer testing: // jitdump case perf record <app with jitdump> perf inject --jit --input perf.data --output perfjit.data // verify mmap "//anon" events present initially perf script --input perf.data --show-mmap-events | grep '//anon' // verify mmap "//anon" events removed perf script --input perfjit.data --show-mmap-events | grep '//anon' // no jitdump case perf record <app without jitdump> perf inject --jit --input perf.data --output perfjit.data // verify mmap "//anon" events present initially perf script --input perf.data --show-mmap-events | grep '//anon' // verify mmap "//anon" events not removed perf script --input perfjit.data --show-mmap-events | grep '//anon' Repro: This issue was discovered while testing the initial CoreCLR jitdump implementation. dotnet/coreclr#26897. Cc: Peter Zijlstra <peterz@infradead.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com> Cc: Jiri Olsa <jolsa@redhat.com> Cc: Namhyung Kim <namhyung@kernel.org> Cc: Stephane Eranian <eranian@google.com> Cc: linux-kernel@vger.kernel.org Signed-off-by: Steve MacLean <Steve.MacLean@Microsoft.com>
This was referenced Jan 31, 2020
DanielShaulov
added a commit
to Granulate/linux
that referenced
this pull request
May 5, 2020
While a JIT is jitting code it will eventually need to commit more pages and change these pages to executable permissions. Typically the JIT will want these colocated to minimize branch displacements. The kernel will coalesce these anonymous mapping with identical permissions before sending an MMAP event for the new pages. This means the mmap event for the new pages will include the older pages. These anonymous mmap events will obscure the jitdump injected pseudo events. This means that the jitdump generated symbols, machine code, debugging info, and unwind info will no longer be used. Observations: When a process emits a jit dump marker and a jitdump file, the perf-xxx.map file represents inferior information which has been superceded by the jitdump jit-xxx.dump file. Further the '//anon*' mmap events are only required for the legacy perf-xxx.map mapping. When attaching to an existing process, the synthetic anon map events are given a time stamp of -1. These should not obscure the jitdump events which have an actual time. Summary: Use thread->priv to store whether a jitdump file has been processed During "perf inject --jit", discard "//anon*" mmap events for any pid which has sucessfully processed a jitdump file. Committer testing: // jitdump case perf record <app with jitdump> perf inject --jit --input perf.data --output perfjit.data // verify mmap "//anon" events present initially perf script --input perf.data --show-mmap-events | grep '//anon' // verify mmap "//anon" events removed perf script --input perfjit.data --show-mmap-events | grep '//anon' // no jitdump case perf record <app without jitdump> perf inject --jit --input perf.data --output perfjit.data // verify mmap "//anon" events present initially perf script --input perf.data --show-mmap-events | grep '//anon' // verify mmap "//anon" events not removed perf script --input perfjit.data --show-mmap-events | grep '//anon' Repro: This issue was discovered while testing the initial CoreCLR jitdump implementation. dotnet/coreclr#26897. Signed-off-by: Steve MacLean <Steve.MacLean@Microsoft.com>
fengguang
pushed a commit
to 0day-ci/linux
that referenced
this pull request
May 27, 2020
**perf-<pid>.map and jit-<pid>.dump designs:
When a JIT generates code to be executed, it must allocate memory and
mark it executable using an mmap call.
*** perf-<pid>.map design
The perf-<pid>.map assumes that any sample recorded in an anonymous
memory page is JIT code. It then tries to resolve the symbol name by
looking at the process' perf-<pid>.map.
*** jit-<pid>.dump design
The jit-<pid>.dump mechanism takes a different approach. It requires a JIT
to write a `<path>/jit-<pid>.dump` file. This file must also be mmapped
so that perf inject -jit can find the file. The JIT must also add
JIT_CODE_LOAD records for any functions it generates. The records are
timestamped using a clock which can be correlated to the perf record
clock.
After perf record, the `perf inject -jit` pass parses the recording
looking for a `<path>/jit-<pid>.dump` file. When it finds the file, it
parses it and for each JIT_CODE_LOAD record:
* creates an elf file `<path>/jitted-<pid>-<code_index>.so
* injects a new mmap record mapping the new elf file into the process.
*** Coexistence design
The kernel and perf support both of these mechanisms. We need to make
sure perf works on an app supporting either or both of these mechanisms.
Both designs rely on mmap records to determine how to resolve an ip
address.
The mmap records of both techniques by definition overlap. When the JIT
compiles a method, it must:
* allocate memory (mmap)
* add execution privilege (mprotect or mmap. either will
generate an mmap event form the kernel to perf)
* compile code into memory
* add a function record to perf-<pid>.map and/or jit-<pid>.dump
Because the jit-<pid>.dump mechanism supports greater capabilities, perf
prefers the symbols from jit-<pid>.dump. It implements this based on
timestamp ordering of events. There is an implicit ASSUMPTION that the
JIT_CODE_LOAD record timestamp will be after the // anon mmap event that
was generated during memory allocation or adding the execution privilege setting.
*** Problems with the ASSUMPTION
The ASSUMPTION made in the Coexistence design section above is violated
in the following scenario.
*** Scenario
While a JIT is jitting code it will eventually need to commit more
pages and change these pages to executable permissions. Typically the
JIT will want these collocated to minimize branch displacements.
The kernel will coalesce these anonymous mapping with identical
permissions before sending an MMAP event for the new pages. The address
range of the new mmap will not be just the most recently mmap pages.
It will include the entire coalesced mmap region.
See mm/mmap.c
unsigned long mmap_region(struct file *file, unsigned long addr,
unsigned long len, vm_flags_t vm_flags, unsigned long pgoff,
struct list_head *uf)
{
...
/*
* Can we just expand an old mapping?
*/
...
perf_event_mmap(vma);
...
}
*** Symptoms
The coalesced // anon mmap event will be timestamped after the
JIT_CODE_LOAD records. This means it will be used as the most recent
mapping for that entire address range. For remaining events it will look at the
inferior perf-<pid>.map for symbols.
If both mechanisms are supported, the symbol will appear twice with
different module names. This causes weird behavior in reporting.
If only jit-<pid>.dump is supported, the symbol will no longer be resolved.
** Implemented solution
This patch solves the issue by removing // anon mmap events for any
process which has a valid jit-<pid>.dump file.
It tracks on a per process basis to handle the case where some running
apps support jit-<pid>.dump, but some only support perf-<pid>.map.
It adds new assumptions:
* // anon mmap events are only required for perf-<pid>.map support.
* An app that uses jit-<pid>.dump, no longer needs
perf-<pid>.map support. It assumes that any perf-<pid>.map info is
inferior.
*** Details
Use thread->priv to store whether a jitdump file has been processed
During "perf inject --jit", discard "//anon*" mmap events for any pid which
has sucessfully processed a jitdump file.
** Committer testing:
// jitdump case
perf record <app with jitdump>
perf inject --jit --input perf.data --output perfjit.data
// verify mmap "//anon" events present initially
perf script --input perf.data --show-mmap-events | grep '//anon'
// verify mmap "//anon" events removed
perf script --input perfjit.data --show-mmap-events | grep '//anon'
// no jitdump case
perf record <app without jitdump>
perf inject --jit --input perf.data --output perfjit.data
// verify mmap "//anon" events present initially
perf script --input perf.data --show-mmap-events | grep '//anon'
// verify mmap "//anon" events not removed
perf script --input perfjit.data --show-mmap-events | grep '//anon'
** Repro:
This issue was discovered while testing the initial CoreCLR jitdump
implementation. dotnet/coreclr#26897.
** Alternate solutions considered
These were also briefly considered
* Change kernel to not coalesce mmap regions.
* Change kernel reporting of coalesced mmap regions to perf. Only
include newly mapped memory.
* Only strip parts of // anon mmap events overlapping existing
jitted-<pid>-<code_index>.so mmap events.
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Steve MacLean <Steve.MacLean@Microsoft.com>
torvalds
pushed a commit
to torvalds/linux
that referenced
this pull request
Aug 11, 2020
**perf-<pid>.map and jit-<pid>.dump designs:
When a JIT generates code to be executed, it must allocate memory and
mark it executable using an mmap call.
*** perf-<pid>.map design
The perf-<pid>.map assumes that any sample recorded in an anonymous
memory page is JIT code. It then tries to resolve the symbol name by
looking at the process' perf-<pid>.map.
*** jit-<pid>.dump design
The jit-<pid>.dump mechanism takes a different approach. It requires a
JIT to write a `<path>/jit-<pid>.dump` file. This file must also be
mmapped so that perf inject -jit can find the file. The JIT must also
add JIT_CODE_LOAD records for any functions it generates. The records
are timestamped using a clock which can be correlated to the perf record
clock.
After perf record, the `perf inject -jit` pass parses the recording
looking for a `<path>/jit-<pid>.dump` file. When it finds the file, it
parses it and for each JIT_CODE_LOAD record:
* creates an elf file `<path>/jitted-<pid>-<code_index>.so
* injects a new mmap record mapping the new elf file into the process.
*** Coexistence design
The kernel and perf support both of these mechanisms. We need to make
sure perf works on an app supporting either or both of these mechanisms.
Both designs rely on mmap records to determine how to resolve an ip
address.
The mmap records of both techniques by definition overlap. When the JIT
compiles a method, it must:
* allocate memory (mmap)
* add execution privilege (mprotect or mmap. either will
generate an mmap event form the kernel to perf)
* compile code into memory
* add a function record to perf-<pid>.map and/or jit-<pid>.dump
Because the jit-<pid>.dump mechanism supports greater capabilities, perf
prefers the symbols from jit-<pid>.dump. It implements this based on
timestamp ordering of events. There is an implicit ASSUMPTION that the
JIT_CODE_LOAD record timestamp will be after the // anon mmap event that
was generated during memory allocation or adding the execution privilege setting.
*** Problems with the ASSUMPTION
The ASSUMPTION made in the Coexistence design section above is violated
in the following scenario.
*** Scenario
While a JIT is jitting code it will eventually need to commit more
pages and change these pages to executable permissions. Typically the
JIT will want these collocated to minimize branch displacements.
The kernel will coalesce these anonymous mapping with identical
permissions before sending an MMAP event for the new pages. The address
range of the new mmap will not be just the most recently mmap pages.
It will include the entire coalesced mmap region.
See mm/mmap.c
unsigned long mmap_region(struct file *file, unsigned long addr,
unsigned long len, vm_flags_t vm_flags, unsigned long pgoff,
struct list_head *uf)
{
...
/*
* Can we just expand an old mapping?
*/
...
perf_event_mmap(vma);
...
}
*** Symptoms
The coalesced // anon mmap event will be timestamped after the
JIT_CODE_LOAD records. This means it will be used as the most recent
mapping for that entire address range. For remaining events it will look
at the inferior perf-<pid>.map for symbols.
If both mechanisms are supported, the symbol will appear twice with
different module names. This causes weird behavior in reporting.
If only jit-<pid>.dump is supported, the symbol will no longer be resolved.
** Implemented solution
This patch solves the issue by removing // anon mmap events for any
process which has a valid jit-<pid>.dump file.
It tracks on a per process basis to handle the case where some running
apps support jit-<pid>.dump, but some only support perf-<pid>.map.
It adds new assumptions:
* // anon mmap events are only required for perf-<pid>.map support.
* An app that uses jit-<pid>.dump, no longer needs
perf-<pid>.map support. It assumes that any perf-<pid>.map info is
inferior.
*** Details
Use thread->priv to store whether a jitdump file has been processed
During "perf inject --jit", discard "//anon*" mmap events for any pid which
has sucessfully processed a jitdump file.
** Testing:
// jitdump case
perf record <app with jitdump>
perf inject --jit --input perf.data --output perfjit.data
// verify mmap "//anon" events present initially
perf script --input perf.data --show-mmap-events | grep '//anon'
// verify mmap "//anon" events removed
perf script --input perfjit.data --show-mmap-events | grep '//anon'
// no jitdump case
perf record <app without jitdump>
perf inject --jit --input perf.data --output perfjit.data
// verify mmap "//anon" events present initially
perf script --input perf.data --show-mmap-events | grep '//anon'
// verify mmap "//anon" events not removed
perf script --input perfjit.data --show-mmap-events | grep '//anon'
** Repro:
This issue was discovered while testing the initial CoreCLR jitdump
implementation. dotnet/coreclr#26897.
** Alternate solutions considered
These were also briefly considered:
* Change kernel to not coalesce mmap regions.
* Change kernel reporting of coalesced mmap regions to perf. Only
include newly mapped memory.
* Only strip parts of // anon mmap events overlapping existing
jitted-<pid>-<code_index>.so mmap events.
Signed-off-by: Steve MacLean <Steve.MacLean@Microsoft.com>
Acked-by: Ian Rogers <irogers@google.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lore.kernel.org/lkml/1590544271-125795-1-git-send-email-steve.maclean@linux.microsoft.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
|
@sdmaclea does this work out of the box or do I need some special build? |
|
To fix all known issues, you should use a build of perf from the linux tip. The last known issue was fixed with torvalds/linux@c8f6ae1 Building perf is not that hard.
|
|
Also worth noting that |
|
Ok, thanks... will give this a try. |
|
Confessing my ignorance here. What can I do If I'm stuck on some older kernel version ( say 4.10)... does building the latest perf from there work? I end up with I also realized I needed to run Now I can sometimes see jitted code, eg one of the five methods below: Do the jit events this play well with the tiering annotations? I notice above you were running debug so didn't see tiering. Also |
|
Yes, building a new perf and using it with an older kernel should be just fine. For the most part, the user-mode tool is backwards compatible, and I've been told forwards compatible... Yes, @kouvel did some work a while back to add tiering labels to the symbols so they should show up just fine in this view. In PerfView they are stripped but can be enabled by looking at an optimization tiers view (you'll see it in the view name) or running PerfView with /ShowOptimizationTiers to enable it for all stack views. |
|
Thanks all. Building the latest seems to be working: |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This initial commit:
JIT_CODE_LOADrecords for JIT generated code and for stubs.It has been lightly tested on Ubuntu 16.04 with perf built from Ubuntu-azure-4.15.0-1057.62 with the patch https://lkml.org/lkml/2019/9/20/862
Basic functionality seems correct, however there are sample which are hitting in
perf-*.map, but not found injit-*.dump.It needs to be further debugged.
Fixes #26842
/cc @dotnet/dotnet-diag