refactor(profiling): unifying the allocation and live memory profilers #13988
Conversation
|
|
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 283 ± 3 ms. The average import time from base is: 285 ± 3 ms. The import time difference between this PR and base is: -2.1 ± 0.1 ms. Import time breakdownThe following import paths have shrunk:
|
BenchmarksBenchmark execution time: 2025-07-16 21:00:27 Comparing candidate commit 003755b in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 370 metrics, 2 unstable metrics. |
Performance SLOsCandidate: alysha.wang/memory_profiler_unification (34b9254) 🔵 No Baseline Data (24 suites)🔵 coreapiscenario - 12/12 (2 unstable)🔵 No baseline data available for this suite
|
alyshawang
changed the title
alyshawang
force-pushed
the
alysha.wang/memory_profiler_unification
branch
from
1d5bc4e to
30a71be
Compare
This PR unifies the Python memory profiler's allocation and live heap profilers into a single, consistent, sampling-based system. Previously, both profilers operated independently with separate implementations, overlapping functionality, and distinct data paths. This change replaces the older allocation profiler logic with an enhanced version of the heap profiler that: - Tracks both live allocations and freed-but-unreported allocations in one place. - Produces enriched samples that include: in_use flag (is this allocation still alive?), reported flag (has this sample been reported before?), count (scaling factor for probabilistic allocation count) - Fully removes the legacy iter_events() and its global_alloc_tracker-based sampling.
avidal
force-pushed
the
alysha.wang/memory_profiler_unification
branch
from
30a71be to
7aa77ea
Compare
alyshawang
force-pushed
the
alysha.wang/memory_profiler_unification
branch
from
479b029 to
7aa77ea
Compare
DD_PROFILING_MAX_EVENTS doesn't do anything since #13988. This commit just cleans up some remaining references we missed.
DD_PROFILING_MAX_EVENTS doesn't do anything since #13988. This commit just cleans up some remaining references we missed.
DD_PROFILING_MAX_EVENTS doesn't do anything since #13988. This commit just cleans up some remaining references we missed.
This PR unifies the Python memory profiler's allocation and live heap profilers into a single, consistent, sampling-based system. Previously, both profilers operated independently with separate implementations, overlapping functionality, and distinct data paths. This change replaces the older allocation profiler logic with an enhanced version of the heap profiler that:
These changes were made to remove redundant sampling code. As well, sizes and counts of objects allocated are reported with more accuracy and less overhead. Now ENV: DD_PROFILING_MAX_EVENTS is deprecated and does nothing, use DD_PROFILING_HEAP_SAMPLE_SIZE instead to control sampling frequency of the memory profiler.
This PR also removes a debug-only GIL release in traceback_free. It is unsafe to release the GIL during PyObject_Free, since it can be called during clean up of interpreter data structures, leaving those structures in an inconsistent state if other Python threads are allowed to run. The debug-only release was causing crashes during testing of this PR unrelated to the correctness of the code.
This was tested end to end with a service along with additional unit tests. These tests verify edge cases with more complex allocation patterns while maintaining accuracy using tracemalloc as the ground truth.
PROF-12068
Checklist
Reviewer Checklist