[PROF-8667] Heap Profiling - Part 3 - Snapshot system

[AlexJF]

What does this PR do?

This PR follows #3287 and reduces the conflict of serialization and sampling which was forcing us to potentially drop some allocation samples on the floor.

How does it work?

Initially the assumption was that sampling and serialization operated on the same structures and thus needed to be protected by a mutex. Due to us never wanting to delay sampling code, we had noted the need for a queueing system to not be forced to drop data on the floor. This was the only realistic implementation back when we were relying on free tracepoints which could invalidate object_records at any point in time.

@ivoanjo noted that:

• With the current implementation where staleness checking does not rely on free tracepoints but is done synchronously and under GVL before iteration.
• st_copy operations seem to be extremely efficient, requiring at most 2 mallocs and 2 memcpy.
• We can guarantee a non-concurrent and strict ordering of heap_recorder_prepare_iteration, heap_recorder_for_each_live_object and heap_recorder_finish_iteration (since they all run from the same serialization thread and we have only one such thread).

Given the above, we can trivially do a copy/snapshot of object_records during heap_recorder_prepare_iteration and have the actual iteration operate on that snapshot without ever conflicting with any concurrent sampling that might happen in-between (object_records_snapshot is read-only after construction, mutations will only be recorded in object_records during any sampling that happens concurrently with serialization).

Motivation:
Never wanting to block the allocation tracepoint and not wanting to drop relevant samples on the floor.

Additional Notes:

How to test the change?
It's quite hard to manually or deterministically trigger the conditions whereupon serialization could conflict with sampling and which justified this entire endeavour. Hopefully the fact that all existing specs plus the new "long serialization simulation" specs can provide enough assurance?

For Datadog employees:

• If this PR touches code that signs or publishes builds or packages, or handles
  credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.

Unsure? Have a question? Request a review!

[codecov-commenter]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (bb96f8f) 98.23% compared to head (d58e2f1) 98.23%.
Report is 5 commits behind head on master.

Files	Patch %	Lines
spec/datadog/profiling/stack_recorder_spec.rb	92.85%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3328   +/-   ##
=======================================
  Coverage   98.23%   98.23%           
=======================================
  Files        1253     1253           
  Lines       72998    73020   +22     
  Branches     3427     3429    +2     
=======================================
+ Hits        71712    71734   +22     
  Misses       1286     1286           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ivoanjo]

Here sir, take my 👍 . I've left a few notes on how I don't quite think this is the way to go, but leave the decision in your hands :)

[ivoanjo]

Some last-minute notes from a last pass I did yesterday.