fix(gevent): fix gevent compatibility with "module cloning" approach

[P403n1x87]

This pull request fixes incompatibility between ddtrace and gunicorn (and probably other frameworks that depend on gevent). It does this by making ddtrace hold copies of the modules it loads before gevent.monkey.patch_all() has been called, thus avoiding conflicts between greenlets and real threads subject to startup race conditions.

"Module cloning", as we're calling this approach, is configurable by the DD_UNLOAD_MODULES_FROM_SITECUSTOMIZE environment variable, documented in the release note.

The change accomplishes this by deleting the modules from sys.modules. This causes the Python interpreter to create new copies of the modules on import, not touching the copies originally imported before the deletion (as described in broad strokes here). The result after sitecustomize.py has executed is that ddtrace internal code holds references to modules that have not been monkeypatched by gevent, and application code holds references to modules that may have been monkeypatched by gevent. This means that ddtrace can maintain its real threads for things like remote config and profiling without needing to worry about whether they will turn into greenlets at some point.

The clearest proof that this change improves the gevent+ddtrace compatibility story is the fact that this test passes.

Tips for Reviewers

The changes in this pull request to files other than sitecustomize.py are mostly intended to make existing tests pass under this new paradigm. sitecustomize is the most important file in this diff to understand deeply for a review.

Risks

The main risk inherent to this approach is that some module behaves in an unexpected way when copied. We're mitigating this risk in two ways. The first is by making sure all existing tests pass (obviously), and the second is by putting this new functionality behind a feature flag environment variable that defaults to retaining the current behavior form 1.x.

Checklist

• Change(s) are motivated and described in the PR description.
• Testing strategy is described if automated tests are not included in the PR.
• Risk is outlined (performance impact, potential for breakage, maintainability, etc).
• Change is maintainable (easy to change, telemetry, documentation).
• Library release note guidelines are followed.
• Documentation is included (in-code, generated user docs, public corp docs).
• Don't unload logging
• Add feature flag
• update gunicorn compatibility documentation
• merge dependency PR refactor(tracing): patch all integrations on import #4294
• fix failing CI checks
• merge dependency PR chore(internal): make some internal modules play nicer with tests that create threads #5102
• merge dependency PR chore(testing): make tests slightly more readable #5103

Reviewer Checklist

• Title is accurate.
• No unnecessary changes are introduced.
• Description motivates each change.
• Avoids breaking API changes unless absolutely necessary.
• Testing strategy adequately addresses listed risk(s).
• Change is maintainable (easy to change, telemetry, documentation).
• Release note makes sense to a user of the library.

[brettlangdon]

Link to generated docs for config page changes: https://output.circle-artifacts.com/output/job/d89b9a6d-a048-408a-9406-92bd20e098fb/artifacts/0/tmp/docs/configuration.html#DD_UNLOAD_MODULES_FROM_SITECUSTOMIZE

(I don't see issues, just sharing)

Same for release notes (you have to go to _release_notes_all.html to see the release note as part of open PR)

https://output.circle-artifacts.com/output/job/d89b9a6d-a048-408a-9406-92bd20e098fb/artifacts/0/tmp/docs/_release_notes_all.html#bug-fixes

(again, no issues, just showing)

[brettlangdon]

The only real test changes we need here are that this feature when enabled works with gunicorn/gevent in some way because it is off by default in theory our other tests should cover that the existing behavior doesn't change.

This is correct?

I assume we may want/need more tests added before enabling by default (or "auto"), but I didn't miss any other coverage that this needs to have now, right?

[emmettbutler]

@brettlangdon that's right!

[brettlangdon]

@brettlangdon the presence or absence of gevent doesn't affect how much work module cloning does. If you'd like, I'd be happy to do some performance testing. I do think this is mergeable without that because all of the new functionality is behind a default-off feature flag.

my only real argument to performance testing it now (even just locally with some basic examples) is we are documenting it as a feature for customers, so if the performance was really bad we cannot just yank the feature entirely, we'd have to deprecate it and wait for 2.0 to remove it.

yeah, that is the drastic scenario, but good to consider if it ever was the case.

although, if the performance was bad, we could keep the feature since it would solve some customer problems, but we could document the performance impact of it and not enable it by default.

so.... 🤷🏻

[emmettbutler]

@brettlangdon yup, I follow your thinking. I'm inclined to wait on performance tests until we're considering enabling module cloning by default.

[P403n1x87]

I suppose we can be aggressive here because we still haven't loaded much?

[emmettbutler]

Yes, that's probably true.