chore: add a registry for supported integrations #13215
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
|
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
scripts/integration_registry/_update_integration_registry_versions.py
Outdated
Show resolved
Hide resolved
tests/contrib/integration_registry/test_external_dependencies.py
Outdated
Show resolved
Hide resolved
tests/contrib/integration_registry/test_external_dependencies.py
Outdated
Show resolved
Hide resolved
tests/contrib/integration_registry/test_external_dependencies.py
Outdated
Show resolved
Hide resolved
scripts/integration_registry/_update_integration_registry_versions.py
Outdated
Show resolved
Hide resolved
Contributor
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 232 ± 2 ms. The average import time from base is: 237 ± 3 ms. The import time difference between this PR and base is: -5.2 ± 0.1 ms. Import time breakdownThe following import paths have shrunk:
|
BenchmarksBenchmark execution time: 2025-05-01 13:47:28 Comparing candidate commit 5e9753d in PR branch Found 0 performance improvements and 4 performance regressions! Performance is the same for 503 metrics, 5 unstable metrics. scenario:iast_aspects-ljust_aspect
scenario:iast_aspects-ospathdirname_aspect
scenario:iast_aspects-ospathnormcase_aspect
scenario:iast_aspects-strip_aspect
|
tests/contrib/integration_registry/registry_update_helpers/integration_registry_manager.py
Outdated
Show resolved
Hide resolved
tests/contrib/integration_registry/registry_update_helpers/update_orchestrator.py
Outdated
Show resolved
Hide resolved
tests/contrib/integration_registry/registry_update_helpers/update_orchestrator.py
Outdated
Show resolved
Hide resolved
wantsui
approved these changes
Apr 23, 2025
Collaborator
wantsui
left a comment
wantsui
left a comment
There was a problem hiding this comment.
🔥 LGTM - I left some nits but they shouldn't be blocking. (Though I think the README links should be fixed if possible).
tests/contrib/integration_registry/registry_update_helpers/integration_update_orchestrator.py
Show resolved
Hide resolved
erikayasuda
approved these changes
Apr 24, 2025
Contributor
erikayasuda
left a comment
erikayasuda
left a comment
There was a problem hiding this comment.
A few nits, otherwise LGTM 👍
scripts/integration_registry/_update_integration_registry_versions.py
Outdated
Show resolved
Hide resolved
scripts/integration_registry/_update_integration_registry_versions.py
Outdated
Show resolved
Hide resolved
tests/contrib/integration_registry/test_external_dependencies.py
Outdated
Show resolved
Hide resolved
tests/contrib/integration_registry/test_external_dependencies.py
Outdated
Show resolved
Hide resolved
tests/contrib/integration_registry/registry_update_helpers/integration_update_orchestrator.py
Show resolved
Hide resolved
fix pytest plugin tests
tests/contrib/integration_registry/registry_update_helpers/integration.py
Outdated
Show resolved
Hide resolved
erikayasuda
reviewed
Apr 30, 2025
brettlangdon
pushed a commit
that referenced
this pull request
May 6, 2025
## Description
This PR introduces an integration registry, that contains information
for all supported integrations.
`registry.yml` located in `ddtrace/contrib/integration_registry/`
- Integration name
- Dependency names for all packages patched by this integration
- Is_External: whether the patched dependency is an external package
downloadable via Pypi or `False` for stdlib Python packages or abstract
integrations such as dbapi
- Is_Tested: Optional, only present if the integration lacks testing,
e.g. `aioredis` which is untested and planned to be deprecated
- Tested Version to Dependency Mapping: Each dependency this integration
patches, with `min` and `max` versions for what is tested
Along with the `registry.yaml`, this PR makes quite a few changes to
setup the necessary information to connect Riot testing venvs, the
dependencies in those tests, to the integration being tested.
- `riotfile.py` updates:
- any contrib with multiple sub-test suites now shares a common venv
naming schema of: `integration_name:sub-test`
- `django_hosts` -> `django:django_hosts`
- `psycopg2` (our integration name is psycopg) -> `psycopg:psycopg2`
- `graphene` -> `graphql:graphene`
- All integration to dependency or module name mappings moved to
`ddtrace/contrib/integration_registry/mappings.py`
- Update suitespec to use new venv names
- `scripts/freshvenvs.py` updates
- Uses riot to build a mapping of venv names to riot hashes, allowing us
to correlate files from `.riot/requirements/*.txt` to the respective
test environment name. This change ensures that dependency versions
collected are actually correlated with the test suite venv being tested
(as opposed to `test suite X` relying on `dependency Y==0.1`, while
`test suite Y` tests `dependencyY>1.0`, previously the tested version of
`Y` would be recorded as `0.1`, now its `>1.0`.
- Other changes include:
- Using new dependency name to integration name mappings within
`ddtrace/integration_registry/mappings.py`, instead of storing the
mapping in this script file
Automatic updates of integration `registry.yml` for contrib testing
- For all tests testing `tests/contrib/*`, we patch the `getattr`
function and listen for `getattr` calls where `_datadog_patch` is being
called on an object. This call is used in most of our contribs, and
allows us access to the modules being patched during tests. If this
attribute was used in the `getattr` call, we store the object it was
called on, as well as the callstack, for later processing.
- After the test session concludes, we do some processing to
automatically update our `registry.yaml` with data collected from the
patched object and traceback.
- `registry.yaml` update process during testing:
- Patched objects are stored by `IntegrationRegistryManager` class,
which later processes the objects, checks if they were valid contribs,
gets the dependency name of the patched top level module, ie: `module
kafka -> (using importlib.metadata) -> we get package names producing
this module -> ie: `confluent-kafka``.
- Also gets the version of the module being patched and stores for later
processing.
- After `IntegrationRegistryManager` runs, we run the
`IntegrationRegistryUpdater` class in a separate virtual environment,
since the class relies on `riot` and `pyyaml`, both of which are not
available in riot virtual environments, so we make our own.
- `IntegrationRegistryUpdater`:
- Parses our data exported by `IntegrationRegistryManager`, which
includes tested integration, patched dependency name, patched dependency
version
- Reads our current `registry.yml`, and decides whether to update the
file, depending on if the dependency / integration combo is already
present in the file, or if the tested version is outside the current
files version range for the dependency, if `True`, runs
`scripts/integration_registry/update_and_format_registry.py`.
- Updates and formats the registry automatically for new integrations,
dependencies, or dependency versions!
New Scripting:
- `scripts/integration_registry/_format_integration_registry.py`:
Formats the integration registry with spaces between integration
registries, since no python yaml packages offer this functionality. Done
for readability
-
`scripts/integration_registry/_update_integration_registry_versions.py`:
Updates the registry with new data from
`supported_versions_output.json`, which is generated from the
`scripts/freshvenvs.py` script.
- `scripts/integration_registry/update_and_format_registry.py`: Runs
`scripts/freshvenvs.py`, and the former two scripts to update all of our
test information and resave the new registry. This is the main script to
call.
Other updates:
- Adds a git diff check to CI runs for contrib tests. If the
`registry.yml` was updated during CI, the new check fails the CI job,
similar to riot lockfile check, and tells the user to update the
`registry.yml` with the new information.
Several tests have been added to ensure that the `registry.yaml` stays
updated when new contribs are added:
Registry Tests
- test that every dir within `ddtrace/contrib/internal` has an
associated registry entry
- test that `registry.yaml` conforms to strict schema
- test that every `external` dependency has corresponding tested version
information in `registry.yaml`
- test that every `external` dependency exists on Pypi
Riotfile Tests
- test that every directory within `ddtrace/contrib/internal` has a
corresponding riot venv name associated with it
- test that every riot venv with a test path containing `tests/contrib/`
uses a venv name that is a directory within `ddtrace/contrib/internal`,
if the venv name is a subtest in the form of `integration[sub-test]`,
tests that the `integration` is located within
`ddtrace/contrib/internal`.
## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))
## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)
---------
Co-authored-by: Federico Mon <federico.mon@datadoghq.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces an integration registry, that contains information for all supported integrations.
registry.ymllocated inddtrace/contrib/integration_registry/Falsefor stdlib Python packages or abstract integrations such as dbapiaiorediswhich is untested and planned to be deprecatedminandmaxversions for what is testedAlong with the
registry.yaml, this PR makes quite a few changes to setup the necessary information to connect Riot testing venvs, the dependencies in those tests, to the integration being tested.riotfile.pyupdates:integration_name:sub-testdjango_hosts->django:django_hostspsycopg2(our integration name is psycopg) ->psycopg:psycopg2graphene->graphql:grapheneddtrace/contrib/integration_registry/mappings.pyscripts/freshvenvs.pyupdates.riot/requirements/*.txtto the respective test environment name. This change ensures that dependency versions collected are actually correlated with the test suite venv being tested (as opposed totest suite Xrelying ondependency Y==0.1, whiletest suite YtestsdependencyY>1.0, previously the tested version ofYwould be recorded as0.1, now its>1.0.ddtrace/integration_registry/mappings.py, instead of storing the mapping in this script fileAutomatic updates of integration
registry.ymlfor contrib testingtests/contrib/*, we patch thegetattrfunction and listen forgetattrcalls where_datadog_patchis being called on an object. This call is used in most of our contribs, and allows us access to the modules being patched during tests. If this attribute was used in thegetattrcall, we store the object it was called on, as well as the callstack, for later processing.registry.yamlwith data collected from the patched object and traceback.registry.yamlupdate process during testing:IntegrationRegistryManagerclass, which later processes the objects, checks if they were valid contribs, gets the dependency name of the patched top level module, ie:module kafka -> (using importlib.metadata) -> we get package names producing this module -> ie:confluent-kafka``.- After
IntegrationRegistryManagerruns, we run theIntegrationRegistryUpdaterclass in a separate virtual environment, since the class relies onriotandpyyaml, both of which are not available in riot virtual environments, so we make our own.-
IntegrationRegistryUpdater:IntegrationRegistryManager, which includes tested integration, patched dependency name, patched dependency versionregistry.yml, and decides whether to update the file, depending on if the dependency / integration combo is already present in the file, or if the tested version is outside the current files version range for the dependency, ifTrue, runsscripts/integration_registry/update_and_format_registry.py.New Scripting:
scripts/integration_registry/_format_integration_registry.py: Formats the integration registry with spaces between integration registries, since no python yaml packages offer this functionality. Done for readabilityscripts/integration_registry/_update_integration_registry_versions.py: Updates the registry with new data fromsupported_versions_output.json, which is generated from thescripts/freshvenvs.pyscript.scripts/integration_registry/update_and_format_registry.py: Runsscripts/freshvenvs.py, and the former two scripts to update all of our test information and resave the new registry. This is the main script to call.Other updates:
registry.ymlwas updated during CI, the new check fails the CI job, similar to riot lockfile check, and tells the user to update theregistry.ymlwith the new information.Several tests have been added to ensure that the
registry.yamlstays updated when new contribs are added:Registry Tests
ddtrace/contrib/internalhas an associated registry entryregistry.yamlconforms to strict schemaexternaldependency has corresponding tested version information inregistry.yamlexternaldependency exists on PypiRiotfile Tests
ddtrace/contrib/internalhas a corresponding riot venv name associated with ittests/contrib/uses a venv name that is a directory withinddtrace/contrib/internal, if the venv name is a subtest in the form ofintegration[sub-test], tests that theintegrationis located withinddtrace/contrib/internal.Checklist
Reviewer Checklist