Skip to content

PROF-10509: Add profiling scenarios to Python onboarding tests #3002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
szegedi merged 2 commits into from
Feb 5, 2025
Merged

PROF-10509: Add profiling scenarios to Python onboarding tests #3002

szegedi merged 2 commits into from
Feb 5, 2025

Conversation

@szegedi
Copy link
Contributor

@szegedi szegedi commented Sep 9, 2024

Motivation

Python APM library suports DD_PROFILING_ENABLED=auto since 1.37.0, see DataDog/dd-trace-py#9904

Changes

Adds the three standard onboarding _PROFILING scenarios to the Python test matrix.

Workflow

  1. ⚠️ Create your PR as draft ⚠️
  2. Work on you PR until the CI passes (if something not related to your task is failing, you can ignore it)
  3. Mark it as ready for review
    • Test logic is modified? -> Get a review from RFC owner. We're working on refining the codeowners file quickly.
    • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

  • If PR title starts with [<language>], double-check that only <language> is impacted by the change
  • No system-tests internal is modified. Otherwise, I have the approval from R&P team
  • CI is green, or failing jobs are not related to this change (and you are 100% sure about this statement)
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added (or removed)?

@szegedi szegedi force-pushed the szegedi/enable-python-profiling-onboarding branch 2 times, most recently from f5f7ea6 to 7493f2e Compare January 20, 2025 10:43
@szegedi
Copy link
Contributor Author

szegedi commented Jan 21, 2025

I executed a GitLab pipeline with the branch, with the following results:

  • CONTAINER_AUTO_INJECTION_INSTALL_SCRIPT_PROFILING only failed on RedHat 7.9 on amd64 for both test-app-python and test-app-python-alpine due to a timeout. It is possible a re-run would fix them. In fact, I'll attempt a re-run.
  • HOST_AUTO_INJECTION_INSTALL_SCRIPT_PROFILING succeeded on all targets: {Amazon_Linux_2023, RedHat_9_0, Ubuntu_22} x {amd64, arm64} x {test_app_python}.
  • SIMPLE_AUTO_INJECTION_PROFILING failed on 6 of 28 targets. All failures were for the exact same targets where HOST_AUTO_INJECTION_INSTALL_SCRIPT_PROFILING scenario succeeded – that is, all test_app_python, while all tests with test_app_python_container and test_app_python_alpine succeeded on all platforms. Maybe there's a correlation between the two?

@szegedi szegedi force-pushed the szegedi/enable-python-profiling-onboarding branch from 7493f2e to 48bfaa6 Compare January 27, 2025 13:50
@szegedi szegedi force-pushed the szegedi/enable-python-profiling-onboarding branch from 48bfaa6 to 3db3b5d Compare February 4, 2025 15:38
@szegedi
Copy link
Contributor Author

szegedi commented Feb 5, 2025 •
edited
Loading

Apparently now all tests with test-app-python-alpine fail; I added a JIRA issue for them. All other Python profiling tests (including k8s) pass 🎉 See: https://gitlab.ddbuild.io/DataDog/system-tests/-/pipelines/54867553

@szegedi szegedi marked this pull request as ready for review February 5, 2025 10:56
@szegedi szegedi requested a review from a team as a code owner February 5, 2025 10:56
@szegedi szegedi merged commit 4dda265 into main Feb 5, 2025
434 of 436 checks passed
@szegedi szegedi deleted the szegedi/enable-python-profiling-onboarding branch February 5, 2025 14:20
@szegedi szegedi mentioned this pull request Feb 5, 2025
Merged
2 tasks
sezen-datadog added a commit that referenced this pull request Feb 10, 2025
@sezen-datadog
# This is the 1st commit message:
d2064eb 
Extend vulnerability location data with class

re-enable aws integrations tests (#3733)

feat(tests/test_data_integrity): update test_datadog_external_env for Go v1.72.0 and forward (#3961)

Activate ruff rules on tests/ folder (#3999)

[python] Avoid passing global tracer to pin in weblog apps (#4004)

All classes must declare feature ids (#4003)

Extend mypy scope (#4002)

Onboarding: bug marker profiling (#4005)

Docker SSI: fix scenario (#4006)

[ruby] Enable IP blocking tests for Ruby (#3937)

Activate ruff rules on tests/ folder (#4007)

[nodejs] remove auto login event skip (#3998)

Email HTML Injection detection in IAST Java (#3906)

Co-authored-by: Mario Vidal Domínguez <60353145+Mariovido@users.noreply.github.com>

Add test to check absence of client computed stats (#3812)

[java] Skip payara/CROSSED_TRACING_LIBRARIES/prod (#4009)

Add GraphQL error extension tests (#3986)

Co-authored-by: William Conti <william.conti@datadoghq.com>

Use prod agent, dev agent broke lot  of tracers (#4011)

update xpassing baggage tests for unimplemented languages (#3773)

[NodeJS] skip failing baggage tests (#4015)

[python] fix 500 errors in sql queries (#3997)

Activate ruff rules on tests/ folder (#4010)

Hotfix

Fix fuzzer

[Nodejs] Enable untrusted deserialization stack trace test for Node.js (#3995)

[python] use main again for dev branch (#4008)

Co-authored-by: erikayasuda <153395705+erikayasuda@users.noreply.github.com>
Co-authored-by: Charles de Beauchesne <charles.debeauchesne@datadoghq.com>

Revert agent dev fix (#4013)

[PHP] Enable rasp telemetry tests for PHP (#3972)

[skipci] Update CODEOWNERS for static files (#4012)

[Java] Enable more easy wins (#4018)

[java] Bump GraalVM system test to JDK 22 (#4001)

[NodeJS] skip more failing baggage tests (#4021)

[Debugger] Update dotnet Exception Replay tests (#3974)

Test multiple rasp during one request (#3989)

Add test for location extended data (#3978)

Fix APPSEC_NO_STATS scenario name (#4019)

Avoid false XPASS on APPSEC_WAF_TELEMETRY (#4029)

[java] Enable Test_Blocking_strip_response_headers in some variants (#4033)

[java] Remove some outdated manifest entries (#4039)

[java] Fix xpass for Test_SecurityEvents_Appsec_Metastruct_Disabled (#4038)

Consolidate remote config tests into same directory/file (#4031)

[python] use last patch version of python for django weblogs (#4025)

crashtracking: assert si_signo is set to 11 (#4023)

# This is the commit message #2:

[nodejs] Fix get_target_branch - "quotes, parens and # (#3952)"" (#3953)

Co-authored-by: Charles de Beauchesne <charles.debeauchesne@datadoghq.com>
# This is the commit message #3:

Fix Standalone Billing values (#3965)

# This is the commit message #4:

Add success to valid WAF telemetry tags (#3966)

# This is the commit message #5:

Ensure tracer release coherence (#3967)

[Nodejs] Code injection telemetry metric (#3959)

Do not use special chars in "redacted" string used in scrubber (#3977)

fix Context Propagation: Extraction feature_id (#3970)

[dotnet] Enable Custom Blocking Response tests (#3971)

[python] update flask base image (#3979)

[ruby] Skip failing test for APMAPI-1141 (#3980)

update weblog sampling_test use sampling rules instead of deprecated envvar (#3984)

[python] enable Python IAST+SSRF tests (#2512)

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

[python] IAST Header source in werkzeug 3.1 (#3991)

[python] switch flaky decorator for bug (#3990)

[python] update versions for major release (#3993)

[test optimization] Update config shape sent as telemetry (#3992)

Add profiling scenarios to Python onboarding tests (#3002)

Update changelog

re-enable aws integrations tests (#3733)

feat(tests/test_data_integrity): update test_datadog_external_env for Go v1.72.0 and forward (#3961)

[python] Avoid passing global tracer to pin in weblog apps (#4004)

All classes must declare feature ids (#4003)

Extend mypy scope (#4002)

Onboarding: bug marker profiling (#4005)

Docker SSI: fix scenario (#4006)

[ruby] Enable IP blocking tests for Ruby (#3937)

[nodejs] remove auto login event skip (#3998)

[NodeJS] skip failing baggage tests (#4015)

[python] fix 500 errors in sql queries (#3997)

Hotfix

Fix fuzzer

[python] use main again for dev branch (#4008)

Co-authored-by: erikayasuda <153395705+erikayasuda@users.noreply.github.com>
Co-authored-by: Charles de Beauchesne <charles.debeauchesne@datadoghq.com>

Revert agent dev fix (#4013)

[skipci] Update CODEOWNERS for static files (#4012)

[Java] Enable more easy wins (#4018)

[java] Bump GraalVM system test to JDK 22 (#4001)

[NodeJS] skip more failing baggage tests (#4021)

[Debugger] Update dotnet Exception Replay tests (#3974)

Test multiple rasp during one request (#3989)

Add test for location extended data (#3978)

Fix APPSEC_NO_STATS scenario name (#4019)

Avoid false XPASS on APPSEC_WAF_TELEMETRY (#4029)

[java] Enable Test_Blocking_strip_response_headers in some variants (#4033)

[java] Remove some outdated manifest entries (#4039)

[java] Fix xpass for Test_SecurityEvents_Appsec_Metastruct_Disabled (#4038)

Consolidate remote config tests into same directory/file (#4031)

[python] use last patch version of python for django weblogs (#4025)

crashtracking: assert si_signo is set to 11 (#4023)

class vs path
sezen-datadog added a commit that referenced this pull request Feb 10, 2025
@sezen-datadog
# This is the 1st commit message:
d4259f6 
Extend vulnerability location data with class

re-enable aws integrations tests (#3733)

feat(tests/test_data_integrity): update test_datadog_external_env for Go v1.72.0 and forward (#3961)

Activate ruff rules on tests/ folder (#3999)

[python] Avoid passing global tracer to pin in weblog apps (#4004)

All classes must declare feature ids (#4003)

Extend mypy scope (#4002)

Onboarding: bug marker profiling (#4005)

Docker SSI: fix scenario (#4006)

[ruby] Enable IP blocking tests for Ruby (#3937)

Activate ruff rules on tests/ folder (#4007)

[nodejs] remove auto login event skip (#3998)

Email HTML Injection detection in IAST Java (#3906)

Co-authored-by: Mario Vidal Domínguez <60353145+Mariovido@users.noreply.github.com>

Add test to check absence of client computed stats (#3812)

[java] Skip payara/CROSSED_TRACING_LIBRARIES/prod (#4009)

Add GraphQL error extension tests (#3986)

Co-authored-by: William Conti <william.conti@datadoghq.com>

Use prod agent, dev agent broke lot  of tracers (#4011)

update xpassing baggage tests for unimplemented languages (#3773)

[NodeJS] skip failing baggage tests (#4015)

[python] fix 500 errors in sql queries (#3997)

Activate ruff rules on tests/ folder (#4010)

Hotfix

Fix fuzzer

[Nodejs] Enable untrusted deserialization stack trace test for Node.js (#3995)

[python] use main again for dev branch (#4008)

Co-authored-by: erikayasuda <153395705+erikayasuda@users.noreply.github.com>
Co-authored-by: Charles de Beauchesne <charles.debeauchesne@datadoghq.com>

Revert agent dev fix (#4013)

[PHP] Enable rasp telemetry tests for PHP (#3972)

[skipci] Update CODEOWNERS for static files (#4012)

[Java] Enable more easy wins (#4018)

[java] Bump GraalVM system test to JDK 22 (#4001)

[NodeJS] skip more failing baggage tests (#4021)

[Debugger] Update dotnet Exception Replay tests (#3974)

Test multiple rasp during one request (#3989)

Add test for location extended data (#3978)

Fix APPSEC_NO_STATS scenario name (#4019)

Avoid false XPASS on APPSEC_WAF_TELEMETRY (#4029)

[java] Enable Test_Blocking_strip_response_headers in some variants (#4033)

[java] Remove some outdated manifest entries (#4039)

[java] Fix xpass for Test_SecurityEvents_Appsec_Metastruct_Disabled (#4038)

Consolidate remote config tests into same directory/file (#4031)

[python] use last patch version of python for django weblogs (#4025)

crashtracking: assert si_signo is set to 11 (#4023)

# This is the commit message #2:

[nodejs] Fix get_target_branch - "quotes, parens and # (#3952)"" (#3953)

Co-authored-by: Charles de Beauchesne <charles.debeauchesne@datadoghq.com>
# This is the commit message #3:

Fix Standalone Billing values (#3965)

# This is the commit message #4:

Add success to valid WAF telemetry tags (#3966)

# This is the commit message #5:

Ensure tracer release coherence (#3967)

[Nodejs] Code injection telemetry metric (#3959)

Do not use special chars in "redacted" string used in scrubber (#3977)

fix Context Propagation: Extraction feature_id (#3970)

[dotnet] Enable Custom Blocking Response tests (#3971)

[python] update flask base image (#3979)

[ruby] Skip failing test for APMAPI-1141 (#3980)

update weblog sampling_test use sampling rules instead of deprecated envvar (#3984)

[python] enable Python IAST+SSRF tests (#2512)

Signed-off-by: Juanjo Alvarez <juanjo.alvarezmartinez@datadoghq.com>

[python] IAST Header source in werkzeug 3.1 (#3991)

[python] switch flaky decorator for bug (#3990)

[python] update versions for major release (#3993)

[test optimization] Update config shape sent as telemetry (#3992)

Add profiling scenarios to Python onboarding tests (#3002)

Update changelog

re-enable aws integrations tests (#3733)

feat(tests/test_data_integrity): update test_datadog_external_env for Go v1.72.0 and forward (#3961)

[python] Avoid passing global tracer to pin in weblog apps (#4004)

All classes must declare feature ids (#4003)

Extend mypy scope (#4002)

Onboarding: bug marker profiling (#4005)

Docker SSI: fix scenario (#4006)

[ruby] Enable IP blocking tests for Ruby (#3937)

[nodejs] remove auto login event skip (#3998)

[NodeJS] skip failing baggage tests (#4015)

[python] fix 500 errors in sql queries (#3997)

Hotfix

Fix fuzzer

[python] use main again for dev branch (#4008)

Co-authored-by: erikayasuda <153395705+erikayasuda@users.noreply.github.com>
Co-authored-by: Charles de Beauchesne <charles.debeauchesne@datadoghq.com>

Revert agent dev fix (#4013)

[skipci] Update CODEOWNERS for static files (#4012)

[Java] Enable more easy wins (#4018)

[java] Bump GraalVM system test to JDK 22 (#4001)

[NodeJS] skip more failing baggage tests (#4021)

[Debugger] Update dotnet Exception Replay tests (#3974)

Test multiple rasp during one request (#3989)

Add test for location extended data (#3978)

Fix APPSEC_NO_STATS scenario name (#4019)

Avoid false XPASS on APPSEC_WAF_TELEMETRY (#4029)

[java] Enable Test_Blocking_strip_response_headers in some variants (#4033)

[java] Remove some outdated manifest entries (#4039)

[java] Fix xpass for Test_SecurityEvents_Appsec_Metastruct_Disabled (#4038)

Consolidate remote config tests into same directory/file (#4031)

[python] use last patch version of python for django weblogs (#4025)

crashtracking: assert si_signo is set to 11 (#4023)

class vs path
szegedi added a commit to DataDog/dd-trace-py that referenced this pull request Feb 20, 2025
@szegedi
ci: enable a profiling onboarding test scenario (#12229)
6bf9381 
## Overview:
Adds `SIMPLE_AUTO_INJECTION_PROFILING` scenario to the onboarding tests'
matrix, thus effectively enabling a simple profiling onboarding scenario
to run for each PR in addition to a simple tracing scenario. (When
`SCENARIO` is not specified, it defaults to `[
SIMPLE_INSTALLER_AUTO_INJECTION ]` so this override is additive.)

## Motivation:
Just as `SIMPLE_INSTALLER_AUTO_INJECTION` provides an early indication
that a PR is breaking *tracing* functionality with SSI on host and
container deployments, `SIMPLE_AUTO_INJECTION_PROFILING` does the same
for *profiling*.

This complements a broader set of tested scenarios that are executed as
part of the nightly auto-instrumentation CI, with Python profiling just
having been added into the matrix there with
DataDog/system-tests#3002.

JIRA: [PROF-11178]

## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met 
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)


[PROF-11178]:
https://datadoghq.atlassian.net/browse/PROF-11178?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
robertomonteromiguel pushed a commit that referenced this pull request Mar 17, 2025
@szegedi @robertomonteromiguel
Add profiling scenarios to Python onboarding tests (#3002)
1fb6622
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@robertomonteromiguel robertomonteromiguel robertomonteromiguel approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@szegedi @robertomonteromiguel