Start dropping python2 support #30035

chouquette · 2024-10-11T07:28:35Z

What does this PR do?

Start dropping python2 as we don't ship it with the agent anymore.
It handles:

the CI job parameters
rtloader
omnibus
The MSI package

Motivation

We don't need to clutter the code with various parameters or branches for something we don't support anymore.

There are most likely some missing pieces and a lot of remaining references to python2, but this serves as a good starting point

Describe how to test/QA your changes

I'm not sure this needs dedicated QA as long as the CI still passes since we're only removing code, but since the change is large, I can definitely convinced otherwise

Possible Drawbacks / Trade-offs

Additional Notes

This is probably better review commit by commit. The changes should be very close to atomic, and things can be split out in multiple PRs if we chose to.

Associated PRs:

cit-pr-commenter · 2024-10-11T07:57:37Z

Regression Detector

Regression Detector Results

Run ID: 8fa7ba51-edb2-4636-b05a-46f0c8360bce Metrics dashboard Target profiles

Baseline: 5f34bad
Comparison: 8ce88a9

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	+3.06	[+2.32, +3.79]	1	Logs
➖	pycheck_lots_of_tags	% cpu utilization	+0.78	[-1.75, +3.31]	1	Logs
➖	file_tree	memory utilization	+0.61	[+0.48, +0.73]	1	Logs
➖	idle	memory utilization	+0.37	[+0.33, +0.41]	1	Logs bounds checks dashboard
➖	quality_gate_idle	memory utilization	+0.26	[+0.22, +0.31]	1	Logs bounds checks dashboard
➖	file_to_blackhole_1000ms_latency	egress throughput	+0.24	[-0.25, +0.73]	1	Logs
➖	idle_all_features	memory utilization	+0.18	[+0.08, +0.28]	1	Logs bounds checks dashboard
➖	file_to_blackhole_500ms_latency	egress throughput	+0.06	[-0.19, +0.30]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	+0.00	[-0.07, +0.08]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.00	[-0.01, +0.01]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	-0.01	[-0.34, +0.32]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	-0.01	[-0.24, +0.21]	1	Logs
➖	otel_to_otel_logs	ingress throughput	-0.05	[-0.86, +0.76]	1	Logs
➖	file_to_blackhole_300ms_latency	egress throughput	-0.07	[-0.25, +0.11]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-0.12	[-0.20, -0.05]	1	Logs
➖	basic_py_check	% cpu utilization	-0.16	[-2.92, +2.60]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	-0.54	[-0.64, -0.44]	1	Logs bounds checks dashboard

Bounds Checks

perf	experiment	bounds_check_name	replicates_passed
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_300ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	idle	memory_usage	10/10
✅	idle_all_features	memory_usage	10/10
✅	quality_gate_idle	memory_usage	10/10
✅	quality_gate_idle_all_features	memory_usage	10/10

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CelianR

Good job! Added minor suggestions

rtloader/common/builtins/datadog_agent.c

tasks/agent.py

tasks/github_tasks.py

tasks/gotest.py

tasks/msi.py

tasks/trace_agent.py

tasks/system_probe.py

tasks/rtloader.py

tasks/process_agent.py

julien-lebot

There's some diff issue & the PR is missing the changes to the Python compression & decompression in the MSI installer.

tools/windows/DatadogAgentInstaller/WixSetup/Datadog Agent/AgentInstaller.cs

gjulianm

I have a suggestion for the tasks code

tasks/system_probe.py

chouquette · 2024-10-11T12:51:25Z

Python compression & decompression in the MSI installer.

I think I addressed this now, but if not I might need a few more pointers/help

agent-platform-auto-pr · 2024-10-11T12:57:18Z

Gitlab CI Configuration Changes

Modified Jobs

.agent_7_build

  .agent_7_build:
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    variables:
      AGENT_MAJOR_VERSION: 7
      FLAVOR: base
-     PYTHON_RUNTIMES: '3'

.agent_build_common

  .agent_build_common:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
-     --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+     --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+     --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+     --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    variables:
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi

.agent_build_common_dmg

  .agent_build_common_dmg:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    script:
    - echo "About to build for $RELEASE_VERSION"
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - mkdir -p $OMNIBUS_PACKAGE_DIR
    - "if [[ \"$(( RANDOM % 2 ))\" == \"1\" ]]; then\n  GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_1 key_b64) || exit $?; export GITHUB_KEY_B64\n  GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_1 app_id) || exit $?; export GITHUB_APP_ID\n  GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_1 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
      \  echo \"Using GitHub App instance 1\"\nelse\n  GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_2 key_b64) || exit $?; export GITHUB_KEY_B64\n  GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_2 app_id) || exit $?; export GITHUB_APP_ID\n  GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_2 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
      \  echo \"Using GitHub App instance 2\"\nfi\n"
    - $S3_CP_CMD $S3_ARTIFACTS_URI/agent-version.cache .
    - export VERSION_CACHE_CONTENT=$(cat agent-version.cache | base64 -)
    - python3 -m pip install -r tasks/libs/requirements-github.txt
    - inv -e github.trigger-macos --workflow-type "build" --datadog-agent-ref "$CI_COMMIT_SHA"
-     --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION" --python-runtimes
?                                                                                   ^^ ^  ---------
+     --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION" --destination
?                                                                                   ^^^ ^^^^^
-     "$PYTHON_RUNTIMES" --destination "$OMNIBUS_PACKAGE_DIR" --version-cache "$VERSION_CACHE_CONTENT"
-     --integrations-core-ref "$INTEGRATIONS_CORE_VERSION"
+     "$OMNIBUS_PACKAGE_DIR" --version-cache "$VERSION_CACHE_CONTENT" --integrations-core-ref
+     "$INTEGRATIONS_CORE_VERSION"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    timeout: 3h

.agent_build_script

  .agent_build_script:
  - echo "About to build for $RELEASE_VERSION"
  - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
  - rm -f modcache.tar.xz
  - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
  - rm -rf $OMNIBUS_PACKAGE_DIR/*
  - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
  - mkdir -p /tmp/system-probe
  - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
  - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
  - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
  - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
  - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
-   --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
-   --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-   --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+   --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+   --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+   --install-directory "$INSTALL_DIR"
  - ls -la $OMNIBUS_PACKAGE_DIR
  - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json

.common_build_oci

  .common_build_oci:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    script:
    - echo "About to build for $RELEASE_VERSION"
    - AGENT_VERSION="$(inv agent.version -u)-1" || exit $?
    - export INSTALL_DIR=/opt/datadog-packages/datadog-agent/"$AGENT_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
+     --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
-     --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --host-distribution=ociru --install-directory="$INSTALL_DIR"
+     --system-probe-bin=/tmp/system-probe --host-distribution=ociru --install-directory="$INSTALL_DIR"
?    +++++++++++++++++++++++++++++++++++++
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    variables:
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi

.dogstatsd_build_common

  .dogstatsd_build_common:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
-   - inv -e omnibus.build --release-version $RELEASE_VERSION_7 --major-version 7 --python-runtimes
?                                                                                   ^^^^^^  -------
+   - inv -e omnibus.build --release-version $RELEASE_VERSION_7 --major-version 7 --base-dir
?                                                                                   ^^^^ ++
-     3 --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
?    -------------
+     $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
      --target-project dogstatsd
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    variables:
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi

.heroku_build_base

  .heroku_build_base:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-x64
    - go_deps
    - generate_minimized_btfs_x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - echo "About to build for $RELEASE_VERSION"
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - set +x
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
      || exit $?; export DEB_SIGNING_PASSPHRASE
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
+     --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+     --system-probe-bin=/tmp/system-probe --flavor heroku
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
-     --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor heroku
    - ls -la $OMNIBUS_PACKAGE_DIR
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-heroku-agent_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-heroku-agent-dbg_*_${PACKAGE_ARCH}.deb
      $S3_ARTIFACTS_URI/$DESTINATION_DBG_DEB
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    variables:
      DD_CC: x86_64-unknown-linux-gnu-gcc
      DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
      DD_CXX: x86_64-unknown-linux-gnu-g++
      DD_PKG_ARCH: x86_64
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: amd64

.integration_tests_windows_base

  .integration_tests_windows_base:
    before_script:
    - $tmpfile = [System.IO.Path]::GetTempFileName()
    - (& "$CI_PROJECT_DIR\tools\ci\fetch_secret.ps1" -parameterName "$Env:VCPKG_BLOB_SAS_URL"
      -tempFile "$tmpfile")
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    - $vcpkgBlobSaSUrl=$(cat "$tmpfile")
    - Remove-Item "$tmpfile"
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - $ErrorActionPreference = "Stop"
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e CI_JOB_URL="${CI_JOB_URL}"
      -e CI_JOB_NAME="${CI_JOB_NAME}" -e CI_PIPELINE_ID="${CI_PIPELINE_ID}" -e CI_PROJECT_NAME="${CI_PROJECT_NAME}"
+     -e AWS_NETWORKING=true -e GOMODCACHE="c:\modcache" -e VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}"
-     -e AWS_NETWORKING=true -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache"
-     -e VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}" -e PIP_INDEX_URL=${PIP_INDEX_URL}
-     486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+     -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
?    ++++++++++++++++++++++++++++++++++
      c:\mnt\tasks\winbuildscripts\integrationtests.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: integration_test
    tags:
    - runner:windows-docker
    - windowsversion:1809

.lint_macos_gitlab

  .lint_macos_gitlab:
    before_script:
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      echo "Don''t forget to regularly delete unused versions. Here are the installed
      versions and their memory usage on the runner:"
  
      du -sh $HOME/.gimme/versions/*
  
      '
    - "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
      \ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
      \ \"$PYTHON_VERSION\" ]; then\n  echo \"Python version $PYTHON_VERSION is different\
      \ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
      \  pyenv uninstall -f datadog-agent\n  echo \"Installing Python $PYTHON_REPO_VERSION...\"\
      \n  pyenv virtualenv 3.11.8 datadog-agent\n  pyenv activate datadog-agent\nelse\n\
      \  echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
      \ Keeping the existing environment.\"\nfi\n"
    - python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
    - pyenv rehash
-   - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+   - inv -e rtloader.make
    - inv -e rtloader.install
    - inv -e install-tools
    needs:
    - go_deps
    - go_tools_deps
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e linter.go --cpus 12 --debug --timeout 60
    stage: lint
-   variables:
-     PYTHON_RUNTIMES: '3'

.lint_windows_base

  .lint_windows_base:
    needs:
    - go_deps
    - go_tools_deps
    script:
    - $ErrorActionPreference = "Stop"
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - 'docker run --rm -m 16384M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
      -e AWS_NETWORKING=true -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME}
-     -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:$Env:DATADOG_AGENT_WINBUILDIMAGES
?    ----------------------------------
+     -e GOMODCACHE="c:\modcache" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:$Env:DATADOG_AGENT_WINBUILDIMAGES
      c:\mnt\tasks\winbuildscripts\lint.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: lint
    tags:
    - runner:windows-docker
    - windowsversion:1809

.linux_lint

  .linux_lint:
    needs:
    - go_deps
    - go_tools_deps
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "3"
?                                                              ----------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e install-tools
    - inv -e linter.go --cpus $KUBERNETES_CPU_REQUEST --debug $FLAVORS $EXTRA_OPTS
    stage: lint
    variables:
      FLAVORS: --flavor base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi

.linux_tests

  .linux_tests:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit: '**/junit-out-*.xml'
      when: always
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e install-tools
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - inv -e sds.build-library
    - inv -e agent.build
-   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
?                                                                           ^^^^ ^^ ^^^^^ -
+   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
?                                                                           ^ ^^ ^^
-     "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+     --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
-     $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
?    ------------------
+     --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
?                                                                           ++++++++++++++
-     --test-washer
    stage: source_test
    variables:
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      FLAVORS: --flavor base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi
      TEST_OUTPUT_FILE: test_output.json

.macos_gitlab

  .macos_gitlab:
    before_script:
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      echo "Don''t forget to regularly delete unused versions. Here are the installed
      versions and their memory usage on the runner:"
  
      du -sh $HOME/.gimme/versions/*
  
      '
    - "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
      \ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
      \ \"$PYTHON_VERSION\" ]; then\n  echo \"Python version $PYTHON_VERSION is different\
      \ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
      \  pyenv uninstall -f datadog-agent\n  echo \"Installing Python $PYTHON_REPO_VERSION...\"\
      \n  pyenv virtualenv 3.11.8 datadog-agent\n  pyenv activate datadog-agent\nelse\n\
      \  echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
      \ Keeping the existing environment.\"\nfi\n"
    - python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
    - pyenv rehash
-   - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+   - inv -e rtloader.make
    - inv -e rtloader.install
    - inv -e install-tools
-   variables:
-     PYTHON_RUNTIMES: '3'

.ot_agent_7_build

  .ot_agent_7_build:
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    variables:
      AGENT_MAJOR_VERSION: 7
      FLAVOR: ot
-     PYTHON_RUNTIMES: '3'

.rtloader_tests

  .rtloader_tests:
    before_script:
    - source /root/.bashrc && conda activate $CONDA_ENV
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
?                                                              -------------------------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e rtloader.format --raise-if-changed
    - inv -e rtloader.test
    needs:
    - go_deps
    script:
    - '# Skipping go tests'
    stage: source_test

.tests_macos_gitlab

  .tests_macos_gitlab:
    allow_failure: true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit: '**/junit-out-*.xml'
      when: always
    before_script:
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      echo "Don''t forget to regularly delete unused versions. Here are the installed
      versions and their memory usage on the runner:"
  
      du -sh $HOME/.gimme/versions/*
  
      '
    - "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
      \ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
      \ \"$PYTHON_VERSION\" ]; then\n  echo \"Python version $PYTHON_VERSION is different\
      \ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
      \  pyenv uninstall -f datadog-agent\n  echo \"Installing Python $PYTHON_REPO_VERSION...\"\
      \n  pyenv virtualenv 3.11.8 datadog-agent\n  pyenv activate datadog-agent\nelse\n\
      \  echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
      \ Keeping the existing environment.\"\nfi\n"
    - python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
    - pyenv rehash
-   - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+   - inv -e rtloader.make
    - inv -e rtloader.install
    - inv -e install-tools
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
+   - inv -e test --rerun-fails=2 --race --profile --cpus 12 --save-result-json $TEST_OUTPUT_FILE
+     --junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer
-   - inv -e test --rerun-fails=2 --python-runtimes $PYTHON_RUNTIMES --race --profile
-     --cpus 12 --save-result-json $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz"
-     $FAST_TESTS_FLAG --test-washer
    - inv -e invoke-unit-tests
    stage: source_test
    variables:
-     PYTHON_RUNTIMES: '3'
      TEST_OUTPUT_FILE: test_output.json

.tests_windows_base

  .tests_windows_base:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        junit: '**/junit-out-*.xml'
      when: always
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    script:
    - $ErrorActionPreference = "Stop"
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - $FAST_TESTS_FLAG=""
    - If ($FAST_TESTS -eq "true") { $FAST_TESTS_FLAG="--only-impacted-packages" }
    - 'docker run --rm -m 16384M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
      -e DD_ENV=prod -e GITLAB_CI=true -e CI_JOB_URL="${CI_JOB_URL}" -e CI_JOB_NAME="${CI_JOB_NAME}"
      -e CI_JOB_ID="${CI_JOB_ID}" -e CI_PIPELINE_ID="${CI_PIPELINE_ID}" -e CI_PIPELINE_SOURCE="${CI_PIPELINE_SOURCE}"
      -e CI_REPOSITORY_URL="${CI_REPOSITORY_URL}" -e CI_COMMIT_SHA="${CI_COMMIT_SHA}"
      -e CI_PROJECT_NAME="${CI_PROJECT_NAME}" -e CI_COMMIT_REF_NAME="${CI_COMMIT_REF_NAME}"
-     -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e PY_RUNTIMES="${PYTHON_RUNTIMES}"
-     -e GOMODCACHE="c:\modcache" -e JUNIT_TAR="c:\mnt\junit-${CI_JOB_NAME}.tgz" -e
-     PIP_INDEX_URL="${PIP_INDEX_URL}" -e TEST_OUTPUT_FILE="${TEST_OUTPUT_FILE}" -e
-     EXTRA_OPTS="${FAST_TESTS_FLAG}" -e TEST_WASHER=true -e GO_TEST_SKIP_FLAKE="${GO_TEST_SKIP_FLAKE}"
+     -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e GOMODCACHE="c:\modcache"
+     -e JUNIT_TAR="c:\mnt\junit-${CI_JOB_NAME}.tgz" -e PIP_INDEX_URL="${PIP_INDEX_URL}"
+     -e TEST_OUTPUT_FILE="${TEST_OUTPUT_FILE}" -e EXTRA_OPTS="${FAST_TESTS_FLAG}" -e
+     TEST_WASHER=true -e GO_TEST_SKIP_FLAKE="${GO_TEST_SKIP_FLAKE}" -e API_KEY_ORG2="${API_KEY_ORG2}"
-     -e API_KEY_ORG2="${API_KEY_ORG2}" -e CODECOV_TOKEN="${CODECOV_TOKEN}" -e S3_PERMANENT_ARTIFACTS_URI="${S3_PERMANENT_ARTIFACTS_URI}"
?    ----------------------------------
+     -e CODECOV_TOKEN="${CODECOV_TOKEN}" -e S3_PERMANENT_ARTIFACTS_URI="${S3_PERMANENT_ARTIFACTS_URI}"
      -e COVERAGE_CACHE_FLAG="${COVERAGE_CACHE_FLAG}" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
      c:\mnt\tasks\winbuildscripts\unittests.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: source_test
    tags:
    - runner:windows-docker
    - windowsversion:1809
    variables:
      TEST_OUTPUT_FILE: test_output.json

.tests_windows_secagent

  .tests_windows_secagent:
    artifacts:
      paths:
      - $DD_AGENT_TESTING_DIR/site-cookbooks/dd-security-agent-check/files
      when: always
    needs:
    - go_deps
    - go_tools_deps
    script:
    - $ErrorActionPreference = "Stop"
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e AWS_NETWORKING=true
      -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e SIGN_WINDOWS_DD_WCS=true
-     -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL}
-     486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+     -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
?    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      c:\mnt\tasks\winbuildscripts\secagent.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: source_test
    tags:
    - runner:windows-docker
    - windowsversion:1809

.tests_windows_sysprobe

  .tests_windows_sysprobe:
    artifacts:
      paths:
      - $DD_AGENT_TESTING_DIR/site-cookbooks/dd-system-probe-check/files
      when: always
    needs:
    - go_deps
    - go_tools_deps
    script:
    - $ErrorActionPreference = "Stop"
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e AWS_NETWORKING=true
      -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e SIGN_WINDOWS_DD_WCS=true
-     -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL}
-     486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+     -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
?    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      c:\mnt\tasks\winbuildscripts\sysprobe.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: source_test
    tags:
    - runner:windows-docker
    - windowsversion:1809

.windows_main_agent_base

  .windows_main_agent_base:
    after_script:
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    artifacts:
      expire_in: 2 weeks
      paths:
      - omnibus/pkg
    needs:
    - go_mod_tidy_check
    - go_deps
    script:
    - $ErrorActionPreference = 'Stop'
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - if (Test-Path omnibus\pkg) { remove-item -recurse -force omnibus\pkg }
    - mkdir omnibus\pkg
    - 'docker run --rm -m 8192M -v "$(Get-Location):c:\mnt" -e CI_JOB_ID=${CI_JOB_ID}
      -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH}
      -e CI_JOB_NAME_SLUG=${CI_JOB_NAME_SLUG} -e CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME}
      -e OMNIBUS_TARGET=${OMNIBUS_TARGET} -e WINDOWS_BUILDER=true -e RELEASE_VERSION="$RELEASE_VERSION"
-     -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
?                                           ----------------------------------
+     -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
      -e GOMODCACHE="c:\modcache" -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true
      -e TARGET_ARCH="$ARCH" -e DEBUG_CUSTOMACTION="$DEBUG_CUSTOMACTION" -e BUCKET_BRANCH="$BUCKET_BRANCH"
      -e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET" -e USE_S3_CACHING="$USE_S3_CACHING"
      -e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET" -e GO_VERSION_CHECK="true"
      -e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG} -e PIP_INDEX_URL=${PIP_INDEX_URL}
      -e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
      c:\mnt\tasks\winbuildscripts\buildwin.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    - get-childitem omnibus\pkg
    - Invoke-Expression "$S3_CP_CMD omnibus\pkg\version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json"
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: package_build
    tags:
    - runner:windows-docker
    - windowsversion:1809
    variables:
      OMNIBUS_TARGET: main

.windows_msi_base

  .windows_msi_base:
    after_script:
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    artifacts:
      expire_in: 2 weeks
      paths:
      - omnibus/pkg
    needs:
    - go_mod_tidy_check
    - go_deps
    script:
    - $ErrorActionPreference = 'Stop'
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - if (Test-Path omnibus\pkg) { remove-item -recurse -force omnibus\pkg }
    - mkdir omnibus\pkg
    - 'docker run --rm -m 8192M -v "$(Get-Location):c:\mnt" -e CI_JOB_ID=${CI_JOB_ID}
      -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH}
      -e CI_JOB_NAME_SLUG=${CI_JOB_NAME_SLUG} -e CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME}
      -e OMNIBUS_TARGET=${OMNIBUS_TARGET} -e WINDOWS_BUILDER=true -e RELEASE_VERSION="$RELEASE_VERSION"
-     -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
?                                           ----------------------------------
+     -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
      -e GOMODCACHE="c:\modcache" -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true
      -e TARGET_ARCH="$ARCH" -e DEBUG_CUSTOMACTION="$DEBUG_CUSTOMACTION" -e BUCKET_BRANCH="$BUCKET_BRANCH"
      -e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET" -e USE_S3_CACHING="$USE_S3_CACHING"
      -e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET" -e GO_VERSION_CHECK="true"
      -e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG} -e PIP_INDEX_URL=${PIP_INDEX_URL}
      -e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
      c:\mnt\tasks\winbuildscripts\buildwin.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    - get-childitem omnibus\pkg
    - Invoke-Expression "$S3_CP_CMD omnibus\pkg\version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json"
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: package_build
    tags:
    - runner:windows-docker
    - windowsversion:1809

agent_dmg-x64-a7

  agent_dmg-x64-a7:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - changes:
        compare_to: main
        paths:
        - comp/core/gui/guiimpl/systray/**/*
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - changes:
        compare_to: main
        paths:
        - omnibus/**/*
        - .gitlab-ci.yml
        - release.json
        - .gitlab/package_build/**/*
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
    - if: $CI_COMMIT_BRANCH == "main" || $DEPLOY_AGENT == "true" || $RUN_ALL_BUILDS
        == "true" || $DDR_WORKFLOW_ID != null
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - allow_failure: true
      when: manual
    script:
    - echo "About to build for $RELEASE_VERSION"
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - mkdir -p $OMNIBUS_PACKAGE_DIR
    - "if [[ \"$(( RANDOM % 2 ))\" == \"1\" ]]; then\n  GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_1 key_b64) || exit $?; export GITHUB_KEY_B64\n  GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_1 app_id) || exit $?; export GITHUB_APP_ID\n  GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_1 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
      \  echo \"Using GitHub App instance 1\"\nelse\n  GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_2 key_b64) || exit $?; export GITHUB_KEY_B64\n  GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_2 app_id) || exit $?; export GITHUB_APP_ID\n  GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_2 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
      \  echo \"Using GitHub App instance 2\"\nfi\n"
    - $S3_CP_CMD $S3_ARTIFACTS_URI/agent-version.cache .
    - export VERSION_CACHE_CONTENT=$(cat agent-version.cache | base64 -)
    - python3 -m pip install -r tasks/libs/requirements-github.txt
    - inv -e github.trigger-macos --workflow-type "build" --datadog-agent-ref "$CI_COMMIT_SHA"
-     --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION" --python-runtimes
?                                                                                   ^^ ^  ---------
+     --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION" --destination
?                                                                                   ^^^ ^^^^^
-     "$PYTHON_RUNTIMES" --destination "$OMNIBUS_PACKAGE_DIR" --version-cache "$VERSION_CACHE_CONTENT"
-     --integrations-core-ref "$INTEGRATIONS_CORE_VERSION"
+     "$OMNIBUS_PACKAGE_DIR" --version-cache "$VERSION_CACHE_CONTENT" --integrations-core-ref
+     "$INTEGRATIONS_CORE_VERSION"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    timeout: 6h
    variables:
      AGENT_MAJOR_VERSION: 7
-     PYTHON_RUNTIMES: '3'

agent_heroku_deb-x64-a7

  agent_heroku_deb-x64-a7:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-x64
    - go_deps
    - generate_minimized_btfs_x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - echo "About to build for $RELEASE_VERSION"
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - set +x
    - printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
      --batch
    - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
      needs gitlab runner restart"; exit $EXIT; fi
    - DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
      || exit $?; export DEB_SIGNING_PASSPHRASE
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
+     --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+     --system-probe-bin=/tmp/system-probe --flavor heroku
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
-     --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor heroku
    - ls -la $OMNIBUS_PACKAGE_DIR
    - curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
      | tar -xz -C /usr/local/bin dd-pkg
    - find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
    - "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n  find $OMNIBUS_PACKAGE_DIR\
      \ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
      \ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-heroku-agent_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-heroku-agent-dbg_*_${PACKAGE_ARCH}.deb
      $S3_ARTIFACTS_URI/$DESTINATION_DBG_DEB
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_CC: x86_64-unknown-linux-gnu-gcc
      DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
      DD_CXX: x86_64-unknown-linux-gnu-g++
      DD_PKG_ARCH: x86_64
      DESTINATION_DBG_DEB: datadog-heroku-agent-dbg_7_amd64.deb
      DESTINATION_DEB: datadog-heroku-agent_7_amd64.deb
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: amd64
-     PYTHON_RUNTIMES: '3'
      RELEASE_VERSION: $RELEASE_VERSION_7

datadog-agent-7-arm64

  datadog-agent-7-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-23-arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-arm64
    - go_deps
    - generate_minimized_btfs_arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
-     --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+     --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+     --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+     --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_CC: aarch64-unknown-linux-gnu-gcc
      DD_CMAKE_TOOLCHAIN: /opt/cmake/aarch64-unknown-linux-gnu.toolchain.cmake
      DD_CXX: aarch64-unknown-linux-gnu-g++
      FLAVOR: base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: arm64
-     PYTHON_RUNTIMES: '3'

datadog-agent-7-x64

  datadog-agent-7-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-x64
    - go_deps
    - generate_minimized_btfs_x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
-     --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+     --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+     --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+     --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_CC: x86_64-unknown-linux-gnu-gcc
      DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
      DD_CXX: x86_64-unknown-linux-gnu-g++
      FLAVOR: base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: amd64
-     PYTHON_RUNTIMES: '3'

datadog-agent-7-x64-custom-path-test

  datadog-agent-7-x64-custom-path-test:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-x64
    - go_deps
    - generate_minimized_btfs_x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - mkdir /custom
    - export CONFIG_DIR="/custom"
    - export INSTALL_DIR="/custom/datadog-agent"
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
-     --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+     --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+     --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+     --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    - ls -la $OMNIBUS_PACKAGE_DIR
    - ls -la $INSTALL_DIR
    - ls -la /custom/etc
    - (ls -la /opt/datadog-agent 2>/dev/null && exit 1) || echo "/opt/datadog-agent
      has correctly not been generated"
    - (ls -la /etc/datadog-agent 2>/dev/null && exit 1) || echo "/etc/datadog-agent
      has correctly not been generated"
    stage: package_build
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_CC: x86_64-unknown-linux-gnu-gcc
      DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
      DD_CXX: x86_64-unknown-linux-gnu-g++
      FLAVOR: base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: amd64
-     PYTHON_RUNTIMES: '3'

datadog-agent-oci-arm64-a7

  datadog-agent-oci-arm64-a7:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-23-arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-arm64
    - go_deps
    - generate_minimized_btfs_arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - AGENT_VERSION="$(inv agent.version -u)-1" || exit $?
    - export INSTALL_DIR=/opt/datadog-packages/datadog-agent/"$AGENT_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
+     --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
-     --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --host-distribution=ociru --install-directory="$INSTALL_DIR"
+     --system-probe-bin=/tmp/system-probe --host-distribution=ociru --install-directory="$INSTALL_DIR"
?    +++++++++++++++++++++++++++++++++++++
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_CC: aarch64-unknown-linux-gnu-gcc
      DD_CMAKE_TOOLCHAIN: /opt/cmake/aarch64-unknown-linux-gnu.toolchain.cmake
      DD_CXX: aarch64-unknown-linux-gnu-g++
      DESTINATION_OCI: datadog-agent-7-remote-updater-arm64.tar.xz
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: arm64
-     PYTHON_RUNTIMES: '3'

datadog-agent-oci-x64-a7

  datadog-agent-oci-x64-a7:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-x64
    - go_deps
    - generate_minimized_btfs_x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - AGENT_VERSION="$(inv agent.version -u)-1" || exit $?
    - export INSTALL_DIR=/opt/datadog-packages/datadog-agent/"$AGENT_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
+     --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
-     --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --host-distribution=ociru --install-directory="$INSTALL_DIR"
+     --system-probe-bin=/tmp/system-probe --host-distribution=ociru --install-directory="$INSTALL_DIR"
?    +++++++++++++++++++++++++++++++++++++
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_CC: x86_64-unknown-linux-gnu-gcc
      DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
      DD_CXX: x86_64-unknown-linux-gnu-g++
      DESTINATION_OCI: datadog-agent-7-remote-updater-amd64.tar.xz
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: amd64
-     PYTHON_RUNTIMES: '3'

datadog-ot-agent-7-arm64

  datadog-ot-agent-7-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-23-arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-arm64
    - go_deps
    - generate_minimized_btfs_arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
-     --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+     --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+     --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+     --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_CC: aarch64-unknown-linux-gnu-gcc
      DD_CMAKE_TOOLCHAIN: /opt/cmake/aarch64-unknown-linux-gnu.toolchain.cmake
      DD_CXX: aarch64-unknown-linux-gnu-g++
      FLAVOR: ot
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: arm64
-     PYTHON_RUNTIMES: '3'

datadog-ot-agent-7-x64

  datadog-ot-agent-7-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-x64
    - go_deps
    - generate_minimized_btfs_x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
-     --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
-     --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+     --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+     --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+     --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      DD_CC: x86_64-unknown-linux-gnu-gcc
      DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
      DD_CXX: x86_64-unknown-linux-gnu-g++
      FLAVOR: ot
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: amd64
-     PYTHON_RUNTIMES: '3'

dogstatsd-arm64

  dogstatsd-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-23-arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_dogstatsd-binary_arm64
    - go_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
-   - inv -e omnibus.build --release-version $RELEASE_VERSION_7 --major-version 7 --python-runtimes
?                                                                                   ^^^^^^  -------
+   - inv -e omnibus.build --release-version $RELEASE_VERSION_7 --major-version 7 --base-dir
?                                                                                   ^^^^ ++
-     3 --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
?    -------------
+     $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
      --target-project dogstatsd
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:arm64
    variables:
      DD_CC: aarch64-unknown-linux-gnu-gcc
      DD_CXX: aarch64-unknown-linux-gnu-g++
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi

dogstatsd-x64

  dogstatsd-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_dogstatsd-binary_x64
    - go_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
-   - inv -e omnibus.build --release-version $RELEASE_VERSION_7 --major-version 7 --python-runtimes
?                                                                                   ^^^^^^  -------
+   - inv -e omnibus.build --release-version $RELEASE_VERSION_7 --major-version 7 --base-dir
?                                                                                   ^^^^ ++
-     3 --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
?    -------------
+     $OMNIBUS_BASE_DIR  ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
      --target-project dogstatsd
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    variables:
      DD_CC: x86_64-unknown-linux-gnu-gcc
      DD_CXX: x86_64-unknown-linux-gnu-g++
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi

integration_tests_windows-x64

  integration_tests_windows-x64:
    before_script:
    - $tmpfile = [System.IO.Path]::GetTempFileName()
    - (& "$CI_PROJECT_DIR\tools\ci\fetch_secret.ps1" -parameterName "$Env:VCPKG_BLOB_SAS_URL"
      -tempFile "$tmpfile")
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    - $vcpkgBlobSaSUrl=$(cat "$tmpfile")
    - Remove-Item "$tmpfile"
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - $ErrorActionPreference = "Stop"
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e CI_JOB_URL="${CI_JOB_URL}"
      -e CI_JOB_NAME="${CI_JOB_NAME}" -e CI_PIPELINE_ID="${CI_PIPELINE_ID}" -e CI_PROJECT_NAME="${CI_PROJECT_NAME}"
+     -e AWS_NETWORKING=true -e GOMODCACHE="c:\modcache" -e VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}"
-     -e AWS_NETWORKING=true -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache"
-     -e VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}" -e PIP_INDEX_URL=${PIP_INDEX_URL}
-     486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+     -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
?    ++++++++++++++++++++++++++++++++++
      c:\mnt\tasks\winbuildscripts\integrationtests.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: integration_test
    tags:
    - runner:windows-docker
    - windowsversion:1809
    variables:
      ARCH: x64
-     PYTHON_RUNTIMES: 3

lint_flavor_dogstatsd_linux-x64

  lint_flavor_dogstatsd_linux-x64:
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "3"
?                                                              ----------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e install-tools
    - inv -e linter.go --cpus $KUBERNETES_CPU_REQUEST --debug $FLAVORS $EXTRA_OPTS
    stage: lint
    tags:
    - arch:amd64
    variables:
      FLAVORS: --flavor dogstatsd
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi

lint_flavor_heroku_linux-x64

  lint_flavor_heroku_linux-x64:
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "3"
?                                                              ----------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e install-tools
    - inv -e linter.go --cpus $KUBERNETES_CPU_REQUEST --debug $FLAVORS $EXTRA_OPTS
    stage: lint
    tags:
    - arch:amd64
    variables:
      FLAVORS: --flavor heroku
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi

lint_flavor_iot_linux-x64

  lint_flavor_iot_linux-x64:
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "3"
?                                                              ----------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e install-tools
    - inv -e linter.go --cpus $KUBERNETES_CPU_REQUEST --debug $FLAVORS $EXTRA_OPTS
    stage: lint
    tags:
    - arch:amd64
    variables:
      FLAVORS: --flavor iot
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi

lint_linux-arm64

  lint_linux-arm64:
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "3"
?                                                              ----------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e install-tools
    - inv -e linter.go --cpus $KUBERNETES_CPU_REQUEST --debug $FLAVORS $EXTRA_OPTS
    stage: lint
    tags:
    - arch:arm64
    variables:
      FLAVORS: --flavor base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi

lint_linux-x64

  lint_linux-x64:
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "3"
?                                                              ----------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e install-tools
    - inv -e linter.go --cpus $KUBERNETES_CPU_REQUEST --debug $FLAVORS $EXTRA_OPTS
    stage: lint
    tags:
    - arch:amd64
    variables:
      FLAVORS: --flavor base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi

lint_macos_gitlab_amd64

  lint_macos_gitlab_amd64:
    before_script:
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      echo "Don''t forget to regularly delete unused versions. Here are the installed
      versions and their memory usage on the runner:"
  
      du -sh $HOME/.gimme/versions/*
  
      '
    - "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
      \ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
      \ \"$PYTHON_VERSION\" ]; then\n  echo \"Python version $PYTHON_VERSION is different\
      \ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
      \  pyenv uninstall -f datadog-agent\n  echo \"Installing Python $PYTHON_REPO_VERSION...\"\
      \n  pyenv virtualenv 3.11.8 datadog-agent\n  pyenv activate datadog-agent\nelse\n\
      \  echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
      \ Keeping the existing environment.\"\nfi\n"
    - python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
    - pyenv rehash
-   - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+   - inv -e rtloader.make
    - inv -e rtloader.install
    - inv -e install-tools
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e linter.go --cpus 12 --debug --timeout 60
    stage: lint
    tags:
    - macos:monterey-amd64
    - specific:true
-   variables:
-     PYTHON_RUNTIMES: '3'

lint_macos_gitlab_arm64

  lint_macos_gitlab_arm64:
    allow_failure: true
    before_script:
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      echo "Don''t forget to regularly delete unused versions. Here are the installed
      versions and their memory usage on the runner:"
  
      du -sh $HOME/.gimme/versions/*
  
      '
    - "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
      \ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
      \ \"$PYTHON_VERSION\" ]; then\n  echo \"Python version $PYTHON_VERSION is different\
      \ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
      \  pyenv uninstall -f datadog-agent\n  echo \"Installing Python $PYTHON_REPO_VERSION...\"\
      \n  pyenv virtualenv 3.11.8 datadog-agent\n  pyenv activate datadog-agent\nelse\n\
      \  echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
      \ Keeping the existing environment.\"\nfi\n"
    - python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
    - pyenv rehash
-   - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+   - inv -e rtloader.make
    - inv -e rtloader.install
    - inv -e install-tools
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - allow_failure: true
      when: manual
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e linter.go --cpus 12 --debug --timeout 60
    stage: lint
    tags:
    - macos:monterey-arm64
    - specific:true
-   variables:
-     PYTHON_RUNTIMES: '3'

lint_windows-x64

  lint_windows-x64:
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - $ErrorActionPreference = "Stop"
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - 'docker run --rm -m 16384M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
      -e AWS_NETWORKING=true -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME}
-     -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:$Env:DATADOG_AGENT_WINBUILDIMAGES
?    ----------------------------------
+     -e GOMODCACHE="c:\modcache" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:$Env:DATADOG_AGENT_WINBUILDIMAGES
      c:\mnt\tasks\winbuildscripts\lint.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: lint
    tags:
    - runner:windows-docker
    - windowsversion:1809
    timeout: 1h30m
    variables:
      ARCH: x64
-     PYTHON_RUNTIMES: 3

tests_deb-arm64-py3

  tests_deb-arm64-py3:
    after_script:
    - CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
      $?; export CODECOV_TOKEN
    - inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - source /root/.bashrc && conda activate $CONDA_ENV
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
?                                                              -------------------------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e rtloader.format --raise-if-changed
    - inv -e rtloader.test
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e install-tools
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - inv -e sds.build-library
    - inv -e agent.build
-   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
?                                                                           ^^^^ ^^ ^^^^^ -
+   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
?                                                                           ^ ^^ ^^
-     "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+     --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
-     $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
?    ------------------
+     --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
?                                                                           ++++++++++++++
-     --test-washer
    stage: source_test
    tags:
    - arch:arm64
    variables:
      CONDA_ENV: ddpy3
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      FLAVORS: --flavor base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi
-     PYTHON_RUNTIMES: '3'
      TEST_OUTPUT_FILE: test_output.json

tests_deb-x64-py3

  tests_deb-x64-py3:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    - CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
      $?; export CODECOV_TOKEN
    - inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - source /root/.bashrc && conda activate $CONDA_ENV
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
?                                                              -------------------------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e rtloader.format --raise-if-changed
    - inv -e rtloader.test
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e install-tools
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - inv -e sds.build-library
    - inv -e agent.build
-   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
?                                                                           ^^^^ ^^ ^^^^^ -
+   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
?                                                                           ^ ^^ ^^
-     "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+     --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
-     $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
?    ------------------
+     --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
?                                                                           ++++++++++++++
-     --test-washer
    stage: source_test
    tags:
    - arch:amd64
    variables:
      CONDA_ENV: ddpy3
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      FLAVORS: --flavor base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi
-     PYTHON_RUNTIMES: '3'
      TEST_OUTPUT_FILE: test_output.json

tests_flavor_dogstatsd_deb-x64

  tests_flavor_dogstatsd_deb-x64:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - source /root/.bashrc && conda activate $CONDA_ENV
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
?                                                              -------------------------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e rtloader.format --raise-if-changed
    - inv -e rtloader.test
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e install-tools
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - inv -e sds.build-library
    - inv -e agent.build
-   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
?                                                                           ^^^^ ^^ ^^^^^ -
+   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
?                                                                           ^ ^^ ^^
-     "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+     --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
-     $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
?    ------------------
+     --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
?                                                                           ++++++++++++++
-     --test-washer
    stage: source_test
    tags:
    - arch:amd64
    variables:
      CONDA_ENV: ddpy3
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      FLAVORS: --flavor dogstatsd
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi
-     PYTHON_RUNTIMES: '3'
      TEST_OUTPUT_FILE: test_output.json

tests_flavor_heroku_deb-x64

  tests_flavor_heroku_deb-x64:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - source /root/.bashrc && conda activate $CONDA_ENV
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
?                                                              -------------------------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e rtloader.format --raise-if-changed
    - inv -e rtloader.test
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e install-tools
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - inv -e sds.build-library
    - inv -e agent.build
-   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
?                                                                           ^^^^ ^^ ^^^^^ -
+   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
?                                                                           ^ ^^ ^^
-     "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+     --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
-     $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
?    ------------------
+     --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
?                                                                           ++++++++++++++
-     --test-washer
    stage: source_test
    tags:
    - arch:amd64
    variables:
      CONDA_ENV: ddpy3
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      FLAVORS: --flavor heroku
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi
-     PYTHON_RUNTIMES: '3'
      TEST_OUTPUT_FILE: test_output.json

tests_flavor_iot_deb-x64

  tests_flavor_iot_deb-x64:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - source /root/.bashrc && conda activate $CONDA_ENV
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
?                                                              -------------------------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e rtloader.format --raise-if-changed
    - inv -e rtloader.test
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e install-tools
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - inv -e sds.build-library
    - inv -e agent.build
-   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
?                                                                           ^^^^ ^^ ^^^^^ -
+   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
?                                                                           ^ ^^ ^^
-     "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+     --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
-     $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
?    ------------------
+     --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
?                                                                           ++++++++++++++
-     --test-washer
    stage: source_test
    tags:
    - arch:amd64
    variables:
      CONDA_ENV: ddpy3
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      FLAVORS: --flavor iot
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi
-     PYTHON_RUNTIMES: '3'
      TEST_OUTPUT_FILE: test_output.json

tests_macos

  tests_macos:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh "junit-*-repacked.tgz"
    artifacts:
      expire_in: 2 weeks
      paths:
      - test_output.json
      - junit-*-repacked.tgz
      reports:
        junit:
        - '**/junit-out-*.xml'
      when: always
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - setup_agent_version
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
    - if: $CI_COMMIT_BRANCH == "main" || $DEPLOY_AGENT == "true" || $RUN_ALL_BUILDS
        == "true" || $DDR_WORKFLOW_ID != null
    - if: $RUN_UNIT_TESTS == "on"
    - changes:
        compare_to: main
        paths:
        - pkg/fleet/**/*
      variables:
        FAST_TESTS: 'true'
    - allow_failure: true
      when: manual
    script:
    - "if [[ \"$(( RANDOM % 2 ))\" == \"1\" ]]; then\n  GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_1 key_b64) || exit $?; export GITHUB_KEY_B64\n  GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_1 app_id) || exit $?; export GITHUB_APP_ID\n  GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_1 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
      \  echo \"Using GitHub App instance 1\"\nelse\n  GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_2 key_b64) || exit $?; export GITHUB_KEY_B64\n  GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_2 app_id) || exit $?; export GITHUB_APP_ID\n  GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
      \ $MACOS_GITHUB_APP_2 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
      \  echo \"Using GitHub App instance 2\"\nfi\n"
    - $S3_CP_CMD $S3_ARTIFACTS_URI/agent-version.cache .
    - export VERSION_CACHE_CONTENT=$(cat agent-version.cache | base64 -)
    - python3 -m pip install -r tasks/libs/requirements-github.txt
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" = "true" ]]; then FAST_TESTS_FLAG="--fast-tests true"; fi
    - inv -e github.trigger-macos --workflow-type "test" --datadog-agent-ref "$CI_COMMIT_SHA"
+     --version-cache "$VERSION_CACHE_CONTENT" $FAST_TESTS_FLAG --test-washer $COVERAGE_CACHE_FLAG
-     --python-runtimes "$PYTHON_RUNTIMES" --version-cache "$VERSION_CACHE_CONTENT"
-     $FAST_TESTS_FLAG --test-washer $COVERAGE_CACHE_FLAG
    stage: source_test
    tags:
    - arch:amd64
    timeout: 6h
-   variables:
-     PYTHON_RUNTIMES: '3'

tests_macos_gitlab_amd64

  tests_macos_gitlab_amd64:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    - CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
      $?; export CODECOV_TOKEN
    - inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
    allow_failure: true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      echo "Don''t forget to regularly delete unused versions. Here are the installed
      versions and their memory usage on the runner:"
  
      du -sh $HOME/.gimme/versions/*
  
      '
    - "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
      \ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
      \ \"$PYTHON_VERSION\" ]; then\n  echo \"Python version $PYTHON_VERSION is different\
      \ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
      \  pyenv uninstall -f datadog-agent\n  echo \"Installing Python $PYTHON_REPO_VERSION...\"\
      \n  pyenv virtualenv 3.11.8 datadog-agent\n  pyenv activate datadog-agent\nelse\n\
      \  echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
      \ Keeping the existing environment.\"\nfi\n"
    - python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
    - pyenv rehash
-   - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+   - inv -e rtloader.make
    - inv -e rtloader.install
    - inv -e install-tools
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
+   - inv -e test --rerun-fails=2 --race --profile --cpus 12 --save-result-json $TEST_OUTPUT_FILE
+     --junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer
-   - inv -e test --rerun-fails=2 --python-runtimes $PYTHON_RUNTIMES --race --profile
-     --cpus 12 --save-result-json $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz"
-     $FAST_TESTS_FLAG --test-washer
    - inv -e invoke-unit-tests
    stage: source_test
    tags:
    - macos:monterey-amd64
    - specific:true
    variables:
-     PYTHON_RUNTIMES: '3'
      TEST_OUTPUT_FILE: test_output.json

tests_macos_gitlab_arm64

  tests_macos_gitlab_arm64:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    - CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
      $?; export CODECOV_TOKEN
    - inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
    allow_failure: true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      echo "Don''t forget to regularly delete unused versions. Here are the installed
      versions and their memory usage on the runner:"
  
      du -sh $HOME/.gimme/versions/*
  
      '
    - "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
      \ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
      \ \"$PYTHON_VERSION\" ]; then\n  echo \"Python version $PYTHON_VERSION is different\
      \ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
      \  pyenv uninstall -f datadog-agent\n  echo \"Installing Python $PYTHON_REPO_VERSION...\"\
      \n  pyenv virtualenv 3.11.8 datadog-agent\n  pyenv activate datadog-agent\nelse\n\
      \  echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
      \ Keeping the existing environment.\"\nfi\n"
    - python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
    - pyenv rehash
-   - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+   - inv -e rtloader.make
    - inv -e rtloader.install
    - inv -e install-tools
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - allow_failure: true
      when: manual
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
+   - inv -e test --rerun-fails=2 --race --profile --cpus 12 --save-result-json $TEST_OUTPUT_FILE
+     --junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer
-   - inv -e test --rerun-fails=2 --python-runtimes $PYTHON_RUNTIMES --race --profile
-     --cpus 12 --save-result-json $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz"
-     $FAST_TESTS_FLAG --test-washer
    - inv -e invoke-unit-tests
    stage: source_test
    tags:
    - macos:monterey-arm64
    - specific:true
    variables:
-     PYTHON_RUNTIMES: '3'
      TEST_OUTPUT_FILE: test_output.json

tests_rpm-arm64-py3

  tests_rpm-arm64-py3:
    after_script:
    - CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
      $?; export CODECOV_TOKEN
    - inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - source /root/.bashrc && conda activate $CONDA_ENV
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
?                                                              -------------------------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e rtloader.format --raise-if-changed
    - inv -e rtloader.test
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e install-tools
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - inv -e sds.build-library
    - inv -e agent.build
-   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
?                                                                           ^^^^ ^^ ^^^^^ -
+   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
?                                                                           ^ ^^ ^^
-     "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+     --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
-     $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
?    ------------------
+     --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
?                                                                           ++++++++++++++
-     --test-washer
    stage: source_test
    tags:
    - arch:arm64
    variables:
      CONDA_ENV: ddpy3
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      FLAVORS: --flavor base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi
-     PYTHON_RUNTIMES: '3'
      TEST_OUTPUT_FILE: test_output.json

tests_rpm-x64-py3

  tests_rpm-x64-py3:
    after_script:
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    - CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
      $?; export CODECOV_TOKEN
    - inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - source /root/.bashrc && conda activate $CONDA_ENV
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
-   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
?                                                              -------------------------------------
+   - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
    - inv -e rtloader.install
    - inv -e rtloader.format --raise-if-changed
    - inv -e rtloader.test
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - inv -e install-tools
    - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - inv -e sds.build-library
    - inv -e agent.build
-   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
?                                                                           ^^^^ ^^ ^^^^^ -
+   - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
?                                                                           ^ ^^ ^^
-     "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+     --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
-     $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
?    ------------------
+     --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
?                                                                           ++++++++++++++
-     --test-washer
    stage: source_test
    tags:
    - arch:amd64
    variables:
      CONDA_ENV: ddpy3
      EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
      EXTRA_OPTS: --build-exclude=systemd
      FLAVORS: --flavor base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 16Gi
      KUBERNETES_MEMORY_REQUEST: 16Gi
-     PYTHON_RUNTIMES: '3'
      TEST_OUTPUT_FILE: test_output.json

tests_windows-x64

  tests_windows-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        junit:
        - '**/junit-out-*.xml'
      when: always
    needs:
    - go_deps
    - go_tools_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    script:
    - $ErrorActionPreference = "Stop"
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - $FAST_TESTS_FLAG=""
    - If ($FAST_TESTS -eq "true") { $FAST_TESTS_FLAG="--only-impacted-packages" }
    - 'docker run --rm -m 16384M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
      -e DD_ENV=prod -e GITLAB_CI=true -e CI_JOB_URL="${CI_JOB_URL}" -e CI_JOB_NAME="${CI_JOB_NAME}"
      -e CI_JOB_ID="${CI_JOB_ID}" -e CI_PIPELINE_ID="${CI_PIPELINE_ID}" -e CI_PIPELINE_SOURCE="${CI_PIPELINE_SOURCE}"
      -e CI_REPOSITORY_URL="${CI_REPOSITORY_URL}" -e CI_COMMIT_SHA="${CI_COMMIT_SHA}"
      -e CI_PROJECT_NAME="${CI_PROJECT_NAME}" -e CI_COMMIT_REF_NAME="${CI_COMMIT_REF_NAME}"
-     -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e PY_RUNTIMES="${PYTHON_RUNTIMES}"
-     -e GOMODCACHE="c:\modcache" -e JUNIT_TAR="c:\mnt\junit-${CI_JOB_NAME}.tgz" -e
-     PIP_INDEX_URL="${PIP_INDEX_URL}" -e TEST_OUTPUT_FILE="${TEST_OUTPUT_FILE}" -e
-     EXTRA_OPTS="${FAST_TESTS_FLAG}" -e TEST_WASHER=true -e GO_TEST_SKIP_FLAKE="${GO_TEST_SKIP_FLAKE}"
+     -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e GOMODCACHE="c:\modcache"
+     -e JUNIT_TAR="c:\mnt\junit-${CI_JOB_NAME}.tgz" -e PIP_INDEX_URL="${PIP_INDEX_URL}"
+     -e TEST_OUTPUT_FILE="${TEST_OUTPUT_FILE}" -e EXTRA_OPTS="${FAST_TESTS_FLAG}" -e
+     TEST_WASHER=true -e GO_TEST_SKIP_FLAKE="${GO_TEST_SKIP_FLAKE}" -e API_KEY_ORG2="${API_KEY_ORG2}"
-     -e API_KEY_ORG2="${API_KEY_ORG2}" -e CODECOV_TOKEN="${CODECOV_TOKEN}" -e S3_PERMANENT_ARTIFACTS_URI="${S3_PERMANENT_ARTIFACTS_URI}"
?    ----------------------------------
+     -e CODECOV_TOKEN="${CODECOV_TOKEN}" -e S3_PERMANENT_ARTIFACTS_URI="${S3_PERMANENT_ARTIFACTS_URI}"
      -e COVERAGE_CACHE_FLAG="${COVERAGE_CACHE_FLAG}" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
      c:\mnt\tasks\winbuildscripts\unittests.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: source_test
    tags:
    - runner:windows-docker
    - windowsversion:1809
    variables:
      ARCH: x64
-     PYTHON_RUNTIMES: 3
      TEST_OUTPUT_FILE: test_output.json

tests_windows_secagent_x64

  tests_windows_secagent_x64:
    artifacts:
      paths:
      - $DD_AGENT_TESTING_DIR/site-cookbooks/dd-security-agent-check/files
      when: always
    needs:
    - go_deps
    - go_tools_deps
    script:
    - $ErrorActionPreference = "Stop"
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e AWS_NETWORKING=true
      -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e SIGN_WINDOWS_DD_WCS=true
-     -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL}
-     486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+     -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
?    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      c:\mnt\tasks\winbuildscripts\secagent.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: source_test
    tags:
    - runner:windows-docker
    - windowsversion:1809
    variables:
      ARCH: x64
-     PYTHON_RUNTIMES: 3

tests_windows_sysprobe_x64

  tests_windows_sysprobe_x64:
    artifacts:
      paths:
      - $DD_AGENT_TESTING_DIR/site-cookbooks/dd-system-probe-check/files
      when: always
    needs:
    - go_deps
    - go_tools_deps
    script:
    - $ErrorActionPreference = "Stop"
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e AWS_NETWORKING=true
      -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e SIGN_WINDOWS_DD_WCS=true
-     -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL}
-     486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+     -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
?    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      c:\mnt\tasks\winbuildscripts\sysprobe.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: source_test
    tags:
    - runner:windows-docker
    - windowsversion:1809
    variables:
      ARCH: x64
-     PYTHON_RUNTIMES: 3

windows_msi_and_bosh_zip_x64-a7

  windows_msi_and_bosh_zip_x64-a7:
    after_script:
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    artifacts:
      expire_in: 2 weeks
      paths:
      - omnibus/pkg
    before_script:
    - set RELEASE_VERSION $RELEASE_VERSION_7
    needs:
    - go_mod_tidy_check
    - go_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - $ErrorActionPreference = 'Stop'
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - if (Test-Path omnibus\pkg) { remove-item -recurse -force omnibus\pkg }
    - mkdir omnibus\pkg
    - 'docker run --rm -m 8192M -v "$(Get-Location):c:\mnt" -e CI_JOB_ID=${CI_JOB_ID}
      -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH}
      -e CI_JOB_NAME_SLUG=${CI_JOB_NAME_SLUG} -e CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME}
      -e OMNIBUS_TARGET=${OMNIBUS_TARGET} -e WINDOWS_BUILDER=true -e RELEASE_VERSION="$RELEASE_VERSION"
-     -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
?                                           ----------------------------------
+     -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
      -e GOMODCACHE="c:\modcache" -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true
      -e TARGET_ARCH="$ARCH" -e DEBUG_CUSTOMACTION="$DEBUG_CUSTOMACTION" -e BUCKET_BRANCH="$BUCKET_BRANCH"
      -e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET" -e USE_S3_CACHING="$USE_S3_CACHING"
      -e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET" -e GO_VERSION_CHECK="true"
      -e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG} -e PIP_INDEX_URL=${PIP_INDEX_URL}
      -e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
      c:\mnt\tasks\winbuildscripts\buildwin.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    - get-childitem omnibus\pkg
    - Invoke-Expression "$S3_CP_CMD omnibus\pkg\version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json"
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: package_build
    tags:
    - runner:windows-docker
    - windowsversion:1809
    timeout: 2h
    variables:
      AGENT_MAJOR_VERSION: 7
      ARCH: x64
      OMNIBUS_TARGET: main
-     PYTHON_RUNTIMES: '3'

windows_zip_agent_binaries_x64-a7

  windows_zip_agent_binaries_x64-a7:
    artifacts:
      expire_in: 2 weeks
      paths:
      - omnibus/pkg
    before_script:
    - set RELEASE_VERSION $RELEASE_VERSION_7
    needs:
    - go_mod_tidy_check
    - go_deps
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - $ErrorActionPreference = "Stop"
    - $_instance_id = (iwr  -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
      ; Write-Host "Running on instance $($_instance_id)"
    - if (Test-Path omnibus\pkg) { remove-item -recurse -force omnibus\pkg }
    - mkdir omnibus\pkg
    - 'docker run --rm -m 8192M -v "$(Get-Location):c:\mnt" -e CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH}
      -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e CI_JOB_NAME_SLUG=${CI_JOB_NAME_SLUG}
      -e CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME} -e OMNIBUS_TARGET=${OMNIBUS_TARGET}
      -e WINDOWS_BUILDER=true -e RELEASE_VERSION="$RELEASE_VERSION" -e MAJOR_VERSION="$AGENT_MAJOR_VERSION"
-     -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION" -e PY_RUNTIMES="$PYTHON_RUNTIMES"
?                                                                  ^^^^^^^^  -  ^^^^^^^^^^^^^^^^
+     -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION" -e GOMODCACHE="c:\modcache"
?                                                                  ^^ ++++++   ^^^^^^^^^^^
+     -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e BUCKET_BRANCH="$BUCKET_BRANCH"
+     -e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET" -e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET"
+     -e USE_S3_CACHING="$USE_S3_CACHING" -e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG}
-     -e GOMODCACHE="c:\modcache" -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true
-     -e BUCKET_BRANCH="$BUCKET_BRANCH" -e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET"
-     -e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET" -e USE_S3_CACHING="$USE_S3_CACHING"
-     -e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG} -e PIP_INDEX_URL=${PIP_INDEX_URL}
-     -e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+     -e PIP_INDEX_URL=${PIP_INDEX_URL} -e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
?    ++++++++++++++++++++++++++++++++++
      c:\mnt\tasks\winbuildscripts\buildwin.bat
  
      '
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    - get-childitem omnibus\pkg
    - Invoke-Expression "$S3_CP_CMD omnibus\pkg\version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json"
    - If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
    stage: package_build
    tags:
    - runner:windows-docker
    - windowsversion:1809
    variables:
      AGENT_MAJOR_VERSION: 7
      ARCH: x64
      OMNIBUS_TARGET: agent_binaries

Changes Summary

Removed	Modified	Added	Renamed
0	56	0	0

ℹ️ Diff available in the job log.

julien-lebot · 2024-10-11T13:01:24Z

Python compression & decompression in the MSI installer.

I think I addressed this now, but if not I might need a few more pointers/help

There is still code that perform decompression of Python 2 left in the installer, see https://github.com/DataDog/datadog-agent/blob/main/tools/windows/DatadogAgentInstaller/CustomActions/PythonDistributionCustomAction.cs#L137
It doesn't fail the pipeline, but that doesn't mean we should leave dead code around, this should work:

        private static ActionResult DecompressPythonDistribution(
            ISession session,
            string outputDirectoryName,
            string compressedDistributionFile,
            string pythonDistributionName,
            int pythonDistributionSize)
        {
        ...
                if (File.Exists(embedded))
                {
                ....
                }
                else
                {
                    throw new InvalidOperationException($"The file {embedded} doesn't exist, but it should");
                }
        }

        private static ActionResult DecompressPythonDistributions(ISession session)
        {
            var size = 0;
            var embedded3Size = session.Property("embedded3_SIZE");
            if (!string.IsNullOrEmpty(embedded3Size))
            {
                size = int.Parse(embedded3Size);
            }
            return DecompressPythonDistribution(session, "embedded3", "embedded3.COMPRESSED", "Python 3", size);
        }

        private static ActionResult PrepareDecompressPythonDistributions(ISession session)
        {
            try
            {
                var total = 0;
                var embedded3Size = session.Property("embedded3_SIZE");
                if (!string.IsNullOrEmpty(embedded3Size))
                {
                    total += int.Parse(embedded3Size);
                }
                // Add embedded Python size to the progress bar size
                // Even though we can't record accurate progress, it will look like it's
                // moving every time we decompress a Python distribution.
                using var record = new Record(MessageRecordFields.ProgressAddition, total);
                if (session.Message(InstallMessage.Progress, record) != MessageResult.OK)
                {
                    session.Log("Could not set the progress bar size");
                    return ActionResult.Failure;
                }
            }
            catch (Exception e)
            {
                session.Log($"Error settings the progress bar size: {e}");
                return ActionResult.Failure;
            }

            return ActionResult.Success;
        }

GustavoCaso · 2024-10-11T13:29:05Z

rtloader/README.md

 be loaded at runtime. Under the hood the library provides `RtLoader`, a C++ interface
 that must be implemented by any supported backend, see `include/rtloader.h` for details.

 ### Two and Three

-`libdatadog-agent-three` and `libdatadog-agent-two` libraries provide Python support
-for extending and embedding by linking different versions of the CPython library.
+`libdatadog-agent-two` library provides Python3 support. Python2 isn't supported anymore.


Suggested change

`libdatadog-agent-two` library provides Python3 support. Python2 isn't supported anymore.

`libdatadog-agent-three` library provides Python3 support. Python2 isn't supported anymore.

GustavoCaso · 2024-10-11T13:29:50Z

rtloader/README.md

@@ -15,14 +15,13 @@ RtLoader will `dlopen` the proper backend libraries accordingly.
 ### libdatadog-agent-rtloader

 RtLoader exposes its C89-compatible API through `include/datadog_agent_rtloader.h`. By
-using the `make2` and `make3` functions, the corresponding Python backend will
+using the `make3` function, the corresponding Python backend will
 be loaded at runtime. Under the hood the library provides `RtLoader`, a C++ interface
 that must be implemented by any supported backend, see `include/rtloader.h` for details.

 ### Two and Three


Should we remove this header?

agent-platform-auto-pr · 2024-10-11T13:54:48Z

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=46518099 --os-family=ubuntu

Note: This applies to commit aa96c86

tasks/windows_resources.py

GustavoCaso · 2024-10-11T15:37:24Z

pkg/collector/python/init.go

@@ -361,7 +350,7 @@ func resolvePythonExecPath(pythonVersion string, ignoreErrors bool) (string, err
 	// don't want to use the default version (aka "python") but rather "python2" or
 	// "python3" based on the configuration. Also on some Python3 platforms there
 	// are no "python" aliases either.
-	interpreterBasename := "python" + pythonVersion
+	interpreterBasename := "python3"


Should we move this to a constant?

That sounds sensible to do, although I'd rather let someone from the owning team do so (it might also be worth waiting a bit for other variables such as Python2/3_HOME to be removed as well

GustavoCaso

Thanks for doing this 🎉

rtrieu

LGTM

spikat

Approved for the file we (cws) own

FlorentClarret

Not a blocker because I think we have other places which mention Py2, but there's at least one in the Getting started section, in the requirements. Feel free to update it in a follow-up PR!

chouquette · 2024-10-28T13:13:47Z

/merge

dd-devflow · 2024-10-28T13:13:56Z

🚂 MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

Use /merge -c to cancel this operation!

agent-platform-auto-pr · 2024-10-28T13:27:41Z

[Fast Unit Tests Report]

On pipeline 47595959 (CI Visibility). The following jobs did not run any unit tests:

Jobs:

tests_deb-arm64-py3
tests_deb-x64-py3
tests_flavor_dogstatsd_deb-x64
tests_flavor_heroku_deb-x64
tests_flavor_iot_deb-x64
tests_rpm-arm64-py3
tests_rpm-x64-py3
tests_windows-x64

If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help

dd-devflow · 2024-10-28T13:43:33Z

🚂 MergeQueue: pull request added to the queue

The median merge time in main is 22m.

Use /merge -c to cancel this operation!

chouquette added qa/no-code-change No code change in Agent code requiring validation team/agent-delivery labels Oct 11, 2024

github-actions bot added the component/system-probe label Oct 11, 2024

chouquette marked this pull request as ready for review October 11, 2024 08:22

chouquette requested review from a team as code owners October 11, 2024 08:22

pducolin approved these changes Oct 11, 2024

View reviewed changes

CelianR approved these changes Oct 11, 2024

View reviewed changes

julien-lebot requested changes Oct 11, 2024

View reviewed changes

tools/windows/DatadogAgentInstaller/WixSetup/Datadog Agent/AgentInstaller.cs Outdated Show resolved Hide resolved

gjulianm reviewed Oct 11, 2024

View reviewed changes

tasks/system_probe.py Outdated Show resolved Hide resolved

pgimalac approved these changes Oct 11, 2024

View reviewed changes

chouquette force-pushed the chouquette/drop_python_2 branch from 80ee5b4 to 01e489b Compare October 11, 2024 12:54

chouquette force-pushed the chouquette/drop_python_2 branch from 01e489b to d74dbec Compare October 11, 2024 13:07

GustavoCaso reviewed Oct 11, 2024

View reviewed changes

tasks/windows_resources.py Outdated Show resolved Hide resolved

GustavoCaso reviewed Oct 11, 2024

View reviewed changes

GustavoCaso approved these changes Oct 11, 2024

View reviewed changes

rtrieu approved these changes Oct 11, 2024

View reviewed changes

chouquette force-pushed the chouquette/drop_python_2 branch from 703ae1b to 725b9a7 Compare October 14, 2024 06:56

spikat approved these changes Oct 14, 2024

View reviewed changes

FlorentClarret approved these changes Oct 14, 2024

View reviewed changes

chouquette mentioned this pull request Oct 14, 2024

CI: don't pass python runtime to invoke tasks #30082

Merged

chouquette force-pushed the chouquette/drop_python_2 branch from 725b9a7 to aa96c86 Compare October 14, 2024 08:15

vickenty approved these changes Oct 14, 2024

View reviewed changes

lebauce approved these changes Oct 14, 2024

View reviewed changes

ogaca-dd approved these changes Oct 15, 2024

View reviewed changes

alexn-dd approved these changes Oct 15, 2024

View reviewed changes

robertjli approved these changes Oct 23, 2024

View reviewed changes

chouquette added 3 commits October 28, 2024 14:07

MSI: drop python2 support

8e14574

win: drop python2 support

80e52ab

doc: remove mention of python2

8ce88a9

chouquette force-pushed the chouquette/drop_python_2 branch from aa96c86 to 8ce88a9 Compare October 28, 2024 13:08

github-actions bot removed the component/system-probe label Oct 28, 2024

chouquette added the changelog/no-changelog label Oct 28, 2024

dd-mergequeue bot merged commit a576ca0 into main Oct 28, 2024
230 of 238 checks passed

dd-mergequeue bot deleted the chouquette/drop_python_2 branch October 28, 2024 14:22

github-actions bot added this to the 7.60.0 milestone Oct 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start dropping python2 support #30035

Start dropping python2 support #30035

chouquette commented Oct 11, 2024 •

edited

Loading

cit-pr-commenter bot commented Oct 11, 2024 •

edited

Loading

Fine details of change detection per experiment

Explanation

CelianR left a comment

julien-lebot left a comment

gjulianm left a comment •

edited

Loading

chouquette commented Oct 11, 2024

agent-platform-auto-pr bot commented Oct 11, 2024 •

edited

Loading

Modified Jobs

julien-lebot commented Oct 11, 2024

GustavoCaso Oct 11, 2024

GustavoCaso Oct 11, 2024

agent-platform-auto-pr bot commented Oct 11, 2024 •

edited

Loading

GustavoCaso Oct 11, 2024

chouquette Oct 14, 2024

GustavoCaso left a comment

rtrieu left a comment

spikat left a comment

FlorentClarret left a comment

chouquette commented Oct 28, 2024

dd-devflow bot commented Oct 28, 2024

agent-platform-auto-pr bot commented Oct 28, 2024

dd-devflow bot commented Oct 28, 2024

	`libdatadog-agent-two` library provides Python3 support. Python2 isn't supported anymore.
	`libdatadog-agent-three` library provides Python3 support. Python2 isn't supported anymore.

Start dropping python2 support #30035

Start dropping python2 support #30035

Conversation

chouquette commented Oct 11, 2024 • edited Loading

What does this PR do?

Motivation

Describe how to test/QA your changes

Possible Drawbacks / Trade-offs

Additional Notes

cit-pr-commenter bot commented Oct 11, 2024 • edited Loading

Regression Detector

Regression Detector Results

No significant changes in experiment optimization goals

Fine details of change detection per experiment

Bounds Checks

Explanation

CelianR left a comment

Choose a reason for hiding this comment

julien-lebot left a comment

Choose a reason for hiding this comment

gjulianm left a comment • edited Loading

Choose a reason for hiding this comment

chouquette commented Oct 11, 2024

agent-platform-auto-pr bot commented Oct 11, 2024 • edited Loading

Gitlab CI Configuration Changes

Modified Jobs

Changes Summary

julien-lebot commented Oct 11, 2024

GustavoCaso Oct 11, 2024

Choose a reason for hiding this comment

GustavoCaso Oct 11, 2024

Choose a reason for hiding this comment

agent-platform-auto-pr bot commented Oct 11, 2024 • edited Loading

Test changes on VM

GustavoCaso Oct 11, 2024

Choose a reason for hiding this comment

chouquette Oct 14, 2024

Choose a reason for hiding this comment

GustavoCaso left a comment

Choose a reason for hiding this comment

rtrieu left a comment

Choose a reason for hiding this comment

spikat left a comment

Choose a reason for hiding this comment

FlorentClarret left a comment

Choose a reason for hiding this comment

chouquette commented Oct 28, 2024

dd-devflow bot commented Oct 28, 2024

agent-platform-auto-pr bot commented Oct 28, 2024

dd-devflow bot commented Oct 28, 2024

chouquette commented Oct 11, 2024 •

edited

Loading

cit-pr-commenter bot commented Oct 11, 2024 •

edited

Loading

gjulianm left a comment •

edited

Loading

agent-platform-auto-pr bot commented Oct 11, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Oct 11, 2024 •

edited

Loading