🚨✨ Implement tracing sampling strategy (🚧 devops 🚧)

[bisgaard-itis]

N.B.

I rerequested reviews from everyone because I had to do quite a lot of changes to how we configure the tracing.

What do these changes do?

• Enable a parent based tracing sampling strategy throughout simcore services. This strategy is a head based sampling strategy (see Configure our opentelemetry tracing so we can sample it and enable opentelemetry tracing in prod osparc-ops-environments#1090). That means the first service a request entering the stack hits determines whether or not the trace of the request should be sampled or not (based on a probablility configured via env vars). If the trace is sampled, then all following services will also sample the trace. This is one of the simplest sampling strategies ensuring all sampled traces are complete (no missing spans).
• Unit tests have been added here to test the sampling strategy, but once this goes in master we should also test there using a lower sampling probability (I suggest 0,1) and then propagate this to PROD eventually.
• During the development of the tests I discovered a that the TraceProvider we use from the opentelemetry library is global, which poses a limitation in our tests. Hence, I started using a local TraceProvider which is passed around everywhere. This is not supported by the aiohttp server instrumentation lib (Allow passing a TracerProvider when instrumenting an aiohttp server open-telemetry/opentelemetry-python-contrib#3801) so I had to fix the middleware from that lib.
• This refactoring gave me the opportunity to disable tracing in many tests.

Related issue/s

• Completes the first step in Configure our opentelemetry tracing so we can sample it and enable opentelemetry tracing in prod osparc-ops-environments#1090
• Allow passing a TracerProvider when instrumenting an aiohttp server open-telemetry/opentelemetry-python-contrib#3801
• Tune open-telemetry to remove errors and warnings during tests #8095
• Add tracing sampling percentage osparc-ops-environments#1230

How to test

• Unit tests have been added to service-library test suite

Dev-ops

• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/1628 adds the TRACING_SAMPLING_PROBABILITY env var to all repo.config files.
• As a follow up to this PR TRACING_SAMPLING_PROBABILITY now should also configure the tracing sampling strategy on ops traefik (similar to here). This PR should also ensure that TRACING_OPENTELEMETRY_COLLECTOR_SAMPLING_PERCENTAGE is not required anywhere (as it has now been deleted from the repo.config files. This has been added here: Add tracing sampling percentage osparc-ops-environments#1230

[codecov]

Codecov Report

❌ Patch coverage is 71.38889% with 103 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.63%. Comparing base (98bd68a) to head (009d00c).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8421      +/-   ##
==========================================
+ Coverage   87.32%   87.63%   +0.30%     
==========================================
  Files        1877     1999     +122     
  Lines       73118    77828    +4710     
  Branches     1333     1338       +5     
==========================================
+ Hits        63850    68201    +4351     
- Misses       8869     9227     +358     
- Partials      399      400       +1     

Flag	Coverage Δ
integrationtests	64.12% <50.90%> (+0.43%)	⬆️
unittests	86.32% <71.38%> (+0.32%)	⬆️

Components	Coverage Δ
pkg_aws_library	93.59% <ø> (ø)
pkg_celery_library	84.27% <ø> (ø)
pkg_dask_task_models_library	79.33% <ø> (ø)
pkg_models_library	93.07% <ø> (ø)
pkg_notifications_library	85.20% <ø> (ø)
pkg_postgres_database	87.95% <ø> (ø)
pkg_service_integration	70.17% <ø> (ø)
pkg_service_library	70.98% <75.75%> (+0.06%)	⬆️
pkg_settings_library	90.20% <100.00%> (+0.01%)	⬆️
pkg_simcore_sdk	84.95% <ø> (ø)
agent	93.10% <45.45%> (-0.44%)	⬇️
api_server	91.88% <70.58%> (∅)
autoscaling	95.72% <81.81%> (+<0.01%)	⬆️
catalog	92.27% <45.45%> (-0.09%)	⬇️
clusters_keeper	99.14% <90.00%> (+<0.01%)	⬆️
dask_sidecar	92.37% <100.00%> (+0.58%)	⬆️
datcore_adapter	97.95% <90.00%> (+0.01%)	⬆️
director	75.72% <50.00%> (-0.18%)	⬇️
director_v2	90.93% <74.07%> (+0.62%)	⬆️
dynamic_scheduler	96.84% <68.75%> (-0.07%)	⬇️
dynamic_sidecar	90.44% <77.77%> (+0.01%)	⬆️
efs_guardian	89.83% <81.81%> (+0.09%)	⬆️
invitations	90.90% <54.54%> (-0.52%)	⬇️
payments	92.80% <76.47%> (+0.08%)	⬆️
resource_usage_tracker	92.22% <87.50%> (+0.17%)	⬆️
storage	86.49% <40.00%> (-0.26%)	⬇️
webclient	∅ <ø> (∅)
webserver	87.28% <68.42%> (-0.04%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 98bd68a...009d00c. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sanderegg]

you should:

• add the Red lamp to your PR description since it seems you wanna test stuff
• create MRs in the osparc-config repos concerning your ENV var

[pcrespov]

thx

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for 009d00c.

🟢 All jobs passed!

But CI Insights is watching 👀

[matusdrobuliak66]

👍

[GitHK]

thanks

[pcrespov]

thx for the effort!

[mrnicegyu11]

Very nice, left some comments but most are minor or questions. Thanks a lot for this big contribution 🎉

[bisgaard-itis]

@Mergifyio queue

[mergify]

queue

🟠 Waiting for conditions to match

• -closed [📌 queue requirement]
• any of: [🔀 queue conditions]
  • all of: [📌 queue conditions of queue default]
    • branch-protection-review-decision = APPROVED [🛡 GitHub branch protection]
    • label=🤖-automerge
    • #approved-reviews-by >= 2 [🛡 GitHub branch protection]
    • #approved-reviews-by>=2
    • #changes-requested-reviews-by = 0 [🛡 GitHub branch protection]
    • #changes-requested-reviews-by=0
    • #review-threads-unresolved = 0 [🛡 GitHub branch protection]
    • #review-threads-unresolved=0
    • -conflict
    • -draft
    • base=master
    • label!=🤖-do-not-merge
    • any of: [🛡 GitHub branch protection]
      • check-skipped = deploy to dockerhub
      • check-neutral = deploy to dockerhub
      • check-success = deploy to dockerhub
    • any of: [🛡 GitHub branch protection]
      • check-success = system-tests
      • check-neutral = system-tests
      • check-skipped = system-tests
    • any of: [🛡 GitHub branch protection]
      • check-success = unit-tests
      • check-neutral = unit-tests
      • check-skipped = unit-tests
    • any of: [🛡 GitHub branch protection]
      • check-success = check OAS' are up to date
      • check-neutral = check OAS' are up to date
      • check-skipped = check OAS' are up to date
    • any of: [🛡 GitHub branch protection]
      • check-success = integration-tests
      • check-neutral = integration-tests
      • check-skipped = integration-tests
    • any of: [🛡 GitHub branch protection]
      • check-success = build-test-images (frontend) / build-test-images
      • check-neutral = build-test-images (frontend) / build-test-images
      • check-skipped = build-test-images (frontend) / build-test-images
    • any of: [🛡 GitHub branch protection]
      • check-success = SonarCloud Code Analysis
      • check-neutral = SonarCloud Code Analysis
      • check-skipped = SonarCloud Code Analysis
• -conflict [📌 queue requirement]
• -draft [📌 queue requirement]
• any of: [📌 queue -> configuration change requirements]
  • -mergify-configuration-changed
  • check-success = Configuration changed

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud