[PROF-8667] Split profiling tests into ractor and non-ractor suites. #3320
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AlexJF
force-pushed
the
alexjf/prof-8667-heap-profiling-split-ractor-tests
branch
from
3c4f242 to
bcae1c5
Compare
|
Looks reasonable! The CI issue I believe it's because we're still trying to run the profiling stuff with Perhaps we shouldn't run in |
Base automatically changed from
alexjf/prof-8667-heap-profiling-part1
to
master
December 12, 2023 12:28
AlexJF
force-pushed
the
alexjf/prof-8667-heap-profiling-split-ractor-tests
branch
from
b17e4fa to
2f69cbd
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #3320 +/- ##
=======================================
Coverage 98.23% 98.23%
=======================================
Files 1253 1253
Lines 72992 72982 -10
Branches 3430 3429 -1
=======================================
- Hits 71701 71693 -8
+ Misses 1291 1289 -2 ☔ View full report in Codecov by Sentry. |
RSpec was warning > WARNING: An expectation of `:run` was set on `nil`. because the setup task may not be loaded. We can remove this `allow` because the whole component is being mocked, so the setup task will never get run in this case.
AlexJF
force-pushed
the
alexjf/prof-8667-heap-profiling-split-ractor-tests
branch
from
dd4083b to
b02a143
Compare
AlexJF
deleted the
alexjf/prof-8667-heap-profiling-split-ractor-tests
branch
2 tasks
ivoanjo
added a commit
that referenced
this pull request
Jan 10, 2024
…rectly **What does this PR do?** This PR tweaks the RSpec configuration to skip any tests that make use of ractors (that are tagged with `:ractor => true`) unless RSpec is run with `-t ractors`. **Motivation:** As discussed in #3320, ractors have a few bugs which cause flaky behavior on other specs we currently have. And thus in the above PR, we changed the way we run the profiling specs in CI and via rake to make sure to separate tests using ractors out. BUT, such a separation did not apply to when running rspec directly, which I do all the time. I even got confused at some point and thought we had flaky tests, when what I was seeing was the flakiness that happened when I ran all tests, together, including the ractor ones. To improve this, this PR adds an automatic skip to ractor tests, unless you specifically opt into them. This way, running `bundle exec rspec <some file> ` won't accidentally run ractor specs, unless you ask for them. **Additional Notes:** N/A **How to test the change?** ``` $ bundle exec rspec --format=progress ./spec/datadog/profiling/native_extension_spec.rb Run options: include {:focus=>true} All examples were filtered out; ignoring {:focus=>true} Randomized with seed 24705 ..............*.*. Pending: (Failures listed here are expected and do not affect your suite's status) 1) Datadog::Profiling::NativeExtension ddtrace_rb_ractor_main_p when Ruby has support for Ractors on a background Ractor # Skipping ractor tests. Use rake spec:profiling:ractors or pass -t ractors to rspec to run. # ./spec/datadog/profiling/native_extension_spec.rb:101 2) Datadog::Profiling::NativeExtension ddtrace_rb_ractor_main_p when Ruby has no support for Ractors # Behavior does not apply to current Ruby version # ./spec/datadog/profiling/native_extension_spec.rb:75 Finished in 0.41116 seconds (files took 0.41574 seconds to load) 18 examples, 0 failures, 2 pending $ bundle exec rspec --format=progress -t ractors ./spec/datadog/profiling/native_extension_spec.rb Run options: include {:focus=>true, :ractors=>true} Randomized with seed 5699 spec/datadog/profiling/native_extension_spec.rb:98: warning: Ractor is experimental, and the behavior may change in future versions of Ruby! Also there are many implementation issues. . Finished in 0.01124 seconds (files took 0.42492 seconds to load) 1 example, 0 failures $ bundle exec rake spec:profiling:ractors # ... etc Run options: include {:focus=>true, :ractors=>true} Randomized with seed 48950 spec/datadog/profiling/collectors/cpu_and_wall_time_worker_spec.rb:716: warning: Ractor is experimental, and the behavior may change in future versions of Ruby! Also there are many implementation issues. ... Finished in 0.04489 seconds (files took 0.53091 seconds to load) 3 examples, 0 failures ```
2 tasks
ivoanjo
added a commit
that referenced
this pull request
Jan 11, 2024
**What does this PR do?** This PR re-enables a garbage collection profiling spec in the `CpuAndWallTimeWorker` that was disabled on Ruby 3. This test was disabled in #2354 as this caused flaky tests due to a Ruby VM bug related to Ractor GC. Recently, this "flaky when combined with Ractors" behavior started showing up in other places and @AlexJF came up with a better solution in #3320 of separating out the Ractor tests. Thus, we can now run this spec on Ruby 3. **Motivation**: Recently, while working on #3356, I noticed that if I commented out the calls to `rb_postponed_job_[trigger|register_one]` in `on_gc_event`, no specs failed. This was pretty surprising to me -- as it was weird that we didn't have test coverage for it. After looking into it, I realized we did have specs for it -- but we weren't running them on Ruby 3, which I was using for local development. The re-enabled spec correctly covers that situation. **Additional Notes:** N/A **How to test the change?** Validate that test now runs on Ruby 3 and test suite is still green.
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Ractor.newwithractors => truetag.profiling:main- Run all tests EXCLUDINGractorstagged tests.profiling:ractors- Run allractorstagged tests.Motivation:
As soon as you execute
Ractor.new, the VM transitions to a multi-ractor state. This transition cannot be reversed and will change the behaviour of the VM. For example,ObjectSpace::_id2refwill start to raise errors on unshareable objects.This essentially introduces an unavoidable cross-test side-effect that may make tests flaky depending on execution order. By containing tests that create actors to their own rspec run, we scope this side-effect to those tests that would always trigger it anyway.
Additional Notes:
How to test the change?
bundle exec rake spec:profilingbundle exec rake spec:profiling:mainbundle exec rake spec:profiling:ractorsspec:profiling:main: https://app.circleci.com/pipelines/github/DataDog/dd-trace-rb/12856/workflows/933470cd-2e63-4891-babb-9aca834e0660/jobs/483058/parallel-runs/4?filterBy=ALLspec:profiling:ractors: https://app.circleci.com/pipelines/github/DataDog/dd-trace-rb/12856/workflows/933470cd-2e63-4891-babb-9aca834e0660/jobs/483058/parallel-runs/5?filterBy=ALLFor Datadog employees:
credentials of any kind, I've requested a review from
@DataDog/security-design-and-guidance.Unsure? Have a question? Request a review!