-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dropping tests with knapsack_pro #147
Comments
Hi! sorry that you ran into this issue! Unfortunately it is not possible for us to instrument all the existing test runners from the start: there are so many of them and most of them skew the default way to run test suites. We need to instrument every such case in a custom way to propagate context and ensure that all tests are connected to sessions, modules, and suites. Currently we don't support To use it, configure require "datadog/ci"
if ENV["DD_ENV"] == "ci"
Datadog.configure do |c|
c.ci.enabled = true
# this line sets test visibility mode
c.ci.force_test_level_visibility = true
c.service = "komoju-ci"
c.ci.instrument :rspec
end
end As for knapsack_pro instrumentation, I will mark this issue as a feature request and add it to our backlog. Contributions are always welcome if you would like to take a stab on that yourself! |
Alright, I guess that works for us in the interim, thank you. |
I am also experiencing this issue. However, when I try KnapsackPro 7.0.1 with
@anmarchenko, any thoughts about this message? I imagine it might have something to do with the Knapsack configuration @sulami Are you able to share any more of the Knapsack configuration that allowed this to work with 7.0.1?
@anmarchenko are there details about how to add test runner instrumentation to |
It's literally just the original config I shared plus the We also switched from the rake task to using the knapsack_pro binary, which is recommended in the docs to fix various issues, so our invocation is now |
Hi @timlkelly! I am puzzled that you don't see your tests with Did it work for you with As for adding the I see that there is some considerate interest in knapsack_pro integration, so I will definitely prioritize this after Intelligent Test Runner and code coverage for Ruby are released, so I would say some time in Q3 2024 it could happen (just an estimation). |
@anmarchenko I did switch from the rake command to the knapsack binary and saw no change. Thank you, I will compile some logs and send them to you! I appreciate the help. |
Ah! @anmarchenko I did more testing and discovered it was not working as expected because of the competing The documentation I read suggested that this was an optional setting for |
Oh yes, @timlkelly it happens a lot when tracing is disabled by accident when running tests. We had very unfortunate recomendation in our docs before to use if ENV["DD_ENV"] == "ci"
Datadog.configure do |c|
# Configures the tracer to ensure results delivery
c.ci.enabled = true
# The name of the service or library under test
c.service = 'my-ruby-app'
# Enables the RSpec instrumentation
c.ci.instrument :rspec
end
end ... and never touch the I am happy that you found the issue and it works now |
As you said above, I know that adding Knapsack support is not an active work effort (and may never be), but I wanted to share what I've found. In case it's helpful in the future or at least provide a little more context on what's happening. Knapsack Queue implements its own runner, which is why Datadog CI is unable to start/finish the test session and modules. While Knapsack does prepend the RSpec Runner with their own extensions, it also uses The change in method naming makes it a little more difficult to follow the existing approach of prepending Datadog on top of the RSpec methods. For some context, I created a new Rails app (ruby 3.3, rails 7.1). Then added This is only Knapsack Queue. I haven't looked at the non-queue Knapsack runner yet. I hope to spend more time digging into this to find a solution but just wanted to knowledge share my findings so far 😃 |
Thanks for your insights @timlkelly! The main challenge in the instrumentation would be to create a test suite to reproduce knapsack_pro behaviour in our datadog-ci's tests. Maybe we could just (hehe) instrument knapsack__run_specs directly, but there could be a catch: if CI nodes that knapsack_pro use to run tests are not forked from main process but managed somehow by external processes then we would need to propagate context from the main process that runs specs to all of the queue runners (which might be impossible without submitting a patch to knapsack_pro itself). We have kind of a similar issue with Circle CI parallel runners and for them our current approach is to treat each of the parallel runs as a separate test session (which is far from optimal but gives us at least some level of visibility). |
@timlkelly , your diagnosis is correct here
We introduced changes only for Knapsack Pro Queue Mode. Regarding to what @anmarchenko said here #147 (comment) datadog-ci-rb/lib/datadog/ci/contrib/rspec/runner.rb Lines 20 to 41 in 042bb0f
You can create a free account for testing purposes https://knapsackpro.com/
The knapsack_pro command is started on each parallel CI node as a new process. We do not fork the Ruby process. Does it make it easier?
We have integration tests for RSpec in Knapsack Pro Queue Mode to reproduce different RSpec configurations. Perhaps you could use that: Alternatively, we have a Rails app with an example test suite: I hope this helps. |
@ArturT thanks a lot for the info, it helps! If knapsack-pro does not but rather run a specific subset of tests on each independent runner, I think for the first version our best chance would to instrument each runner as a separate test session so that we would have N test sessions if test suite is split across N test runners. After that we'll see if any deeper integration is possible using knapsack API to propagate Datadog context (test_session_id) across test runners: maybe we won't have to do that and simple runners instrumentation will be quite enough. |
You can use a CI build ID for the Datadog context ( You would have to detect CI build IDs based on different CI provider env vars. For example, if Knapsack Pro is running inside of the Github Actions then CI build ID is based on Here is a list of supported CI providers. |
@sulami @timlkelly I added Knapsack Pro support for test sessions, it will be released together with 1.0 release next week. To upgrade follow the upgrade guide here: https://github.com/DataDog/datadog-ci-rb/blob/main/docs/UpgradeGuide.md If you use other Datadog products (APM/ASM/profiling), you'll need to upgrade to 2.0 of datadog library too: https://github.com/DataDog/dd-trace-rb/blob/master/docs/UpgradeGuide2.md Please share your experience after upgrading! I want to know everything you encounter when running your tests with Knapsack and Datadog to see if there is anything else I can do to make it better. |
Reopening this issue because the following problems were found:
|
@anmarchenko in the next few days @ArturT is unavailable, but I'm around if you have specific questions related to Knapsack Pro I can help with. |
Current behaviour
This is a similar issue to #130 in its results, but with a different root cause (I suspect). We're observing a lot of tests and suites being discarded without a session or module ID.
We're using RSpec + knapsack_pro (in queue mode) to run tests in CI. Knapsack runs the tests in small batches which are dynamically distributed to worker nodes, which speeds up overall test runtime.
knapsack_pro 7 made a lot of changes to the runner, which might be related. It seems like we're losing track of the session and module through this process somehow.
I've confirmed that knapsack_pro 6 is fine, so the massive PR is at fault somewhere. I've also confirmed that it only occurs in queue mode. I'm not entirely sure which project to log this on, but it feels like something that's probably easier to fix here(?)
Expected behaviour
Tests and suites have the correction session and module IDs and make it to Datadog.
Steps to reproduce
bundle exec rake knapsack_pro:queue:rspec
Annoyingly this requires a paid Knapsack Pro account. I'm happy to run anything to help on ours to aid in debugging.
Environment
Datadog.configure ...
):The text was updated successfully, but these errors were encountered: