-
Notifications
You must be signed in to change notification settings - Fork 13.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for TeamCity Test history via Teamcity Service Messages #44813
Conversation
… of Rust tests into TeamCity.
r? @aturon (rust_highfive has picked a reviewer for you, use r? to override) |
Thanks for the PR! Integration with existing test frameworks has long been a desire for us and historically hasn't seen much action unfortunately. At this time though I think we're not quite ready to accept a PR like this. We'd ideally not be adding one-off support for all sorts of test suites and frameworks to libtest, but rather we'd have a more fleshed out design of where we want to go to start out with. We've thought about this in the past and it tends to boil down to "bring your own libtest" rather than adding support to libtest directly itself. I realize though this probably isn't the response you wanted :(. There's some more info at rust-lang/rfcs#816 as well I believe. I'm curious about what you think about this? |
Totally understand - hence wanted an early review. Ideally we want to split test runners from test listeners. We'd like to be agnostic of the test-runner for team city service messages. The testing framework should have a logging/reporting interface and the teamcity service messages could be one such listener (the console and crate cargo-test-junit being other implementations of test listeners). Is it currently possible to switch out the test runner for another? It feels a little like bench is a separate test runner. |
(Btw, on a separate note, just want to say how much I love that when you compile tests the compiler complains about unused code that isn't referenced from the tests. It's effectively a compile time coverage tool - super cool.) |
I indeed share many of the same desires! Unfortunately right now though it's not possible to switch out the test runner, that's mostly what rust-lang/rfcs#816 is targeted at. |
@alexcrichton so should this PR be closed, pending resolution to rust-lang/rfcs#816? |
This PR enables Teamcity Service Messages ( https://confluence.jetbrains.com/display/TCD10/Build+Script+Interaction+with+TeamCity ) to be output when running tests or benchmarks. This enables easy integration of Rust projects into TeamCity and hopefully enables easier test runner integration into Intelli Rust ( #43381 ). This will help Rust be easily integrated into existing enterprise environments.
One neat touch is that the rust benchmarks are turned into TeamCity Stats and can then be easily graphed over time through the TeamCity UI.
To turn on having a '--teamcity' option like NUnit3 or by having an environment variable set (RUST_TEST_TEAMCITY_SERVICE_MESSAGES=1) should turn these on. They should not be on by default.
(I believe no corresponding changes to the Cargo project need be made.)
All feedback welcome!