-
-
Notifications
You must be signed in to change notification settings - Fork 95
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Codecov integration #219
Comments
We'd need to add support for expecto as an OpenCover Strategy, like others do here: https://github.com/OpenCover/opencover/tree/master/main/OpenCover.Extensions/Strategy |
No plans but I would be very open to it. I'm not sure what would be required on the Expecto side. |
I don't know if this is helpful but... TrackExpectoTestMethodsusing System.Collections.Generic;
namespace OpenCover.Extensions.Strategy
{
/// <summary>
/// Track Expecto test methods
/// </summary>
public class TrackExpectoTestMethods : TrackedMethodStrategyBase
{
private const string ExpectoStrategyName = "ExpectoTest";
private static readonly IList<string> TrackedAttributeTypeNames = new List<string>
{
"Expecto.FTestsAttribute" ,
"Expecto.TestsAttribute",
"Expecto.PTestsAttribute" ,
};
public TrackExpectoTestMethods() : base(ExpectoStrategyName, TrackedAttributeTypeNames)
{
}
}
} need to add builder.RegisterType<TrackExpectoTestMethods>().As<ITrackedMethodStrategy>(); and add Assert.IsTrue(strategies.Any(s => s is TrackExpectoTestMethods)); and then some version of this? |
I'll be honest I'm relatively new to F#, new to codecov and super duper new to expecto, so while I'm happy to help I also definitely need some sanity checks here. |
@voronoipotato This is a good initial start! The hard part about extending codevoc to understand expecto is that expecto has 2 modes of operation:
What I mean by this is that expecto allows you to explicitly construct a list of TestCases and run them, as opposed to the reflection Attribute-based detection methods. So this method would cover the first way, but not the second way. |
Cool so by some contrived definition we're halfway done :P |
It seems like every test list is always under a test attribute in the samples. Is it possible to use a testlist without a test attribute? If it's not possible to use a testlist without the test attribute I think we might be able to check for the testlists under the TrackExpectoTestMethods constructor. Somehow. |
The samples that use attributes is the runner via 'runTestsInAssembly', which uses reflection. There's also runTests, which takes a text value and runs it. This is used in the second case I mention above. |
OpenCover says they only run on windows. :( |
I guess there's https://github.com/SteveGilham/altcover which does work on linux, but it's also run by a single superman, which is somewhat concerning. |
We need anything added to core expecto to work cross platform. So if you have a super-man, then let's use his super-work ;) May I suggest an alternative? Simpler even. Instead of |
@haf as far as I'm aware everything we've been discussing here is in the context of changes to external tools like openCover or altcover. These tools run your test runner for you after discovering the test methods through various means (like reflection), and these discovery methods are what need to be implemented per-adapter. So I don't think we'll need to change much in expecto itself, rather just implement logic in openCover/altcover to learn how to discover expecto tests via attribute or by looking for values of type 'TestCase', etc. |
The problem you're trying to find a decidable solution to is Gödel's Halting Problem, and as such you can't solve it by reflection (decidedly). |
That said, you could do what VSCode does and scan the IL with Cecil; might be able to pick his code straight from the source. Of course, you won't catch computed tests or async tests, due to the above reason (and the fact that there's no code-evaluator shipped with Cecil) |
Can I check what you need from the test functions. They can be parameterised in interesting ways so that may be a factor. I don't fully know how Codecov works. Do you need to know the each test function and the code it calls? Then you instrument all that code? I thinking there could be a way to get the information you need from calling the test exe. |
Disclaimer I don't know what I'm doingCodecov consumes an xml report of what lines of what files were hit by running the testing framework. OpenCov uses reflection, figures out what tests are run and then tries to figure out what lines of code those tests call and what paths they went down using the PDB. So basically instead of trying to figure out all the possible paths a test could go, it just observes what paths the tests DID go. This does make your coverage report theoretically nondeterministic with fscheck, but in practice it's fine for the level of fidelity that code coverage reports are for. Code coverage is really a rough way to communicate the existence of tests. 90% coverage doesn't assure any of that coverage is meaningful, but 20% coverage is a good way to communicate that this project is WIP. |
https://www.npmjs.com/package/codecov.io Seems to say they accept |
Sorry I don't quite understand. Does it instrument the code and run the tests to observe the paths? Or does it work it all out from just reflection? If it instruments the code then the test exe could list the tests and then they could be run one by one recording the paths. |
Opencov as I understand it instruments the code. |
Then it may be possible that this can be done using a new Something like the following maybe:
|
I started playing with minicover with expecto and they added some more fsharp support here: lucaslorentz/minicover#56 The branch I was playing with: https://github.com/TheAngryByrd/Hopac.Websockets/tree/codecov To run it call ./codecov.sh It ends up generating a console report like:
And a html report: I didn't get around to seeing how codecov plays with their xmlreport but it should be supported here: lucaslorentz/minicover#30 |
I've now been able to do something similar with altcover and expecto
Feeding the https://github.com/danielpalme/ReportGenerator (works with mono) I got a decent report One thing I've noticed is it slows my tests way down. Normal:
Altcover weaved:
But at the very least these other coverage tools do work with Expecto so hopefully this gives someone a place to work from. |
Any updates here people? |
You can use AltCover and YoloDev.Expecto.TestSdk with |
Great, thanks @TheAngryByrd |
Did anyone tested https://devblogs.microsoft.com/dotnet/whats-new-in-our-code-coverage-tooling/? Update: I think it's working. This is the report I generated in my project Fubernetes: |
Hey are there any plans to integrate with codecov? Should this be done by Fake? It seems like a great way to evangelize expecto and F# testing in general.
The text was updated successfully, but these errors were encountered: