CoreCLR support for FsCheck.Xunit

[josephdecock]

Is it possible to add support for the Core CLR to FsCheck.Xunit? I'm getting "The dependency FsCheck.Xunit 2.2.4 ... does not support framework DNXCore,Version=v5.0." I would prefer to send a pull request rather than just opening an issue, and I have cloned the repo and poked around a bit. Unfortunately I'm still learning paket and the rest of the tools in the build, and what to do was not immediately obvious. Can I get a nudge in the right direction?

As far as I can tell, neither FsCheck nor Xunit alone have any trouble running on the Core CLR, so I would assume that there isn't anything fundamental in the way of this happening. Is that true?

[mausch]

Last I heard ( SuaveIO/suave#334 ) "the [F#] compiler is very ... very alpha quality"

[kurtschelfthout]

Interesting, are you saying you actually ran FsCheck as is on Core CLR and it works? Because afaik that's never been tested.

As @mausch says, I was expecting to wait with this until the F# compiler is not quite so alpha. So I feel like the nudge would be more like a shove down the rabbit hole :)

[josephdecock]

Wow, I seem to have stumbled into much deeper waters than I intended. Hopefully I'm not about to take the blue pill and wake up with wires coming out of my brain...

Anyway, what I've done with FsCheck on the Core CLR is pretty minimal. I followed this guide to getting started with xunit using dnx. Then I installed FsCheck and was able to write and compile a test that included a call to Prop.ForAll. The output of that test didn't quite make sense, but that may have been my fault. I'm just getting into FsCheck and so far have used it with Xunit. I hoped things would make more sense with FsCheck.Xunit and tried installing it, and that brought me to the dependency errors.

[josephdecock]

I've cleaned up the tests that I made, and FsCheck definitely works when targeting the CoreCLR with C#. I have a demo here.

[kurtschelfthout]

That's pretty cool!

I don't know enough about CoreCLR to really understand things. I wonder why FsCheck.Xunit complains but FsCheck itself does not; they are compiled in the same way in the same MSBuild step. It must have something to do with the xunit dependency, which does have different versions for CoreCLR afaik.

So I'm currently not in a position to give you any nudges :( Happy to answer any questions though. I'll try to do some research on CoreCLR to see what can reasonably be done here, might take a bit of time.

[jhamm]

Thanks for getting the demo setup. It was very helpful!

To get dnvm, dnu, and dnx commands, follow the instructions: https://github.com/aspnet/home

To run the demo

> dnvm install 1.0.0-rc1-update1 -r coreclr -a x86

> dnvm list

Active Version           Runtime Architecture OperatingSystem Alias
------ -------           ------- ------------ --------------- -----
  *    1.0.0-rc1-update1 coreclr x86          win

> cd {project folder}/Point
> dnu restore
> dnu build
> dnx test

To see the error

in the Point/project.json, add to the dependencies; "FsCheck.Xunit": "2.2.4",
Add the using "FsCheck.Xunit"
Add a sample test like:

        [Property]
        public bool SameCoordsShouldBeEqual(int x, int y)
        {
            var p1 = new Point(x, y);
            var p2 = new Point(x, y);
            return p1 == p2;
        }

Then run:

> dnu restore
> dnu build
> dnx test

[jhamm]

Here is a nudge in the right direction,

Have a look at the "PCL" folder in FsCheck solution, ex: FsCheck.Portable.Profile7. Each of the files in the projects are links to the non-portable FsCheck file so if one changes the portable will also use the same file.

You will need to add 2 references that the non-portable FsCheck.Xunit project uses, FsCheck and xunit.execution.desktop (need to use paket add for this one).

Take a look at the src\FsCheck\paket.template file to see how to add the portable versions to the nuget package.

The build.fsx is how Paket will Pack the nuget file for a new release.

Let me know if you need another nudge 😄

[kurtschelfthout]

Here's an interesting tutorial on migrating F# to CoreCLR.

https://github.com/enricosada/fsharp-dotnet-cli-samples/wiki/Migrate-fsproj-To-.NET-CLI

[josephdecock]

Thanks for all the input, especially @jhamm. I've had some partial success. I've created a Portable Profile for FsCheck.Xunit, and added the resulting dll to the nuget package. Using that package in my demo project, I'm able to run tests created with the PropertyAttribute! The code is here.

As I said, this was only a partial success however. The duration for tests created with the Property Attribute are missing from the test runner. Any idea why that's the case? In case that's unclear, here's what I'm looking at in the test runner:

The one test missing duration info is the one I've defined as a [Property]. The rest are my old tests defined as [Fact]s. For a while I thought maybe the test wasn't getting run at all, but I am able to debug the test and step through the code in the property.

I also wonder if creating more pcl profiles is the right approach. To get started, I just created one (FsCheck.Xunit.Portable.Profile7). Would you want Profile78 and Profile259 too? Then if nunit support was ever considered, would you want 3 Profiles for that too? Seems like a lot of very similar files, but then again I don't have prior experience compiling PCLs, so maybe its just normal.

[kurtschelfthout]

This is really interesting and cool work.

Can someone explain the link between (existing) PCLs and CoreCLR or give a pointer? Seems like certain PCL profiles work on CoreCLR, or something?

Sorry don't directly know why the time is not showing - if you debug into the PropertyAttribute, does it return a TestPassed with a timer.Total that makes sense?

EDIT: forgot to mention, yeah don't worry about the different files. Seems to work fine and not to cause too much hassle (perhaps counter-intuitively).

[josephdecock]

This blog from last year discusses how PCLs and CoreCLR interact. It says that PCLs can run as-is on CoreCLR. I'm still looking for more thorough documentation, but it's a start.

When I debug into the attribute, the timer.Total doesn't make sense. I get a non-zero number of ticks less than 1 millisecond. I've got a test that takes several seconds to run (just wasting cpu cycles in a big loop), and still the timer.Total is just a few thousand ticks (.2 ms).

[kurtschelfthout]

Thanks for that link. Clears up things a bit. It says profile259+ runs on CoreCLR, so I would be fine if that would be the only profile for FsCheck.Xunit (i.e. we can skip 7 and 78 if we like, I put those in because FsCheck is used with the F# compiler tests which tests those two profiles as well)

The other thing it says is to put the dll in the lb/dotnet folder in NuGet, and to list dependencies explicitly. Though it seems like things work without it. May be a bit dodgy.

Timer looks like a separate bug to me. I wonder if/how this works on NET4.5. The ExecutionTimer is instantiated but never used. It looks to me like it should time the Check.Method execution. @jhamm and/or @ploeh may know a bit more about this?

[kurtschelfthout]

Can you try the timing now, after pulling that commit I just pushed?

[josephdecock]

With that commit I get the the timing I would expect. Thanks a lot!

I've been trying to get my head around the future of PCLs in Net Core. This talk from NDC London was really interesting. Damien Edwards and David Fowler talk about the new .NET CLI and .NET Platform Standard (documented here). The platform standard is the successor to PCLs and it looks like a really good direction to go in. You just say "I'm building my library against version x.y of the standard" and then any platform that implements that version or later of the standard can use it. You aren't tied down to specifying in advance which platforms can use your library.

All that said, I think for now it makes sense to wait to adopt the platform standard. Work on it is still in progress . In the talk, they said that targeting the platform standard would be the big change to come in ASP.NET 5's rc 2, which should be coming Real Soon Now(tm).

With all these big changes on the horizon, would you even want to consider a PR of what I'd done reworked as a profile259?

[kurtschelfthout]

Thanks for the info. I agree I don't really want to do the "dotnet" stuff, seems like they're going to get rid of it, and the platform standard seems like it's the way forward, but as you say not quite ready yet.

That said, I'd definitely be open to accept a PR that adds Profile259 support for FsCheck.Xunit. From my perspective supporting an additional profile for FsCheck.Xunit has low marginal cost. I've been pretty happy with how stable PCL's are, and reading that documentation (haven't watched the video yet) it sounds like there is a good path forward to targeting the Standard Platform from PCL. Most of the bits are already there in the build scripts etc for FsCheck anyway.

I leave it up to you. If you're worried perhaps it's worth just sending what you have without any additional work, and then we know exactly what we're talking about?

[enricosada]

@kurtschelfthout i can do a .NET Core build with .NET CLI if you want, what branch should i target? master or fscheck3? it must use f# 4 (no FSharp.Core 3.x on .NET Core), so it's fscheck3 the best place?

Usually it's only a project.json and some ifdef about migration .NET Full -> .NET Core.

the netstandard framework now is ok ( dotnet and dnxcore50 are migrated to netstandard ).

fscheck it's interesting because i want to try dotnet test using xunit (the only test framework supported atm) and f#.
FsCheck it's used by fsharp (visualfsharp) repo so it's a dependancy anyway for a full .net core build of the compiler.

[kurtschelfthout]

Yeah, fscheck3 is the right branch in that case. Yes, I'm interested! Thanks.

On 23 Mar 2016, at 01:58, Enrico Sada notifications@github.com wrote:

@kurtschelfthout i can do a .NET Core build with .NET CLI if you want, what branch should i target? master or fscheck3? it must use f# 4 (no FSharp.Core 3.x on .NET Core), so it's fscheck3 the best place?

Usually it's only a project.json and some ifdef about migration .NET Full -> .NET Core.

the netstandard framework now is ok ( dotnet and dnxcore50 are migrated to netstandard ).

fscheck it's interesting because i want to try dotnet test using xunit (the only test framework supported atm) and f#.
FsCheck it's used by fsharp (visualfsharp) repo so it's a dependancy anyway for a full .net core build of the compiler.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[la-yumba]

I'm trying to use fsCheck with .NET Core 1.0.0 and (preferably) NUnit and get the following results:

Package dependencies:

• Adding FScheck 2.5.0 dependency causes no problems
• Adding FSCheck.NUnit (or .XUnit) as dependencies does not work (does not support framework .NETCoreApp); I've seen there's some WIP in this area; please include NUnit as well :-)

Behaviour:

• the only way to get a failing test to fail is running the tests from the console with dotnet test, and using .QuickCheckThrowOnFailure()
• if I use the test runner within VS the test always passes
• if I use QuickCheck() the test always passes (even when running from the command line)

[enricosada]

I'll send a pr tonight. I. Converted fscheck, fscheck.xunit to dotnet core for my webcast about .net core.

[kurtschelfthout]

@la-yumba Yes FsCheck itself should work as one of the profiles we ship is compatible with .NET Standard Platform. We don't currently ship those for FsCheck.Xunit or Nunit.

Also without any integration (which won't work for now with .net core) you have to use QuickCheckThrowOnFailure so FsCheck throws an exception on a failing test. Simple QuickCheck will just print the result to stdout but otherwise not do anything.

[enricosada]

ok added new pr, with both xunit and nunit, targeting .net core.
the dotnet test is supported (and used in fscheck tests)

[kurtschelfthout]

@enricosada's PR was merged to fscheck3 branch. Closing this issue as a result.

[la-yumba]

@kurtschelfthout can you please notify us on this thread when the code changes re .net Core support are published, with the relevant version of the NuGet package?
Thanks,

[kurtschelfthout]

I'll try to remember. Meanwhile you can use the 3.0.0-alpha builds on the AppVeyor feed:

https://ci.appveyor.com/project/kurtschelfthout/fscheck/history

See readme for detailed instructions.