dotnet test fails at runtime when module values are used in tests (xunit/mstest)

[neoeinstein]

This is the SDK match to dotnet/fsharp#1371.

The issue reported here has been around for a long time, but has only become an issue with .NET Core moving to produce Exe output and target netcoreapp1.x. Under the new scheme, if your entire test consists of one file containing the following:

module MyTest
open Xunit

let myValue = [ 2 ]

[<Fact>]
let ``myValue is populated`` () =
  let [ x ] = myValue in Assert.StrictEqual(2, x)

This test will fail at runtime with a NullReferenceException, as myValue will never be set. fsc creates an implicit main function which would do the initialization, but when running tests, main is never called. As a workaround, I have been adding a dummy Program.fs which moves myValue's initialization to a static constructor:

module Program
let main _ = 0

This workaround is non-intuitive, and will catch those creating tests for .NET Core F# projects off guard.

A related .fsproj for your enjoyment:

<Project Sdk="FSharp.NET.Sdk;Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp1.1</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="MyTests.fs" />
    <!--Compile Include="Program.fs" /-->
    <PackageReference Include="FSharp.NET.Sdk" Version="1.*" PrivateAssets="All" />
    <PackageReference Include="FSharp.Core" Version="4.*" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0-preview-20170106-08" />
    <PackageReference Include="xunit" Version="2.2.0-rc*" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0-rc*" />
  </ItemGroup>
</Project>

Attempting to set the OutputType to Library results in the Test SDK bits not being placed in the correct place, giving an error "Could not find testhost.dll". Downgrading the TargetFramework to netcoreapp1.0 has no effect.

[enricosada]

@neoeinstein just to be sure. if i add a Program.fs with an empty main, the problem is resolved?
@dsyme adding an empty main is enough to force initialization?

If is like that, i just need to update the dotnet new templates i think

[neoeinstein]

@enricosada That seems to be correct. To be safe, nothing should be in the entry module other than main. To get rid of the "empty main" warning, I add a no-warn directive. My complete Program.fs

module Program = let [<EntryPoint>] main _ = 0
#nowarn "988"

[enricosada]

good @neoeinstein i'll update these asap. i hope we are in time for rtm (dunno about that).
but at least the issue has an easy workaround

[lambdakris]

So I was thinking about updating the built-in testing templates shipped with dotnet/templating by adding a Program.fs file with the entry point, but in doing so I began to have doubts about the way Compile items are handled in the .fsproj file.

Currently a wildcard is used to include all .fs files:

  <ItemGroup>
    <Compile Include="**/*.fs" />
  </ItemGroup>

I suppose the assumption was that all .fs files would be independent of each other such that compilation order would not matter.

With the introduction of a Program.fs file this assumption begins to break down if any other file is alphabetically below the Program.fs file. While this particular issue can be mitigated like so:

  <ItemGroup>
    <Compile Include="**/*.fs" Exclude="Program.fs" />
    <Compile Include="Program.fs" />
  </ItemGroup>

I wonder if it might not be best to just use explicit file ordering for all files and not make assumptions about file independence:

  <ItemGroup>
    <Compile Include="Tests.fs" />
    <Compile Include="Program.fs" />
  </ItemGroup>

Does anyone have any objections to this?

[enricosada]

@lambdakris I think the explicit (no wildcards) is better.
Also please updat mstest and the .netcore2.0 templates? Same change. Are near the xunit one

[lambdakris]

OK, pr with changes (incl both mstest & xunit, both 1.x & 2.0) has submitted and been merged

[enricosada]

Awesome, thx @lambdakris !

Closing, fixed with dotnet/templating@934a6e8

[enricosada]

@lambdakris if you want another nice and easy thing to fix, there is #93 (comment) :D
just mentioning it because you be interested 😄

[buvinghausen]

Removing Program.fs yields the following warning when you compile against netcoreapp (all versions)

warning FS0988: Main module of program is empty: nothing will happen when it is run

Including Program.fs but changing the TargetFramework to net452 yields the following error

error FS0222: Files in libraries or multiple-file applications must begin with a namespace or module declaration. When using a module declaration at the start of a file the '=' sign is not allowed. If this is a top-level module, consider removing the = to resolve this error.

What I had to do to support targeting both new & old and not generate warnings or errors was conditionally include Program.fs

<ItemGroup Condition="'$(TargetFramework)' != 'net452'">
  <Compile Include="Program.fs" />
</ItemGroup>

Example .fsproj

Hopefully this will save someone out there the hours of time I lost trying to get the dotnet cli to work on nuget packages that are made available to both full .NET Framework as well as .NET Standard.

@enricosada Perhaps a similar conditional can be added to the template to better support FSharp testing from the dotnet cli for multiplatform projects. I'm sure a better conditional could be included this was just all I needed for my particular use case.

[samuela]

I just came across this warning and it led me to this particular issue. I would like to say that I appreciate the template being adjusted, but honestly the first thing that I did after dotnet new xunit -lang F# was to remove Program.fs because I figured I didn't need it: this is a testing project after all.

Is there way to get around this warning without adding a placeholder EntryPoint? All I want to do is test some stuff anyways.

[buvinghausen]

AFAIK with .NET Core SDK 2.1.400 you no longer need the entrypoint but you do with the earlier versions.

[samuela]

@buvinghausen I'm on .NET Core SDK 2.1.403 and I'm still seeing the same warning.

/Users/skainswo/dev/kumo/new-world/tests/api.tests/Tests.fs(133,4): warning FS0988: Main module of program is empty: nothing will happen when it is run [/Users/skainswo/dev/kumo/new-world/tests/api.tests/api.tests.fsproj]

[buvinghausen]

@samuela hmm I'm not sure I know when the 2.1.400 SDK dropped I was able to remove it from my F# Nuget repository. You can see on this commit I was able to remove the Program.fs as well as the conditional include from the fsproj file.

buvinghausen/Swashbuckle.NodaTime.AspNetCore@6061d45

Do you have Microsoft.NET.Test.Sdk 15.8.0 referenced in your test fsproj?

[samuela]

@buvinghausen That did the trick! Thank you so much! I was on Microsoft.NET.Test.Sdk 15.7.0 and upgrading to 15.8.0 fixed it.