Improve and dogfood the dotnet core sdk support

[enricosada]

This PR

1. Uses latest released .NET Core Sdk (the latest dotnet cli) preview4 to build and publish fsc
2. Creates a nuget package Microsoft.FSharp.Compiler.Sdk.netcore for use in .net core sdk fsproj.

Additional info, and some context about how the new msbuild/fsproj/sdk work https://gist.github.com/enricosada/539da221ef3d87f0151d395f7889f94c

This direction will eventually lead to the removal of project.json for the Core CLR build, leading to the use of the new fsproj msbuild based, with a normal msbuild-based templates for .net core sdk projects** (so exactly like dotnet new -l fsharp -t console).

This gives us the basics we need to polish and finalize support for the new msbuild+fsproj format, and dogfood .the net core sdk fsproj in visualfsharp repo. This will be crucial to ensure the extensibility points in the SDK are adequate for our needs.

Steps in the PR are:

• install correct sdk
• restore
• build fsc
• publish
• package
• test end to end of the built packages in isolation.

This PR builds and packages just fsc (not fsharp.core or fsharp.compiler) - the other already built assemblies are used as normal <Reference />. So this can be run in parallel to the existing build script for .NET Core to use the advanced functionality of .net core sdk (like publish) and the other projects
can be converted in future steps.

The script is added as additional step in build.cmd when passing the netcoresdk argument. Generated packages are put in the usual $(Configuration)\artifacts directory.

This PR makes no changes to the current build script/code for .NET Core (the coreclr) argument, that's why all code is addition, but duplicated code is minimal.

Documentation

• To use the .net core sdk script: see ./scripts/use_dotnet_core_sdk.README.md

• Build script documentation: see comment at end of ./build-using-netcore-sdk.proj

• Examples (to easy locally dev VF#): see examples/dotnetcoresdk/simple/README.md

The Microsoft.FSharp.Compiler.Sdk.netcore package

The Package target creates a new package Microsoft.FSharp.Compiler.Sdk.netcore with

• the correct layout to be consumed by .fsproj in .net core sdk using props files who specify fsc/fsharp.build path.
  these property are used by FscTask inside FSharp.NET.Sdk package.
• the fsc inside build/netcoreapp1.0 is a normal console app published as framework dependent deployment.
  That mean can be run with dotnet build/netcoreapp1.0/fsc.dll like all normal .net core app published with dotnet publish.
• this package is tested End to End in isolation in target TestE2E

How fsc is built

The src/fsharp/Fsc/Fsc.netcore.fsproj is added. It's a normal console app fsproj for .NET Core Sdk 1.0.0-preview4.

• build coreclr (because use the fsharp.core/fsharp.compiler assemblies already built)
• cd src/fsharp/Fsc
• dotnet restore Fsc.netcore.fsproj --configfile Fsc.netcore.NuGet.Config (needed until the first package is on nuget.org)
• dotnet publish Fsc.netcore.fsproj -c Release

NOTE: At the moment the generated fsc.dll doesnt work because is not signed correctly. To workaround this issue for now, the already built fsc.exe is copied as fsc.dll (so the dotnet build output is overwritten) Running dotnet msbuild build-using-netcore-sdk.proj /p:Configuration=Release /t:PublishFsc will generate the fixed published fsc. This will be fixed in a later PR, see TODO

Example .NET Core sdk preview4 fsproj

This PR adds an example of a .NET Core Sdk fsproj, see examples/dotnetcoresdk/simple/README.md This can compile using both the dotnet path/to/fsc.ll (netcore) or fsc-proto.exe (.net full),
just set the FscToolPath/Exe as usual.

The Microsoft.FSharp.Compiler.Sdk.netcore can be overriden by adding as PackageReference, like in E2E tests.

Install script

All the steps are scripted using msbuild (run dotnet msbuild build-using-netcore-sdk.proj, see same file for help). Why?

• See README ./scripts/use_dotnet_core_sdk.README.md for doc
• This is easy to use. Just run .\scripts\use_dotnet_core_sdk.bat (or .ps1). After that the right sdk is in PATH, so just dotnet --version
• This is fast. The .net core sdk is downloaded/install only if needed. If is already in PATH (like in machine wide install) that installation is used
• This gives reproducible builds. the version is always checked, if doesnt match the one in DotnetCoreSdkVersion.txt it's downlaoded and installed
  in .dotnetsdk directory and added to PATH (like current install script)
• The sdk used by scripts/test is always the one in PATH (because is not in a fixed path) and easy to check with dotnet --version
• This is maintenable. Download and use the official install script from http://github.com/dotnet/cli repo, when needed

Build scripts

The ./build-using-netcore-sdk.proj and in ./build/*.

• Based on a simplified structure of the build script used by http://github.com/dotnet/cli
• Can be run with msbuild 15, so dotnet msbuild
• All phases are distinct, and new packages can be easy added.
• Docs is at end of ./build-using-netcore-sdk.proj and all targets are commented about usage
• Should be ok for xplat

How is tested

E2E test in isolation, based on fsharp suite.

• The tests (TestE2E target) are like the one in fsharp suite (reused lot of code of fsharp suite, same dsl).
• All tests methods can be run indivisually (like fsharp suite, so run fsi is an option), no xunit/nunit magic (/cc @dsyme).
• Atm suite runs with dotnet run so is just a normal console app.
• Use xunit (nunit doesnt work yet with dotnet vstest) but not any features of xunit, it's only based on the
  official xunit test template (dotnet new -t xunittest -l fsharp)
• the test use the built package to restore/build/run/pack in isolation (doesnt mess with cache) a console and a lib app.

TODO not in this pr (next one)

• Fix signing issue for fsc, so the fsc.netcore.fsproj will works ok with just dotnet publish
• add converted .\scripts\use_dotnet_core_sdk.ps1 as shell script .\scripts\use_dotnet_core_sdk.bat.sh
• run the ./build-using-netcore-sdk.proj xplat
• run tests with dotnet test (instead of dotnet run), need configuration of logging (trx) for cI, localdev
• the examples/dotnetcoresdk/simple/Simple.sln doesnt reference the fsproj
• minimize duplication of code (msbuild .props define), this PR prioritize parallel code instead of replace existing in one shot (safer)
• fix generation of assemblyfile.fs who contains assemblyinfoattributes

/cc people for review

• @dsyme the added test suite is like fsharp suite, no nunit magic, and can be fixed easy to be run by fsi

• @KevinRansom that's using only released .net core sdk and packages. Add ihmo a good base to use fully the .net core sdk for build. i didnt changed/messed with internal build tools/script, so no regression. The msbuild is inside .net core sdk (dotnet msbuild), like nuget (dotnet nuget), so is using normal tools/commands provided by sdk (and used by normal devs already). workflow is the same basic of developing a .net core app with sdk, i excluded advanced like crossgen (multi framework), pack (packages are generaeted from a prepared directory, easy to check). I think we can use some part of sdk (like publish/pack) to make it easier to create normal netcore apps (fsc) or lib (fsharp.core), without changing a lot the build script/projects.

• @cartermp @brettfo this add dogfooding and enable lots of smaller steps, see TODO

• @vasily-kirichenko @cloudRoutine @dungpa i hope ./examples and the easy install script will help to add support for .net core proj to vf# extension (i added you all because you develop the extension). i think lots can be done using https://github.com/dotnet/roslyn-project-system as base (easier maybe than chaning existing) and defining a fsharp capability for languageservice (maybe this require discussion in a new issue)

• @forki i hope the installs script, build script are ok for contributors (feedback wanted about that), i tried to maximize (in the limits of msbuild) the readability and developer workflow (phases, simple targets, and normal project can be run directly from cli using normal dotnet commands, the same run by script)

[enricosada]

adding https://gist.github.com/enricosada/539da221ef3d87f0151d395f7889f94c as additional docs about this PR

[dsyme]

I'd like to know from @KevinRansom and @brettfo whether this PR looks vaguely in the right direction. A lot looks right to me but it needs to be analyzed carefully. I spoke with @enricosada and he says it's really important to get the new Microsoft.FSharp.Compiler.Sdk.netcore package being built to unblock the dogfooding of F# support for .net core sdk fsproj.

• @enricosada I suspect very few of the people have actually use Preview 4 and the new .csproj/msbuild-based .NET Core SDK. I had zero real context on all of this until I read this PR.

• @enricosada The duplication of build/install scripts is pretty disturbing. I think it's heading in the right direction, since the new .fsproj needs to be supported and it make sense to do even more with MSBuild for the reasons you say. But it would be great to see some files deleted as part of this :) We now have .bat, .sh, .travis.yml, autogen, Makefile (fsharp/fsharp) and msbuild fragments in our build logic. We need to be reducing the overall complexity. (That said, I know why you haven't deleted anything - this is still wet paint - but I think it's a new kind of paint that's at first hard to get our head around. And you're of course a community hero for complexity reduction, removing all those .bat files when rewriting the test drivers :) Perhaps listing all the stuff that will eventually be deleted would be helpful)

• @enricosada Could you publish a prelim version of Microsoft.FSharp.Compiler.Sdk.netcore yourself? Or is the signing issue to problematic?

• @enricosada The list of TODOs is a bit confusing. Could you edit them to make them crystal clear? To accept the PR we will need to know exactly what remains to be done.

[forki]

I suspect very few of the people have actually use Preview 4 and the new .csproj/msbuild-based .NET Core SDK.

yes maybe very few - but we already use it in production (with dotnet cli rc4) and it works like a charme

[dsyme]

adding https://gist.github.com/enricosada/539da221ef3d87f0151d395f7889f94c as additional docs about this PR

@enricosada Please put that information in the PR description right at the top. I'll help you edit it into a condensed form. We can then add it to the repo as a document, or put it on http://github.com/fsharp/fsharp.github.io as part of the compiler guide.

[dsyme]

@enricosada I think replacing the "build coreclr" path (rather than adding a new one) would be better - why would we keep the old one active in master?

[enricosada]

@enricosada I think replacing the "build coreclr" path (rather than adding a new one) would be better - why would we keep the old one active in master?

No reason at all, is just another pr (i can do that) because i fear big one shot changes.
I can replace build coreclr completly adding new fsproj in parallel, and use .net core to build it. That remove all complexity for coreclr from fsproj/build script (see below).
Build artifacts (fsc.dll, fsharp.core.dll, etc) can be placed in expected directories, so rest of build.cmd doesnt change.
that was my idea, just splitted in two PR (this is the first one)

@enricosada I suspect very few of the people have actually use Preview 4 and the new .csproj/msbuild-based .NET Core SDK. I had zero real context on all of this until I read this PR.

The csproj VS tooling is out of preview in VS2017 RC3. the sdk itself is really near rtm.
I'll probably use that at work too when VS2017 goes RTM, to replace our current aspnetcore with project.json and VS2015 btw (and we use it with .NET FW/iis, not .net core).

For c# guys is better, and they are gonna use that, because msbuild format is easier than project.json (less alien). Also VS2017 already support it, so is not difficult to sell at all.

@enricosada Could you publish a prelim version of Microsoft.FSharp.Compiler.Sdk.netcore yourself? Or is the signing issue to problematic?

I can do that, that's what the build script does btw. I just need to change the name, but is not a problem. can be put inside FSharp.Compiler.Tools too (so it support netcore and net40).
I can generate the package with script, and overwrite the assemblies with the signed one generated by build coreclr . result is here in dev feed, i already use that to boostrap things (because easier).

Working example using dev feed are inside this PR

cd examples\dotnetcoresdk\simple\console
dotnet restore
dotnet run

@enricosada The duplication of build/install scripts is pretty disturbing.
Perhaps listing all the stuff that will eventually be deleted would be helpful.

There are two parts, build script orchestration and project files. Fsc.fsproj is a proj, but build require nuget restore/packaging/signing/runtests. That can be offloaded to .net core sdk. Some deletion will
greatly simplify files, but not remove it completly

I cannot migrate project files until VS support new fsproj, but we can simplify greatly current script. It's ortogonal to my pr (needed to improve sdk), can be done anyway (just give me a go).

I assume i will build the .net core version of components with new fsproj and sdk, so we remove current complexities in old fsproj.
I assume build orchestration goes in a msbuild .proj, using the sdk featuers (like my pr)
I assume all stuff like restore/packaging/runtests will be executed by .net core sdk

To delete:

• msbuild location code -> dotnet msbuild
• .nuget directory => dotnet nuget
• Tools directory (minus specific stuff like localitazion target) because is pratically a custom packaged version of the sdk. And not used because new fsproj build netcore. Signing is already support by sdk.
• init-tools and install script, useless, because are a custom version of the official .net core install script (same features)
• all project.json
• all netcore related stuff
• tests runner from build script complexity => dotnet run or dotnet test

it's possibile to dotnet build the old fsproj (will use normal targets), just use the new msbuild.exe. That remove lot of complexity in build script. and make easier to run xplat

Old fsproj in parallel to new fsproj. used for .net core. Common properties like defines can be shared using specific .props file (like my pr)
Having both side by side, the new fsproj can also include build of net40 too, so when vs support it, we can just remove the old one.

Note, the .NET Core SDK can build ALL framework, so .NET Framework, .NET Core, Portalble profiles, etc in same fsproj. is not only for netcore.
And dotnet build -> dotnet msbuild /t:Build, no magic

• where is msbuild? dotnet msbuild
• where is nuget? dotnet nuget
• restore all needed packages? dotnet restore --packages /path/to/save/packages (aftet that the fsproj know that path, no need to manually fix reference)
• build fsc as net40? dotnet build src\fsharp\fsc\fsc.fsproj -f net40 -c Release
• generate the fsharp core package? dotnet pack src\fsharp\fsharp.core\fsharp.core.fsprj -c Release /p:Version=4.1.1.0 (the package metadata are property inside fsproj, or extenal props file, or from cmdargs)

like before, it's msbuild so following works:

• build fsc as using proto? dotnet build src\fsharp\fsc\fsc.fsproj -f netcoreapp1.0 -c Release /p:FscPath=protodir/fsc.exe
• run cambride test? dotnet run tests\fsharp\FSharp.Tests.FSharpSuite.fsproj (or dotnet test for nunit like experience, with logs/filter/etc). Will continue to work from fsi obv.

The same build workflow can be used by developers too

• cd src/fsharp/Fsc
• dotnet restore
• dotnet run -f net40 //build all deps, run fsc for net40
• dotnet publish -f netcoreapp1.0 //build all deps, save in a dir an xcopy deployable fsc for .netcore
• dotnet publish -f net40 //build all deps, save in a dir an xcopy deployable fsc for .net 40

about other required stuff, like signing and vsix creation, can be done like dotnet/cli repository.

[enricosada]

@enricosada The list of TODOs is a bit confusing. Could you edit them to make them crystal clear? To accept the PR we will need to know exactly what remains to be done.

will do, and fix merge conflict

[dsyme]

@enricosada

I can replace build coreclr completly adding new fsproj in parallel, and use .net core to build it. That remove all complexity for coreclr from fsproj/build script (see below).

I think that would be better. Then the PR will act as a definitive guide even if @KevinRansom decides to do things a bit differently in the end. The PR should then contain all the necessary updates to the build and getting started material too (some of which you've written above)

Looking at all those commands above there's a lot to learn. TBH I don't even know the difference between dotnet msbuild and dotnet build :) ? And what is the -f flag?

(just after I wrote that I realize you said dotnet build -> dotnet msbuild /t:Build)

[enricosada]

As a note, to continue fsproj/sdk work, i'll use FSharp.Core and Fsharp.Compiler.Tools packages. as detailed in https://github.com/dotnet/netcorecli-fsc/wiki/Roadmap /cc @dsyme

We can discuss how to continue this pr to use sdk inside visualfsharp repo later, so is not urgent for fsproj/sdk/netcore like was before ( /cc @cartermp ).

More one-shot work for me instead of integrated+scripted+tested inside this repo, but at least i can continue to finalize fsproj 😄 (was that the goal)

[enricosada]

Closing this.

Used FSharp.Core and FSharp.Compiler.Tools packages, for FSharp.NET.Sdk v1.0.0.

Will rework this PR to use .net core sdk 1.0 rtm in build to replace build coreclr, later, after discussion with vf# team on how to proceed. if needed, otherwise np.

[forki]

Noone from Redmond commented on this? Not interested?