[.Net Native] F# Support

[charlesroddie]

[2018-12-06 update] F# UWP apps are releasable via the Windows Store. #6055 (comment)

[2018-06-17 update] UWP apps are compiling and running. More progress by the community in the last 4 days than Microsoft made in 3 years. The main issue is sprintf and we are looking into workarounds.

Problem

F# libraries cannot be used to create native UWP store apps. This results from the lack of F# support in the .NET Native UWP AOT compiler, which is required for publishing UWP store apps. In this post UWP will mean native UWP, i.e. compiled to IL or machine code.

Why support is needed

1. Windows application programming.
   a) UWP is actively developed (the platform itself and 3rd party components), unlike WPF.
   b) UWP also allows AR/VR and XBox apps, and Windows 10 IoT.
2. F# becomes toxic for cross-platform applications or controls that must support UWP.
   a) Xamarin.Forms F# developers end up without Windows app support.
   b) Creating Xamarin.Forms controls in F# for general consumption is effectively ruled out since UWP support is expected.
   c) Xamarin.Forms developers must avoid using .Net libraries if they are written in F#.

There has been huge user demand for this, with 1500+ votes for Add F# support for .NET Native and 2000+ votes for F# support in .Net native for UWP when those were active. This thread is a continuation of F# Support in .NET Native and UWP.

Status of work

Unfortunately users outside Microsoft cannot work on .Net Native. Microsoft has promised support:

2015-10 @crutkas: We are committed to bringing F# support to .NET Native... we’ve been working on .NET Native support for more advanced .NET IL code, like tail call instructions and complex generic types

2016-07 @cartermp: We get recent internal builds of the toolchain that we run tests on, we add tests to try and see what else we can break, and we report issues. These issues get logged into the .NET Native engineering backlog as soon as they come up.

2017-08 @cartermp: We'll remain subject matter experts for the UWP team to work with on .NET Native support, push for F# on their priority list after their .NET Standard support is completed, and we'll also contribute in identifying problems that need to be solved for proper .NET Native support.

Unfortunately the issue appears to have slipped between Microsoft teams with no-one currently working on it.

Technical issues

Two hard issues have been mentioned from the beginning, but the first seems to have been a red herring, and the second may have been fixed.

Tailcall support
2016-16-07 @dsyme: Yes, it's feasible for tailcalls just to be ignored in first-cut support. UWP Native would just have to ignore the tailcall prefix.

Generics
2017-09-04 @KevinRansom: I'm pretty sure the nested generics got fixed.

There is no public information about other issues and therefore it is not publicly known what is blocking F# usage.

What the community can do

Test .Net Native and CoreRT

CoreRT testing is in progress. It is now possible to test .Net Native. See @DjArt 's post here.

You can use the scaffolding here to test .Net Native and CoreRT. The F# cheatsheet works and we are starting to test the F# test suite. You can also try your UWP/Xamarin Forms apps. Generally avoid printf etc. and ToString() on F# record types and discriminated unions.

Please report issues in this (CoreRT) repo. If issues in .Net Native can be fixed in CoreRT, the fixes can probably be moved over into .Net Native.

Workarounds

• F# can be used to create non-Store UWP apps that are sideloaded onto Windows PCs that are in "developer mode".
• F# can be used to create Store apps using non-UWP technology (using desktop bridge with e.g. WPF or javascript transpliation with e.g. Fable/React Native).
• Xamarin Forms has a preview of WPF support. Whether this becomes a viable platform depends on support by 3rd party controls.
• (Speculative) In the .Net Core 3.0 timeframe, UWP controls can be used in WPF apps. Presumably .Net Native is not required. We may be able to create a store app using WPF (desktop bridge) hosting a UWP app.

[charlesroddie]

@jkotas @MichalStrehovsky Can you comment on the CoreRT bit? Will getting F# support in CoreRT make .Net Native support much easier? If that's true I'm sure we can get at least F#/CoreRT testing going quite rapidly.

[forki]

While I'd love to see F# support for .NET Native I think it's not needed for UWP.
If you really want to do that then you can use the workaround with Fable and React Native.

[zpodlovics]

Take a look at this, last time I checked the trivial helloworld example at least compiled:
#5648

However a "working" helloworld nowhere near what's needed to have real .NET Native / CoreRT support for F#. What is the current status of tailcalls in .NET Native and in CoreRT? The next project could be port the F# testsuite to CoreRT. Unless radical changes, at progress rate, you'll probably better if you learn everything and write your own compier for F#...

[forki]

regarding tailcalls: fable shows that this is not really needed (at least a lot can work without that optimization).

[martinsuchan]

@forki Depends what's your definition of UWP. As the original comment says in the Workarounds, it's possible to use F# in Desktop WPF/Xamarin apps and publish this app using Desktop Bridge into Microsoft Store. In a similar way it's possible to use F# and transcompile it into JavaScript together with React Native.

But the goal of this issue is full F# support in .NET Native UWP apps, without Desktop Bridge or JavaScript.

[forki]

what I'm trying to say is: this shouldn't be framed as UWP issue.

[7sharp9]

Xamarin mobile does not output tail calls either, its always been recommended to turn that option off as it has no effect.

[charlesroddie]

@forki I have added a working definnition of UWP to the OP, distinct from "Store" which includes JS and desktop bridge. If you don't like this definition just read "native UWP" where UWP is written and everything should make sense. I put Fable and React Native explicitly in the workarounds.

[Mike-E-angelo]

Really nice work with this post here, @charlesroddie. Thank you for taking the time to write it and for considering all the issues surrounding it. 👍

[jkotas]

Can you comment on the CoreRT bit? Will getting F# support in CoreRT make .Net Native support much easier?

Yes, it would make it somewhat easier. .Net Native shares many CoreRT parts. We have issues #2057 or #3929 opened about it.

If you have feedback about F# support for UWP specifically, the uservoice issues are better place to comment on it than this repo.

[Noemata]

F# was (and still could be) a cool way to develop XAML based software solutions. UWP is the XAML future of Windows, with Winforms (pre-XAML), WPF and Silverlight all being dead ends at present. As things stand F# has no future on Windows. It's really that stark and that stupid. Any suggestion that desktop bridge is a viable option is like saying hydrogen filled balloons are a viable way to travel. Yes, viable if like going slowly, in whatever direction the winds are blowing and plan on eventually blowing up.

[forki]

like saying hydrogen filled balloons

... Meanwhile others just ship apps happily...

[touseefbsb]

Functional programming seems to be an improved revolution in programming world ( I may be wrong ) and if we can incorporate that into uwp apps I think it would be great, I dont care whether F# support is brought for uwp or maybe c# 8 gets all functional features which F# has, ultimately making it easier for both world, maybe then f# developers can start using c# as well and united the .net community under 1 language which can do it all.

[DjArt]

Another argument in support of .NET Native compiling of F#: Windows 10 IoT. Even when we can install debug version of app, this is not suitable for commercial purposes.

[nbevans]

It would be great if Miguel or Don could get some heads together in MSFT on this one. It seems like it could be fixed relatively easily now.

[zpodlovics]

I am afraid there will be no easy solution here. I did some digging by porting a few F# core tests to CoreRT. Take a look at this issue for example: dotnet/fsharp#4954

[charlesroddie]

@nbevans 1st step is to test CoreRT. I am more optimistic than @zpodlovics because so far only minor issues have been found (limited support for non-zero based arrays and printf). @FoggyFinder can start helping with testing now. The CoreRT team is usually great at fixing issues.

In this way we can make it as easy as possible for the F# and .Net Native teams to finish the work, but they have to put in the small amount of effort it will take to do this. @dsyme critical to have some leadership on this.

[nbevans]

Agreed the printf issue looks pretty minor to me. Is that the only blocker in FSharp.Core? Nice!

[cartermp]

@charlesroddie Just as a note w.r.t the main issue, the following should probably change:

UWP is actively developed (the platform itself and 3rd party components), unlike WPF.

The most recent announcement is that the .NET team is porting WPF and WinForms such that their "base" .NET dependency is .NET Core, not .NET Framework. Kind of hard to word without sounding confusing (it always sounds confusing whenever I say it), but they'll run on .NET Core on Windows only.

The recent UWP XAML Islands may also be worth calling out here. You will be able to host UWP XAML controls, which are modern, inside WPF and WinForms apps (running as Win32). This "unblocks" F# from some of those recent innovations.

So the problem is less severe for people who just want to do modern Windows 10-based desktop app development, and also want some of the recent UWP innovations but don't require the use of .NET Native.

@DjArt

Another argument in support of .NET Native compiling of F#: Windows 10 IoT. Even when we can install debug version of app, this is not suitable for commercial purposes.

I don't recall the exact solution off the top of my head, but my understanding is that you can use Windows 10 IoT to some extent with CoreCLR (and not just as a debug build). This has unblocked some customers in the past. Additionally, .NET Core runs on various IoT devices and targets that may already unblock you.

[charlesroddie]

@cartermp This is worth following. I had added a point to "Workarounds" after Build 2018. I will update when we get more information on release dates. It's possible that you could take a Xamarin.Forms.UWP app, create a Xamarin.Forms.UWP control out of it, and put that inside WPF using the wrapper they plan to release. It depends on several things working and future developments so I'd say it's still speculative.

However we can instead just solve the problem. If the community and the CoreRT team can get close (with CoreRT/F# support) we will need your help to finish it off within .Net Native.

[nbevans]

Isn't that the Desktop Bridge (or some derivative) solution? Irrespective of debatable elegancy/correctness of that approach; it would still rule out using F# to produce UWP components meant for third party consumption.

[DjArt]

Nice news. I have workaround for this problem. That doesn't solve them, but in some cases can help.
You should find and open file "C:\Program Files (x86)\Microsoft SDKs\UWPNuGetPackages\runtime.win10-x64.Microsoft.Net.Native.Compiler\2.1.8\tools\x64\ilc\tools\GatekeeperConfig.xml". File may have another location depends on version of compiler or target architecture.
After, you should find block

<FSharpRule on="true">
<ErrorLevel>error</ErrorLevel>
<Message>Rule_FSharpRule</Message>
<Category>FSharp</Category>
</FSharpRule>

You can delete this rule, change on to false or set ErrorLevel to warning. After, save changes and try to compile your project. Something in your project can don't work at all or works strange, but remember that we unlock some unfinished features.

[nbevans]

Looks like doing that disables the fail-fast when it encounters a F# assembly. So possibly then it will at least "try" to compile it with .NET Native but am unsure what happens when it encounters a tail call instruction, or deeply nested generics, or any of the other unsupported MSIL code patterns.

[charlesroddie]

Great find @DjArt ! @nbevans : let's find out!

[DjArt]

@nbevans Yeah, of course. But something better then nothing. Not all F# code uses tailcalls or something else.

[nbevans]

I also see in that file this:

        <TailCallRule on="true">
          <ErrorLevel>warning</ErrorLevel>
          <Message>Rule_TailCallRule</Message>
          <Category>RuntimeSupport</Category>
        </TailCallRule>

[charlesroddie]

OK what we need to start with I think is a github repo with three projects:

1. an F# .Net Standard library, initially containing "hello world".
2. A CoreRT project that references 1, with instructions on how to build and run.
3. A UWP project that references 1, with instructions.

That will allow users to fork and add their own tests and we can add more interesting tests. 2. and 3. may as well be standard C# projects.

@FoggyFinder can you try this please?

[FoggyFinder]

@FoggyFinder can you try this please?

Yep, I can try but I'm not sure about the necessity of this steps.

I'm a bit new here, but that's how I understand it now:

1. For simple F# app the steps are the same as for HelloWorld

So testing simple app is not a problem.

2. There is a problem with printf #4954
   Not sure I understand why this issue is opened there. I guess it is problem with CoreRT not FSharp.Core, isn't it?

So, I think it make sense to create new issue with proper repo here.

[FoggyFinder]

Not sure is it bug or not but request for name for type of some type return the weird result:

typeof<int>.GetType().Name //$BlockedFromReflection_0_fc985ada 

I guess it has to return RuntimeType as in Core

[abelbraaksma]

My guess on that last issue is that type aliases don't work. I'd assume it works correctly when you use the original type directly?

[FoggyFinder]

nope, as it's not related to F#, I created separate issue - #5951

[charlesroddie]

@zpodlovics Agreed that we need an efficient way to do this, and agreed that we need to substitute versions of printf/sprintf etc. that can run on CoreRT and UWP. Great if you can propose or implement something. @FoggyFinder and I can try to help. I am a bit out of my depth though having no experience of testing or build processes.

The couple of examples I tried were done manually, commenting lines out until the code could run. UWP doesn't give any useful exception information so I can just see success or failure of each compile and run as a whole, and each compile is time-consuming. Do you see a way to automate the process?

[FoggyFinder]

Events also do not work:

open System

type ChannelChangedHandler = delegate of obj * int -> unit
type C() =  
    let channelChanged = new Event<ChannelChangedHandler,_>()

    [<CLIEvent>]    
    member self.ChannelChanged = channelChanged.Publish
    member self.ChangeChannel(n) = channelChanged.Trigger(self,n)

let test() = 
    let c = C()
    let h1 = ChannelChangedHandler(fun _ ch -> Console.WriteLine("Channel = {0}", ch))
    c.ChannelChanged.AddHandler(h1)
    c.ChangeChannel(3)

Exception:

Unhandled Exception: System.TypeInitializationException: A type initializer threw an exception. To determine which type, inspect the InnerException's StackTrace property. ---> System.ArgumentException: The input sequence was empty.
Parameter name: source
   at Microsoft.FSharp.Collections.SeqModule.ExactlyOne[T](IEnumerable`1) + 0xd6
   at ConsoleApp!<BaseAddress>+0x21d0be
   at ConsoleApp!<BaseAddress>+0x1d99d0

   --- End of inner exception stack trace ---
   at ConsoleApp!<BaseAddress>+0x1d9a78
   at ConsoleApp!<BaseAddress>+0x1d98a4
   at Microsoft.FSharp.Control.FSharpEvent`2.Trigger(Object, TArgs) + 0x23
   at Program.main(String[]) + 0x58
   at ConsoleApp!<BaseAddress>+0x271b02

probably for the same reason as printfn

[charlesroddie]

@charlesroddie The F# CheatSheet works under UWP and CoreRT.

Our Xamarin.Forms app also works on UWP, after removing sprintf, and .ToString() and Newtonsoft.Json serialization on record types and DUs. We'll need to try the usual approaches to getting serailization to work.

So early testing suggests that F# basically works under UWP.

Main remaining step: see how much serialization and other string methods we can get working using rd.xml files on UWP and CoreRT and document what is needed.

[ArtGangsta]

If it looks like Microsoft is treating C# as a first class language compared to other .NET languages, some of time, perhaps a very small percentage of the time, but as we see it can happen occasionally.. Perhaps a solution would be an F# to C# converter that can handle some very large subset of F#.

[charlesroddie]

@ArtGangsta You haven't read the OP or the latest posts. This issue is very close to solved.

[charlesroddie]

@jkotas @MichalStrehovsky F# works with .Net Native with minor workarounds. Can Microsoft edit GateKeeperConfig.xml in line with @DjArt 's post? Can you ask the .Net Native group to do this? Thanks.

[jkotas]

Looking at the workarounds required - I do not think it works well enough to declare F# as supported by .NET Native in shipping Visual Studio.

[dsyme]

@charlesroddie Please write a comprehensive summary of the known issues please. Perhaps start a new issue since this one is very very long and old. The documentation should be precise, i.e. the sort of thing that would describe the full set of gotchas to a new user

@jkotas We need a transparent process by which an assessment is made, e.g. who is the responsible decision maker(s), what is the full rationale, list of remaining issues, whose responsibility it is to work on issues and so on.

Looking at the workarounds required - I do not think it works well enough to declare F# as supported by .NET Native in shipping Visual Studio.

The question is not about F# generally, but about the specific ban on FSharp.Core library within UWP apps. FSharp.Core is just a .NET DLL, and a relatively harmless one.

As far as I can tell, FSharp.Core now works as well as any other .NET library is guaranteed to work on .NET Native. That is, .NET Native is, by design, an incomplete implementation of .NET (indeed so is .NET Core, which raises PlatformNotSupportedException for some functionality). This means each specific library may well have specific issues, and this applies to C# libraries as much as FSharp.Core.dll. As long as those issues are thoroughly documented I don't see any specific problem with allowing the use of the DLL. We never ban any other specific libraries from running on .NET Native, do we? If so, why? Why would we now single out FSharp.Core?

[dsyme]

Note we may be willing to accept from the community a few small tweaks in FSharp.Core.dll (or a separate build of FSharp.Core.dll for UWP apps) in order to progress this.

[charlesroddie]

@dsyme Agreed that we need a comprehensive summary. Without this my request was a bit premature. We can do this over the next week.

[jkotas]

transparent process

.NET Native for UWP is a closed source product as a whole. The process for it is as transparent as process for other closed source products. I have said above #5780 (comment) that the user voice issues are a better place to comment on it. They are more direct connection to the decision makers on this than this issue.

If you would like to have more transparent .NET runtime for UWP apps, you may consider asking to allow publishing UWP apps with .NET Core instead. It would make F# work as good as it works on .NET Core, no .NET Native specific workarounds required.

We never ban any other specific libraries from running on .NET Native

That's not correct. The gatekeeper tool uses other library names as canary to flag other unsupported cases as well.

[charlesroddie]

the user voice issues are a better place to comment on it.

Well that is not true. The uservoice issues and other feedback mechanisms have just been a huge time sink and more of a crowd control device than a decision making process. In 3 years nothing happened through those mechanisms, whereas the community got it basically working in two weeks once one of us found a back door.

[dsyme]

.NET Native for UWP is a closed source product as a whole. ...

@jkotas Yes indeed :)

The gatekeeper tool uses other library names as canary to flag other unsupported cases as well.

Are there any public docs on which libraries and why? Thanks.

... user voice issues are a better place to comment on it. They are more direct connection to the decision makers on this than this issue.

I think a good technical summary plus adding clarity on the UV thread would be appropriate. When I've spoken to UWP people in the past they've actually been very positive about F#. If the F# community provide technical clarity it's possible progress can be made, since the UWP team have always asked for that.

Whereas the community got it basically working in two weeks once one of us found a back door.

There is something important here: I do fundamentally trust the F# community to deal with as many technical issues as can be dealt with on the F# side of things, and to provide good and active guidance to potential F# adopters. This approach to permitting FSharp.Core to execute would fit with the overall approach taken to F#.

[jkotas]

Are there any public docs on which libraries and why?

https://docs.microsoft.com/en-us/dotnet/framework/net-native/migrating-your-windows-store-app-to-net-native

[dsyme]

Are there any public docs on which libraries and why?
https://docs.microsoft.com/en-us/dotnet/framework/net-native/migrating-your-windows-store-app-to-net-native

@jkotas Great thanks. The FSharp.Core issues are much smaller and more technically constrained/explainable than those on that page. So IMHO a good technical doc should be enough to unblock use of FSharp.Core.

[MichalStrehovsky]

F# works with .Net Native with minor workarounds

While it would be a very pleasant surprise that we only need minor workarounds, I have to ask: What was the testing done to support this statement? Is it based just on the small piece of code in the FoggyFinder/FSharpCoreRtTest repo or did you run more substantial workloads, exercising the typical code patterns for F# (deeply nested and recursive generics, patterns that rely on tail calls, etc.)?

[FoggyFinder]

While it would be a very pleasant surprise that we only need minor workarounds, What was the testing done to support this statement?

@MichalStrehovsky I think it's refer to starting testing F#. Without workaround it was impossible.

[charlesroddie]

What was the testing done to support this statement? Is it based just on the small piece of code in the FoggyFinder/FSharpCoreRtTest repo or did you run more substantial workloads, exercising the typical code patterns for F#?

Also a real app that contains more typical code. A symbolic algebra engine (which tests deeply nested discriminated unions), normal usage of functional and object oriented F# features (first class functions, DUs, record types, generics, to an average level of complexity).

deeply nested and recursive generics, patterns that rely on tail calls, etc.

We have not tested tail recursion or deeply nested generics explicitly. I assume the first is not worth testing because is just not supported (unless you know of any change), but the second would be. We haven't set out deliberately to stretch the compiler yet in any tests.

We still need to document all this.

[charlesroddie]

Documentation here: https://github.com/FoggyFinder/FSharpCoreRtTest/blob/master/README.md#description-of-current-issues

To summarize, we have idenfied three issues affecting F# in .Net Native:

1. sprintf (easily worked around using concrete string methods; in future we could investigate rd.xml and changes to fsharp.core)
2. tail instructions (apparently ignored - doesn't affect the correctness of typical F# code)
3. Quotations to Expressions (used for LINQ: workaround is to avoid LINQ or use C# for LINQ)

No other issues have come up in testing.

@dsyme @MichalStrehovsky @jkotas Do you need anything else? Can we get official editing of GateKeeperConfig.xml now?

[charlesroddie]

Discussion moved to #6055 at @dsyme 's request.

[Mike-E-angelo]

Really great work here, all. @charlesroddie you made a point at the outset to make sure that we were all civil here and I think that objective was met in full esteem. This was a very great display of open source culture at work, IMO. Thank you all again for your diligent work towards this this very valuable addition to the .NET ecosystem and its brand. 👍