Custom Fable TFM

[jwosty]

NuGet Product(s) Involved

NuGet.exe

The Elevator Pitch

Hello NuGet team! I'm representing the Fable project (the popular F#-to-Javascript transpiler), and we want to look into having a custom TFM for fable specifically, so that we can integrate it better into the NuGet/MSBuild ecosystem (similarly to how Tizen, Andriod, iOS, etc works).

Additional Context and Details

I previously opened a discussion item for this here. More of the gory details and relevant links are in there. We would really like to have a conversation with someone from the NuGet team about this.

[nkolev92]

Hey @jwosty,

NuGet owns the mapping for the frameworks, but we need a more general agreement that a Fable is something valuable enough and different from the current platforms approach.

An example is the recent addition of the .NETnanoFramework in here: #10800.
I think we'd need a more involved proposal, including a mapping, and some details about the usage etc.

cc @terrajobst

[terrajobst]

@jwosty

What is your expectation of compatibility?

Say I've a project targeting the new Fable TFM. What other projects can I reference / which packages can I install? Can I install nestandard2.0? net5.0? If the answer is "none", then this is easy because Fable would basically be its own universe.

But if the answer is not none, then we need to talk about how these mappings evolve over time.

[zivkan]

similarly to how Tizen, Andriod, iOS, etc works

android and ios no longer have a target framework identifier, they're platforms within the .NETCoreApp TFM. There used to be old Xamarin TFMs, but that's obsolete since .NET 5/6 with net6.0-android. I don't know what's going on over at Tizen, but while there is a tizen TFM, there's also Tizen MAUI integration, where your project targets net6.0-tizen.

If fable can fit within the "platform" concept, then no changes are needed to NuGet. It will Just Work when the appropriate MSBuild properties are set.

[jwosty]

Hey @jwosty,

NuGet owns the mapping for the frameworks, but we need a more general agreement that a Fable is something valuable enough and different from the current platforms approach.

An example is the recent addition of the .NETnanoFramework in here: #10800. I think we'd need a more involved proposal, including a mapping, and some details about the usage etc.

cc @terrajobst

There's some relevant prior discussion here: fable-compiler/Fable#3549 . There's still some open questions in there but I hope it provides some context. This is more of an ask for help and guidance as to what some good options if this were to be done, rather than a formal request (hence opening it as a discussion originally instead of an issue).

Fable already has a great ecosystem of packages that work with it, but it currently just uses a NuGet tag ("fable") to tell if a package is fable compatible (essentially a poor man's TFM). See a list here: https://www.nuget.org/packages?q=Tags%3A%22fable%22 ; looks like the current count is 494 packages. The NuGet TFM on these packages is actually a lie. There are drawbacks of such an approach (with which we are living), with probably the biggest drawing being that for a package to support both real .NET and Fable, it has to have two separately named packages (for example Thoth.Json and Thoth.Json.Net)

See this comment in particular for two different options for TFM schemes I have been mulling over: fable-compiler/Fable#3549 (comment)

@jwosty

What is your expectation of compatibility?

Say I've a project targeting the new Fable TFM. What other projects can I reference / which packages can I install? Can I install nestandard2.0? net5.0? If the answer is "none", then this is easy because Fable would basically be its own universe.

But if the answer is not none, then we need to talk about how these mappings evolve over time.

It would be its own universe. Referencing a package from a Fable project would need to itself have Fable as a target (because non-Fable packages lack the sources that Fable currently needs to perform its compilation, and not every valid .NET standard program is a valid Fable program).

If fable can fit within the "platform" concept, then no changes are needed to NuGet. It will Just Work when the appropriate MSBuild properties are set.

In terms of MSBuild, yes. But I think even for those NuGet requires itself to be aware of all possible TFMs, right? (I may be wrong and please correct me if so; I last researched this stuff over a year ago)

For reference, here was my attempt at a PoC: https://github.com/jwosty/Fable.MSBuild.Sdk/

[terrajobst]

@jwosty

It would be its own universe. Referencing a package from a Fable project would need to itself have Fable as a target (because non-Fable packages lack the sources that Fable currently needs to perform its compilation, and not every valid .NET standard program is a valid Fable program).

So basically Fable can only consume Fable and can only be consumed by Fable. IOW, Fable is only compatible with itself. That's the easiest model.

To give some background, with .NET 5 we moved away from custom framework mappings in favor of platform-style TFMs (such as net5.0-windows and net5.0-macos. The idea formalizes a common base like netX.Y that can be consumed from netA.B-P where P is a platform and A.B >= X.Y. Before that, NuGet had to know which framework and version is compatible with which other framework and version (e.g. that net461 can reference netstandard2.0 but net45 can't). This was laborious, error prone, and something we had to update each time we shipped. We really don't want that model anymore.

Your model doesn't require that; I just want to set the expectation that if we go with a standalone TFM we can't easily add platform relationships later. You get, of course, the standard version relationship for Fable itself (e.g. fable2.0 can reference fable1.0 but not the other way around).

Does that still sound good?

@zivkan @nkolev92

How much work would it be to add a standalone TFM with no compatibility mapping?

[jwosty]

OK so some of this is coming back to me. I think I had figured out that Fable kind of has to have its own TFM, because if you just piggyback on the "platform" concept, you have to allow referencing the base TFM too, which 100% cannot work for Fable (a fable project cannot reference a net9.0 or even a netstandard2.0 or anything other than another Fable package). Actually, FSharp.Core is an exception treated specially by the transpiler (which basically provides its own implementation a lot of, but not all of FSharp.Core).

Your model doesn't require that; I just want to set the expectation that if we go with a standalone TFM we can't easily add platform relationships later. You get, of course, the standard version relationship for Fable itself (e.g. fable2.0 can reference fable1.0` but not the other way around).

Does that still sound good?

Yep, that is totally fine. Fable can't do that anyway and probably will never be able to tbh.

@MangelMaxime @ncave @nojaf @baronfel

[baronfel]

@jwosty do the different language targets of Fable (js, ts, python rust, etc) need to be substantially different targets from an API surface point of view? And how does the Fable API surface evolve over time as a general rule? Would there eventually need to be fable4.0, fable5.0, and so on?

[terrajobst]

Would there eventually need to be fable4.0, fable5.0, and so on?

I'm putting a stake in the ground by opposing any framework without version. fable implicitly means fable0.0 but if we add a TFM it will implicitly support versions. Change over time is guarantee; it doesn't make sense to me to build a system that wouldn't handle that.

IOW, folks will be able to use fable and it would mean Fable, Version=0.0. By extension, fable1.0 and fable1.1 and so on are syntactically defined as well. Since there is no compatibility matrix, NuGet doesn't need to know them, just be able to parse them. This means the Fable project gets to control which version they will have and when they ship.

[jwosty]

@baronfel

do the different language targets of Fable (js, ts, python rust, etc) need to be substantially different targets from an API surface point of view?

I think so. Since Fable basically just shims the BCL and FSharp.Core with its own definitions, and since this has to be done manually, the supported APIs are the not same across the different targets.

And how does the Fable API surface evolve over time as a general rule? Would there eventually need to be fable4.0, fable5.0, and so on?

Yep, I would imagine the version would just be in lockstep with the Fable compiler's version (in fact version 5.0 is currently in alpha, so I imagine we'd actually be starting with a fable5.0). As far as I understand (as a non-maintainer :) ) the BCL/FSharp.Core shim surface area strictly expands (but is always a subset). But I am not 100% sure about the JS-specific stuff; perhaps @MangelMaxime can speak to that. Browsing Fable on GitHub it seems that there's a Fable.Core which has a mix of such replacements and of its own stuff which is separate for each language target (https://github.com/fable-compiler/Fable/blob/main/src/Fable.Core/Fable.Core.JS.fs).

[nkolev92]

@terrajobst The cost of adding a standalone tfm is pretty low.

I'm curious to learn more about the language targets. Are they something that needs to be expressed in a similar way platforms are expressed, or would all language targets target the exact same tfm?

[jwosty]

@nkolev92

I'm curious to learn more about the language targets. Are they something that needs to be expressed in a similar way platforms are expressed, or would all language targets target the exact same tfm?

I think they would work similarly to .net platforms. Conceptually you can write code that is compileable to all target languages Fable supports (i.e. plain-old-F# using only simple functions and types). It's only when you start using some target language specific things that restricts your code to a particular language runtime. For example: calling Javascript's DOM API, or calling out to Python's NumPy.

I'm honestly unsure whether there should be a single fable-all target, or if you should just have to explicitly list out each target you support. The former would be convenient for users. The latter may be more "correct" because realistically speaking you can't be 100% sure your code works correctly under a target language until you try it (due to fundamental differences in target language runtimes - for example Javascript having only one built-in number type).

[jwosty]

To follow up: would it make sense for NuGet to only concern itself with the general pattern, i.e. fable<X>.<Y>-<language>, allowing Fable the flexibility to add target languages without needing change NuGet? With perhaps the exception of fable<X>.<Y>-all if we were to go down that path (since if I'm not mistaken NuGet would need to know that you can reference that one from all of the others)

[terrajobst]

Is there such as thing as fable.X.Y (without a language)? Or would all use of this new TFM include a language?

[MangelMaxime]

Hello,

I am one of Fable maintainer, I know Fable well but still have trouble with all the NuGet concepts. I don't really know the difference between TFM and platform, and how they work for package resolution. Because of that, I don't understand things like:

Are they something that needs to be expressed in a similar way platforms are expressed, or would all language targets target the exact same tfm?

To clarify a few points from above:

Fable already has a great ecosystem of packages that work with it, but it currently just uses a NuGet tag ("fable") to tell if a package is fable compatible (essentially a poor man's TFM).

Tags

This is not exactly true, the fable tag is only informational to offer a Fable only package explorer via https://fable.io/packages/. This is basically https://www.nuget.org/ but will Fable packages only.

The same applied to others tags like fable-dart, fable-javascript, etc. they are used for filtering the UI. Full tag list

Fable package type

Fable can work with 2 types of NuGet packages:

1. If this is a pure binding then using a standard NuGet package is fine as the .dll contains all the information we need to type check the API.
2. If this is a Fable library, then we need to have a fable folder in the NuGet package which contains the Library fsproj an all the source files .fs, .fsi. The reason is that the .dll is missing some information like do we need to inline that function call or not? By having access to the original source file we are able to get more information.

So basically Fable can only consume Fable and can only be consumed by Fable. IOW, Fable is only compatible with itself. That's the easiest model.

A Fable package can also be a .NET package.

For example, this is the case for FsToolkit.ErrorHandling. This is important because we don't want that because a package is mark as a Fable package to not be able to use it on a standard F# project.

---

And how does the Fable API surface evolve over time as a general rule? Would there eventually need to be fable4.0, fable5.0, and so on?

Yep, I would imagine the version would just be in lockstep with the Fable compiler's version (in fact version 5.0 is currently in alpha, so I imagine we'd actually be starting with a fable5.0). As far as I understand (as a non-maintainer :) ) the BCL/FSharp.Core shim surface area strictly expands (but is always a subset)

Yes Fable API surface evolve over time:

1. We add new BCL API support over time but we don't plan to support 100% of it. This includes API from FSharp.Core (Option, Result, List, etc.) or Standard .NET (System.Math, System.Collections.Generic.Comparer1`, etc.)
2. Fable.Core API evolve over time, we can decide to rework the API surface or add new API to solve Fable specifics problems

---

Something I want to note, if I remember correctly, one of the main reason/benefit for adding a new platform/TFM is the ability to restore different dependencies based on the target runtime.

For example, you could have a Library A which support both JavaScript and Python but depending on the target platform it would need a different dependency list:

Library A
	|- (JavaScript target)
		|- Fable.Core
		|- Fable.Node
	|- (Python target)
		|- Fable.Core
		|- Fable.Python

---

Is there such as thing as fable.X.Y (without a language)? Or would all use of this new TFM include a language?

Because of the above, I think our idea was to always have a language associated with Fable.

The exception, could be for package that can be used on any runtime which is why @jwosty talked about fable<X>.<Y>-all. But a solution to mimic that behavior could be to add all the supported runtime to the package so fable<X>.<Y>-javascript, fable<X>.<Y>-rust, fable<X>.<Y>-python, etc.

And if needed, we could use a custom MSBuild task to automate it for the user. We are already doing some similar automation via https://github.com/fable-compiler/Fable.Package.SDK

Scenarios / Questions

Can a Fable package can consume .NET package?

Currently, this is the case will it be case with the new system or does it means we need FSharp.Core, FsToolkit.ErrorHandling to also be published with a Fable platform/TFM?

Library A (Fable JavaScript only)
	|- FSharp.Core (.NET)
	|- Fable.Core (Fable and .NET)
	|- FsToolkit.ErrorHandling  (Fable and .NET)
	|- Fable.Browser (Fable JavaScript only)

Can we have different list of dependencies based on the Fable target (Rust, Python, JavaScript)?

Currently, we can't support that we need to depends on both which can mess up the compilation sometimes.

Our workaround, is to publish 2 different packages but this doesn't scale well and needs a lot of compiler directives.

Library A
	|- (JavaScript target)
		|- Fable.Core
		|- Fable.Node
	|- (Python target)
		|- Fable.Core
		|- Fable.Python

Can a Fable package be marked to be compatible with any target?

Currently, this is possible. The only requirement is for the Fable package to use only BCL supported API. Note that ATM not all target supports the same list of APIs because some are in beta but in the future we want to have the same level of support.

Library A (Fable)
	|- FSharp.Core (.NET)
	|- Fable.Core (Fable and .NET)
	|- FsToolkit.ErrorHandling  (Fable and .NET)

Summary

• A Fable package can support both Fable and .NET
• A Fable package can target one or more Target (JavaScript, Python, Rust, etc.)
• Fable version can evolve overtime Fable 4, Fable 5, etc.
  • They can be retro compatible
  • It also happens in the past that we had a hard fork meaning that Fable 5 was not compatible with Fable 4, but this is rare so perhaps this will be handled manually if needed

I tried to both answer the questions from above and add my own knowledge of Fable. I hope, I was able to make it somewhat understandable. If more explanations are needed, I stay available.