[All languages] - Universal package management approaches

[alexswan10k]

With the Rust/Dart/Python targets becoming less immature by the day, it has got me thinking... Perhaps it is time to revisit the package management/distribution debate in the light of an approach that could work for all targets.

I am not an expert on any of this, so hoping those that know these systems inside-out will be able to chime in and correct me where I am talking rubbish! I will go back and update this post as new info comes in.

State of the world today

Nuget packages ship binaries only by default. This is not compatible with Fable. In order to make a library compatible, F# sources need to be included, which can then be compiled by the consumer. This is somewhat of a manual process, but there is an interesting discussion on improving this here #2939. Discoverability is also quite limited

https://fable.io/docs/your-fable-project/author-a-fable-library.html
https://fable.io/community/
https://fable.io/resources.html#examples

Previous discussions

#1649
#856
#2939

I should also mention Femto at this point, which already bridges the impedance mismatch between ecosystems (npm/yarn vs Nuget/Paket) in order to create a more unified experience. This approach seems to be valid for bindings.

A brief summary of target ecosystems

Each compilation target generally has its own ecosystem for managing dependencies.

• Python - Poetry
  • .. todo
• Dart
  • todo
• Javascript/Typescript - Npm/Yarn + webpack/browserify/snowpack..etc
  • Distributes compiled js plus .d.ts files for guiding target type system.
  • No F# but no constraints on what can be shipped with a npm package in theory
  • Allows aliasing package names & transitive dependency collisions
• Rust - Cargo
  • Distributes source code
  • Can distribute other assets but requires proprietary build script + generally non-standard to do
  • Allows aliasing package names & transitive dependency collisions
• Php
  • todo
• .NET
  • Distributes compiled .dll's which can be directly loaded by the consumer
  • Language agnostic (works for C# and F#), but can not be consumed by Fable
  • No transitive dependency collision or aliasing support. User has to manually pick 1 version and hope it works for all dependents

Some other hypothetical examples that may help generalize the approach

• Lua
  • Luarocks - I believe this is not necessarily even supported for all targets
  • Modules must be in subfolders, and cannot be anywhere
• C (yeah nuts I know, but since Rust is possible, perhaps this is too!)
  • No module system at all, could an F# focused solution work here?

Frustrations with Nuget/Paket and the .NET ecosystem

One of the core pain points many developers hit eventually is the transitive dependency collision problem. This happens when you have something akin to the following dependency tree:

• Your project (v1)
  • Dep A (v1)
    • Dep C (v2)
  • Dep B (v1)
    • Dep c (v3)

Nuget - Do I use V2 or V3? If you use V3, it breaks Dep A, if you use V2, it breaks Dep B. Checkmate!

Due to the fact that nuget is heavily opinionated on how the .NET CLR loads dll's, it forces the world-view that all libraries are global, and thus there can only ever be 1 source of truth when referring to a namespace/type. Although this is less of a problem in F# because dependency tree's are generally less deep, this seems to me to be a fundamental flaw which is unlikely to ever have a good solution at the .dll level. This puts limits on the complexity of libraries, and their ability to reuse existing code.

More modern ecosystems (such as js - npm and Rust - cargo) get around this problem by supporting scoped, or aliasing of package names, and referring to packages in code by their aliased name.

Questions

• - Is the current approach workable for other languages, including target-specific build switches etc?
• - Can the bar be lowered for package author's to make their packages Fable compatible
  • 1 liner to .fsproj file is probably as good as it gets. This is documented in Authoring section.
• - Is there a way to improve visibility on which packages are compatible with Fable, and if so are they constrained to 1 target?
• - Is there a solution to the transitive dependency problem, is it even important?

[MangelMaxime]

Hello @alexswan10k,

I don't understand what you are suggesting.

Something important with Fable is that it live inside of F# ecosystem and Fable is not an independent language. For example, Fable doesn't really compile or resolve dependencies (I simplify I know). It is the F# compiler which does it, then generate the AST and Fable kicks in from this step.

Distributing the Fable libraries via NuGet allow you to have access to all the F# tools for free: IDEs, analysers, linters, formatters, etc.

---

Currently, there are mainly 2 scenarios when distributing Fable libraries:

• Binding, which are done using normal NuGet package with only DLLs
• Libraries, which are distributed over NuGet with DLLs and F# source code in them hosted under the fable folder

I don't see why targeting another language other than JavaScript would not work the same way it does with JS. Fable 3 and 4, emits files on the disk so it is as if you wrote them manually.

It is true that if one of the library you use depends on a target libs (like React, Express, etc.) then you need to install both the Fable NuGet + the target lib via the your preferred dependency manager (NPM, Yarn, Cargo, LuaRocks, etc.).

Having our own dependency manager would mean having to fork or adapt the existing F# ecosystem, change how F# works so have a custom compiler, make adaptation to IDEs, and a dependency manager is a really complex piece of software to create and maintain. Especially if you want to be multi target.

In the past, deciding to embrace the JavaScript ecosystem tools and standard has always proved to be of a huge benefit for Fable. And I do think, the same will be true for others languages.

IHMO the current position of Fable, allow it to benefit from both F# tooling and target ecosystem tooling for "free". And allows, us to focus on only the specific job of Fable which is translating the F# to language X.

[MangelMaxime]

If a library is written using pure Fable compatible code it will works across all the target language the same way.

For example, if my library consist of:

module SimpleMath

let add a b = a + b

and I ship it to NuGet.

I will be able to consume it for JavaScript, Rust, Python, etc the same way.

If needed compiler directives can be used to make adaptation depending on the target language:

let log msg =
#if FABLE_JS 
    console.log msg
#endif

#if FABLE_Python
    print msg
#endif

[alexswan10k]

Hi @MangelMaxime,

I do apologise I think my knowledge on this may be a little stale, as this has flip-flopped a couple of times over the lifetime of Fable. I have updated the original ticket to reflect your observations.

So it does appear that F# code can just be distributed with a Nuget package, and compiled by the consuming system on demand (a little similar to the Cargo approach, but without internal knowledge of modules). Assuming the target language has working module support, this should just work out of the box as you pointed out. This is a pretty good place to be if so.

I guess maybe the real question then is around refining the user experience of authoring and finding these packages as mentioned here. It was not obvious to me where nuget libraries support or do not support Fable (and which would be language agnostic), which is a fundamentally different ecosystem than .NET.

[MangelMaxime]

@alexswan10k

I do apologise I think my knowledge on this may be a little stale, as this has flip-flopped a couple of times over the lifetime of Fable. I have updated the original ticket to reflect your observations.

No problem, it is true that it changed several times. The NuGet distribution is here since 3 years I think and has been pretty stable since them.

Can the bar be lowered for package author's to make their packages Fable compatible

I still think that not much can be done on this side. Right now creating a Fable package is just about adding:

<!-- Add source files to "fable" folder in Nuget package -->
<ItemGroup>
    <Content Include="*.fsproj; **\*.fs; **\*.fsi" PackagePath="fable\" />
</ItemGroup>

to the *.fsproj.

For pure bindings, nothing special is required.

Is there a way to improve visibility on which packages are compatible with Fable, and if so are they constrained to 1 target?

There has been some initiative in the past:

• Fable awesome which have now been integrated to https://fable.io
• https://fable.io/community/ (provide a basics search)

However both of them require a manual editing and UX is not that great.

IHMO, the good solution would be use tags when authoring Fable package:

<PackageTags>fable;fsharp;json;fable-js;fable-python</PackageTags>

And then have a website/tool which send search request to NuGet.org server.
This means that if we want to list all Fable package we can search for tag fable. If we want to limit to JavaScript Fable packages fable-js is the tag to search for etc.

Is there a solution to the transitive dependency problem, is it even important?

I would vote for "not important" because I don't think I encounter this problem in the past. 🤞

[alexswan10k]

Thanks for this, it fills in a lot of gaps. It looks like most of my concerns basically have solutions today, it's the age-old documentation problem of "you cannot search for what you do not know about" I guess. I noticed a lot of these details are documented too, which is great. I just had trouble finding them. I could not find https://fable.io/community/ though from the main site.

The tagging/nuget api query on tag solution seems like an excellent idea to me.

[MangelMaxime]

About https://fable.io/community/ it never really worked and when merging "Awesome Fable" with the rework of fable.io Alfonso and me kind of agreed on making it deprecated.

Because, it would mean that we need to maintain the list of items manually in 2 different places and also it doesn't fit the look right now and would require work etc.

IHMO, if something is to be done or happen it is to experiment with the tagging/nuget API. Because, this would also a single source of truth without any work require from the maintainers. And also, the experience can be deeply customized if needed.

[ncave]

Thank you @alexswan10k for your questions and thoughts, and thank you @MangelMaxime for your detailed response!
I absolutely agree with what @MangelMaxime said about Fable benefiting from both F# and target ecosystems for "free". I believe that was what @alfonsogarciacaro's goal was from the very start.

My personal opinion has always been that Fable should just be a transpiler, and as such, stay within the design goals that other transpilers like TypeScript have, as stated in their "non-goals" (i.e. their "we don't want to be that" section), more specifically "non-goals" 4 through 7, which IMO can be adopted verbatim in Fable documentation.

[alexswan10k]

Thanks, both, and I do agree with reusing the F# ecosystem where possible etc. I guess I am also just aware that there are limitations due to external constraints such as .NET dll loading behaviour (everything global), which will forever put F# at a disadvantage against its competitors that have a more modern package management philosophy (locally scoped packages eg npm or Cargo). This is absolutely not a Fable problem though, it just turns out Fable is not theoretically constrained by these shackles, which is an opportunity in my view.

Anyway, nuget querying seems like an obvious place to improve matters. Perhaps I can take this away and have a look at crashing out a page for integrating with https://api.nuget.org/v3/ for the fable.io site. I imagine this should be pretty straight forward, and can probably be done through the public api, and thus be entirely client side. I will report back..

[alexswan10k]

Looks like it is possible
https://stackblitz.com/edit/react-ts-d8llqx?file=Search.tsx
https://react-ts-d8llqx.stackblitz.io

Nothing is using tags at the moment, so we might want to fallback on using query as a stopgap.

I will put some UX fluff around it anyway. How would one host something like this in the core site? It looks pretty much markdown-only from a cursory glance. Can we run react components?

Edit - Looks like you cannot query tags according to the api docs. You can query packageType though. Is this an option?

[ncave]

@alexswan10k I don't see why not, but existing Fable packages will have to be republished with a package type so they can be discoverable this way. Also, I don't see package type listed anywhere in the existing UI on nuget.org or nuget.info.

[MangelMaxime]

For hosting, it can be done two ways.

Either inside the current site, because we can generate HTML or load JavaScript on a specific page. For example, this page from Fable.Form is just using the doc generate to have the navbar generated and all the page content is actually an SPA.

Or, we can create a new repo inside of fable-compiler org to host it separately. For example, https://fable.io/packages.

The benefit of the first solution is that the navbar and style are handled by Nacara (doc generator) and so it will easy to have the same look and feel between that and this specific page.

The cons is that the repo will not contains simple documentation anymore but also the source code for that search packages page.

IHMO, it is better to host that specific page/section under it's own repository. It will make it easier to accept contribution, prototype stuff, and if necessary deviate from the standard styles to improve the UX. We just need to use the same color and basic style to have a consistant look and feel.

[alfonsogarciacaro]

Thanks everybody for your comments! Yes, as @MangelMaxime and @ncave say we should likely avoid trying to build our own package manager and reuse existing infrastructure instead. Package management is one of these deceiving problems that are much more complicated than they look at first, as Paket maintainers can corroborate ;) However, if we can do something to improve the experience of Fable users, particularly around discoverability as @alexswan10k says, that'd be great.

Some notes:

• Using Nuget: As commented, using Nuget is the best best as it's tightly integrated with F# tooling. Nuget packages don't include sources by default but this can be done with a small addition to the .fsproj so it's not a big deal. In the very beginning we tried to distribute Fable packages through NPM but this made the experience not so good with IDEs. Another issue is, although most of Fable related packages are Fable-only, there are some that are standard dotnet packages with Fable support (like Fsharp.Control.AsyncSeq or type providers), so these will always be distributed by Nuget.

• Transitive Dependencies: For F# usually Paket is used to solve this. Fable just invokes MSBuild to resolve the dependencies, which will invoke Paket in its place if it's installed in the project.

• Cross-platform Dependencies: It's been problematic when a Fable package had a dependency of an NPM package because we could only tell the user about this in the README and hope they would install the proper versions. Femto was born to solve this issue, by including metadata in the .fsproj, although it's an extra configuration and I don't how many users are actually using it. In any case, it'd be nice if we could extend the tool to support other languages.

• Compilation Performance: Fable needs to compile all packages by source. This hasn't been much of an issue with JS because users try to keep apps and the bundle small, but it may start to be more frequent issue with Rust or Python projects. It's a tricky problem we've discussed some times. The AST for inlined functions can be serialized now, but we still have the problem we cannot guarantee backwards compatibility, so distributing the compiled form is still not an option (another issue is some Fable packages generate different code in DEBUG or RELEASE modes). But for the future we can consider having a local cache where we precompile packages for different Fable versions.

• Discoverability: As @alexswan10k says, this has been one of the weakest points in Fable ecosystem. We've tried several ways, but anything that is a separate step from the actual package publishing requires extra maintenance and becomes stale quickly. So using Nuget API is a great idea, either by checking package tags/type or directly the package description. We also need to remember to include the README in the packages now that Nuget finally made it possible. I myself haven't checked yet how this is done so my packages don't include a README 😅 (unfortunately with Nuget, unless the IDE helps you, you need to remember the particular tags used in the .fsproj for all these operations: including readme, icon, licence, etc)

[alexswan10k]

Thanks all for the input.

I have been pushing forward the discoverability angle, and I think I pretty much have a working search page (using tags after all) that talks to the nuget api. The big problem now is going to be agreeing on all the tags, and getting package authors to correctly follow the convention :)

I propose the following convention for tags:

fable-target:all - No proprietary code. Should compile for all future targets as long as language features supported
fable-target:lang (eg js/dart/python) - Has proprietary code, usually Emit statements for interop etc.

This convention could also be extended to cover bindings

fable-binding:js
fable-binding:php

Here is the working page:
https://react-ts-d8llqx.stackblitz.io

Editable:
https://stackblitz.com/edit/react-ts-d8llqx?file=Search.tsx

I crossloaded the bulma stylesheet from fable.io by the way, so perhaps not the most elegant, but it gets the same component styles and colour scheme etc for free. It is pretty rough around the edges right now, but hopefully, the intent is self-explanatory. We can always do more polishing down the road.

Happy to take direction from here, let me know. Should I make a repo?

[MangelMaxime]

I find that -target suffix add verbosity without much gain compare to just fable.

Personally, I would go with:

• fable: No proprietary code. Should compile for all future targets as long as language features supported

• fable:lang (eg js/dart/python) - Has proprietary code, usually Emit statements for interop etc.

  For the lang suffix I would either use the file extension for all:

  • fable:js
  • fable:dart
  • fable:py
  • fable:rs

  The full name for tall the target for consistency

  • fable:javascript
  • fable:dart
  • fable:python
  • fable:rust

-binding suffix is good to me. We just need to use the same lang convention.

---

UX related:

1. It would be nice if the logo container size could be consistent.

• Right now, if a package doesn't have a logo it feels a bit cramped.
• Not all logo have the same size
  • Elmish: 100x100
  • Fable: 100x73

See I think we should decide on a height and make the logo adapt the best it can to that height.

2. Blue text over blue background is difficult to read.

This is fine for the first text "This will search Nuget.org for any packages with the following tags" but for the main content I don't think this is good.

I wonder if we could use a similar style as for the Blog list for the boxes or if that would be too heavy.

[alexswan10k]

Updated as per feedback

https://github.com/alexswan10k/fable-nuget-search

[alfonsogarciacaro]

FYI: Femto already supports Python packages 🎉 https://twitter.com/zaid_ajaj/status/1567101357470978053

[MangelMaxime]

Thanks to the work done by @alexswan10k I think we can say that it is possible to search for packages using the NuGet API.

We can move the discussion to https://github.com/fable-compiler/packages

I created an issue with a specification proposition: fable-compiler/packages#1

About the design/UX of the tool itself, I have some ideas how we can improve what has been started by @alexswan10k.

[alexswan10k]

Great. Thanks @MangelMaxime.

Do let me know if you need me to move anything, transfer access, host, or whatever.. Looks like it the target repo is empty? Maybe it is just that old eventual-consistency thing :) Will check back in a bit.

[MangelMaxime]

I just created the repo to have a place dedicated to that tool and especially for the hosting the specifications etc.

I have not started yet to upgrade/write the tool itself yet.