[RFC] Build Drivers for cabal

[Kleidukos]

Summary

The proposal is about the creation of a new stanza in the cabal file format: build-drivers. This stanza takes a list of executables that run before the build of the Cabal components, in order to provide any required generated Haskell code, foreign C libraries, etc.
The roles of building foreign packages are distributed as such:

• Dependencies: Left to the build system with which the build driver interacts
• Calling other build systems: This is what the build driver does
• Linking: This part is supported by Cabal & GHC when it comes to C-style interfaces, and provides a wide enough coverage of FFI candidates.

Background

Historically, this function has been occupied by Setup.hs and the Custom build-type, using user hooks like "pre-build". As we are now in a development phase of Cabal where we aim to get rid of Setup.hs, I would like to offer a clear and straightforward migration path for our users on this usage of the user hooks.

Rationale

• This helps us get rid of Setup.hs in a clear and predictable way
• We can remove some less-used logic from cabal-install when it comes to custom preprocessor, and don't have adopt new ones should they emerge (WASM, JS, etc)

Example

Here is an example of a Cabal file that uses the future cargo-driver, to build Cargo packages with the Rust compiler:

library
    hs-source-dirs:     src
    default-language:   Haskell2010
    ghc-options:        -Wall
    build-depends:
        base >=4.10.0.0 && <5,
        bytestring >=0.11.0.0
    data-files:
      native/rust-library
    extra-bundled-libraries:    rustLibrary -- the rust library's C interface that we are building with the cargo driver
    build-drivers:
      cargo-driver -- cargo-driver then parses a TOML file that holds its necessary configuration

# Configuration of the cargo build driver
# Every one of those options is over-ridable by environment variables
# But this is in the cargo-driver logic, not in Cabal

# rustLib.a will then be put in the cabal build directory so that Cabal can statically link to it.
extra-bundled-library = "rustLib.a"

[[cargo-projects]]
  path = "native/rust-library"
  mode = "release" # or "debug"
  default-features = true
  features = ["feature1", "feature2"]

Challenges

Since there won't be any information passing between Cabal and the build drivers, some preliminary data should written on file in order to make this all work. Especially, the precise location of the build directory that needs to contain the .a is something to have.

Community feedback

Having taken the time to survey members of the community beforehand, I took their feedback into account and have reported their questions:

These are real questions I've received, if you have more that I haven't answered here, please ask them to me.

Q: Cabal is going to have to learn how to configure rust/zig/D/C++/meson/bazel projects?

A: No. The logic is offloaded to build drivers (like cargo-driver) that get the necessary information from a custom stanza

---

Q: Cabal already knows how to do it, it's called Setup.hs

A: Setup.hs' unrestricted code execution proves to be a liability. The ability here is to provide a clear and official path
to allow the transition from this Setup.hs use-case. One build driver can be analysed for n projects, but n Setup.hs files
must be analysed for n projects.

---

Q: This cargo-driver of yours, who's going to be in charge?

A: A healthy mix of stakeholders and community members, the idea being that some central coordination with the cabal team is needed and we can't have this left to the good will of the community altogether, because we want to present a coherent and reliable "Rust integration" story. The Haskell Foundation can have a stake in it, for instance. But anybody can make a driver, it's not a walled-garden. cargo-driver will be maintained in a more “official” capacity due to its ecosystem importance and significance.

---

Q: This is too heavy/rigid for me, why can't we just let the user figure it out?

A: The goal is to offer a clear and straightforward path to native code integration without having to retain the knowledge of those systems in Cabal. At the same time, more flexible tools are an open door to unregulated code execution.

---

Q: This is the wrong approach, you should put that knowledge in something higher-level like Bazel

A: This doesn't prevent you from using Bazel but forcing people to learn Bazel in order to do what other languages provide for free is asking too much for too few benefits.

---

Q: You're bloating Cabal, external dependencies shouldn't be necessary to use external dependencies

A: We think that the weight of having a build driver as a compile-time dependency to handle other build dependencies is an acceptable compromise.

---

Q: Realistically, what system only knows how to do cabal build? Can't you run a command before cabal build that takes care of everything?

A: Not every system is able to run arbitrary command when compiling a Haskell package. More specifically each time a package X depends on a package Y, cabal will build the dependency with a standard cabal build. Build platforms that support cabal shouldn't have to adapt to something too flexible when it comes to building cabal packages.

[angerman]

extra-bundled-library = "rustLib.a"

This should probably be build-product, output, or something? I see the idea of aligning extra-bundled-libraries from cabal with the extra-bundled-library here. I find it confusing though as we are not bundling anything with the driver?

We could in principle also bundle shared libraries, they don't necessarily have to be static, there's just more work involved to get that right.

A minor nit: rustLibrary and rustLib.a don't line up. We currently have a convention that we have lib<LANG><name>[-<version>].<ext>. Where <LANG> takes values of HS and C right now. name is base, ffi, ... so we end up with libHSbase.a, libCffi.a, ..., should we encourage this convention? libRSmylib.a?

[NorfairKing]

Soft -1 because the proposal does not discuss how to integrate this with things that use cabal like:

• stack
• nixpkgs
• cabal2nix
• Haskell.nix

[gbaz]

@angerman -- note the proposal isn't for the syntax of that toml file, which is just given as an example. it is just for the build-drivers field in cabal files.

@NorfairKing this is an extension to cabal files that Setup.hs will be taught about. Since afaik all those systems make use of Setup.hs, I believe no special integration with them is necessary, and your comment doesn't apply.

[NorfairKing]

I believe no special integration with them is necessary, and your comment doesn't apply.

+1 then :D

[angerman]

@gbaz I understand. The libraries should still match up for readability. I've in fact been discussing this with @Kleidukos, and mentioned my disliking of the initial x- embedding of the Toml file in the cabal file. I'm raising the naming topic here only, to clarify the embedded example. I very much like the fact that this proposal comes with an example usecase to demonstrate how the full thing could look

[Kleidukos]

Hi, thank you for your feedback.

@NorfairKing Yes I expect the integration with other tools to be transparent. :)

@angerman

should we encourage this convention? libRSmylib.a?

It was my understanding that Rust produced C-compatible libraries, so in my opinion we can call the archive "libCmylib.a".

I find it confusing though as we are not bundling anything with the driver?

Yes you are right it is confusing, I wanted to nod at the feature in Cabal but it doesn't have to be called like the Cabal stanza.

We could in principle also bundle shared libraries, they don't necessarily have to be static, there's just more work involved to get that right.

Could you expand a bit about it?

[fgaz]

To clarify, this is basically an isolated preConf hook except that its arguments are passed as files, right?

Why is build-drivers better than / avoids the prioblems of a custom setup?

Setup.hs' unrestricted code execution proves to be a liability

Isn't a build driver the same?

One build driver can be analysed for n projects, but n Setup.hs files must be analysed for n projects.

This is not necessarily true. A custom setup could simply consist of setup-depends: some-driver in the .cabal file + main = defaultMainFromSomeDriver in Setup.hs. How does this proposal differ from this kind of setup?

---

A remark about the hypothetical cargo driver:

Cabal the library is supposed to just take a set of inputs and produce a set of outputs, and as you write should not manage dependencies. I think this should include Cabal's subprocesses. Cargo can download and manage its own dependencies, and this can be a huge pain for distributions, especially since the cargo call would be deep in the process tree and harder to control. For example, in nixpkgs, cargo packages are built by first vendoring all dependencies, hashing them, then telling cargo where to find them. If cargo is called by another tool, there is no way to do the last part without writing a cargo wrapper. It is usually better to let the "topmost" build tool / package manager (nix in my example) orchestrate all the others, and avoid "build tool -> package manager" chains.

So I do at least partially agree with

This is the wrong approach, you should put that knowledge in something higher-level like Bazel

By the way, in "This doesn't prevent you from using Bazel but forcing people to learn Bazel in order to do what other languages provide for free is asking too much for too few benefits." what do you mean by "what other languages provide for free"?

[gbaz]

My understanding is that its a preconf hook in essence. The arguments aren't passed directly, so they can be in files, or they could not exist -- its outside of the scope. The problem with custom setups is they have access to all sorts of stuff and can configure or do anything -- they can reconfigure which modules are in the package or do lots of other stuff that affects later portions of the build. So custom setups require a fully different build path and produce packages whose basic metadata can't be statically known. This just runs some code without being able to affect other phases or feed anything back into cabal -- so its much more restricted.

(btw, note the range of issues that we have with custom setups being just generally flakier and having corner cases compared to build-type: Simple -- https://github.com/haskell/cabal/issues?q=is%3Aissue+is%3Aopen+label%3A%22Cabal%3A+stanza%2Fcustom-setup%22%2C%22Cabal%3A+custom%22+)

also see the motivation in the "What is needed to remove Custom build type" project: https://github.com/haskell/cabal/projects)

[fgaz]

Another advantage of this proposal is that build-drivers is local to a component, so it does not prevent per-component builds, does it?

[Kleidukos]

@fgaz Elixir has something called Rustler: https://hexdocs.pm/rustler/0.18.0/basics.html / https://github.com/rusterlium/rustler

It has enabled to simplify the integration of projects like html5ever: https://hex.pm/packages/html5ever

Another advantage of this proposal is that build-drivers is local to a component, so it does not prevent per-component builds, does it?

Yes it is a per-component basis. :)

[fgaz]

@fgaz Elixir has something called Rustler: https://hexdocs.pm/rustler/0.18.0/basics.html / https://github.com/rusterlium/rustler

It has enabled to simplify the integration of projects like html5ever: https://hex.pm/packages/html5ever

Yes, I'm not saying it isn't possible. Such a driver would work ok when used through cabal-install in an unrestricted system. But Cabal is meant to be used as basis for other package managers as well, where restrictions may actually be enforced.

For example, from the rustler docs:

:cargo - Specify how to envoke the rust compiler. Options are:

:system (default) - use cargo from the system (must be in $PATH)
{:system, } - use cargo from the system with the given channel. Specified as a string, passed directly to cargo (e.g. "+nightly").
{:rustup, } - use rustup to specify which channel to use. Available options include: :stable, :beta, :nightly, or a string which specifies a specific version (e.g. "1.39.0").
{:bin, "/path/to/binary"} - provide a specific path to cargo.

This kind of option (especially the last two) would have to be patched out to integrate a package that uses rustler in most distributions, where the compiler is provided externally.

As an analogy, calling cargo to install (transitive) rust dependencies is like automatically calling apt to install pkgconfig-depends entries.

But this is only relevant to the hypothetical cargo driver and not to this RFC, since wrapping a package manager can be done with current custom setups anyway.

[angerman]

I'm confused, this is not for installing system-library equivalents, right? This is in the case where you build a hybrid haskell/rust application. Similar to how we have hybrid haskell/c applications today.

If this is intended as a general "my application needs foreign library xyz; let's just pull that in", I'm going to be against that; that's what we have package management systems for, apt, rpm, ports, nix, homebrew, fink, ...

If this is to enable hybrid applications in which parts are written in rust and others in haskell, I think this is a sane approach. We can't go the C route, unless we want to turn cabal into a rust solver and builder; that would be unadvisable IMO.

extra-bundled-libraries is explicitly for the case where you want to bundle something with the package that you can't get from the system, but that is built outside of cabal itself, yet should be part of the package cabal puts into the package db.

If such packages break distributions trying to package these up, they ought to figure out how to do this or work around it; arguing against this from a perspective of "but others won't be able to", is artificially limiting. I did mention that the rustup and absolute path in the rustler solution is nothing that should be permitted.

The utility of this driver to me is to use external tools (rustc, cargo, ...) either on PATH or via environment variables, and encapsulate/abstract the build logic for a foreign library away from cabal, while still being able to tie the result back into the cabal package.

Why are custom setups terrible to deal with? Because they are effectively blackboxes doing arbitrary logic. If this is a driver hook, I can decide to not run it and figure out what to do about it or not. If we have a proliferation of custom setups, and we don't want to run custom setups (for various legit reasons), we are worse off. This adds declarative logic, whereas setup.hs is just an arbitrary blackbox. In the former case, I can whitelist build-drivers I'm ok with, and prohibit anything else just from the cabal description.

[Kleidukos]

@fgaz

But this is only relevant to the hypothetical cargo driver and not to this RFC, since wrapping a package manager can be done with current custom setups anyway.

Yes, this is a discussion we can (and should) concern ourselves with at a later time. :)

[Kleidukos]

@angerman

If this is to enable hybrid applications in which parts are written in rust and others in haskell, I think this is a sane approach. We can't go the C route, unless we want to turn cabal into a rust solver and builder; that would be unadvisable IMO.

We agree.

The utility of this driver to me is to use external tools (rustc, cargo, ...) either on PATH or via environment variables, and encapsulate/abstract the build logic for a foreign library away from cabal, while still being able to tie the result back into the cabal package.

Yes, thank you for the way you've put it, I think it's painting an accurate picture of what I had in mind.

Why are custom setups terrible to deal with? Because they are effectively blackboxes doing arbitrary logic.

And they can alter the configuration as well :|

I can whitelist build-drivers I'm ok with, and prohibit anything else just from the cabal description.

Yup'.

---

Generally I think you've outlined many relevant aspects this proposal.

[typedrat]

Enormous 👍🏻 for this.

Even ignoring support for new programming languages, just having the option to use CMake or similar to build the C/C++ parts of a library with non-trivial foreign code included in it would be life-changing for me.

[fgaz]

I thought more about this and I'm now generally ok with the proposal. A distro can override a build driver simply by providing a package with the same name and that's nice and generic.

My only fear is that this could turn into a second --though smaller-- Setup-like black box to be avoided. This is why I want the advantages to be clear.

Also, since this would be such a big change for distros, I'd like to hear the opinion of some Haskell distro maintainer(s), such as @NixOS/haskell

[fgaz]

^ @peti @cdepillabout @expipiplus1 @maralorn @sternenseemann comments/opinions would be much appreciated!

[sternenseemann]

And they can alter the configuration as well :|

If these build drivers are supposed to be equivalent to preConf hooks, it would be perfectly possible for it to alter the cabal file ahead of configuration, wouldn't it? That would ruin the possibility to statically reason again.

In the former case, I can whitelist build-drivers I'm ok with, and prohibit anything else just from the cabal description.

Is this really a lot better? They still can execute arbitrary code, be arbitrarily configurable and keeping a list of good ones will be a problematic, as it's presumably easy to upload new ones to hackage and depend on them.

[Ericson2314]

I think this is too imperative. Just "please run this exe" sounds, frankly, like something that can only be used for evil.

What I would like to see is some way to provide a custom ninja file or other lingua franca description of what yes,

• what commands to run

but also,

• what they produce
• and why they are needed

This is a small step, but I think makes a huge world of difference in that people compelled to think about what is supposed to happen holistically.

---

Really though, even this makes me umcomfortable. I really want more of things like haskell.nix / cabal2nix to be built into their respective package mangers. I am not holding hope for Nix support per-se, though that would be nice; rather I just want cabal to be able to do all the "planning", but none of the "execution"; put differently, I want these tools to act more like CMake where all the building can be done externally.

This hasn't happened with Cabal/Cargo/etc. because they are.....big balls of imperative code and we don't really have a spec for their operational behavior!! This is the elephant in the room!!

I assume based on the example that this feature is desired to make polyglot easier. Well, we can replace Setup.hs, build.rs, etc. etc. with features like this, but then each lang-specific package manager is vying to be the "master" driving the others. No good!

Conversely, if they all can spit out a mere plan we can then take those plans and combine them together, and execute them together. Actually compositional!

To the extent we want to have the ninja extra rules thing I alluded to, it's because the plans are in fact higher order: external libraries, etc. are inputs that might be pre built, or may need to be built. Either way, the generated plan shouldn't care, it's just a parameter.

[Kleidukos]

@Ericson2314 Okay I can get behind the idea of spitting out an execution plan when using out-of-cabal builds, that's extremely reasonable and fair.

I think this is too imperative. Just "please run this exe" sounds, frankly, like something that can only be used for evil.

I understand how it sounds, factually that hasn't been the case in the Elixir ecosystem, but it's not unreasonable.

what they produce

I'd be extremely grateful if you could provide me the desired level of details here.

Let's say the endgame is to compile a rust project located in ./native/rust-bridge, and produce a C-compatible archive (whatever.a) that cabal will incorporate with extra-bundled-libraries, would you like to get a plan of the artifacts that will get generated, or really a statically-known list of what cabal will use?

For example, when bundling libsodium, I have a list of things that I'm checking at build time:

https://github.com/haskell-cryptography/cryptography-libsodium-bindings/blob/main/Setup.hs#L43-L45

but while I know some things in advance (in my Setup.hs script), I also need the directory full of source. Should I aim to list them all, or would a directory path be enough?

[Ericson2314]

I'd be extremely grateful if you could provide me the desired level of details here.

Let me start by saying how it is done with Nix.

• https://github.com/kolloch/crate2nix is a nice too, kinda like haskell.nix, to take the lockfile and generate a build plan with one derivation per crate.

• cabal2nix and haskell.nix both map external C libraries to packages. they have a little known-names remapper, and otherwise just plop in the same name.

extra-bundled-libraries is, I believe, the wrong tool for the job here. That would mean a C library that is built as part of the cabal package. But Cabal packages can only do that in simple cases, and in the case of Rust it comes from a rust package which is not the cabal package.

In Rust and Haskell, there is also a temptation to vendor external C libraries for FFI libraries, and this royally fucks up up distro / Nix users! So please make the libsodium vendoring optional! I will grant that in this case the Haskell and Rust are clearly part of the same "project", not downstream FFI bindings, but projects are not packages. I would argue it's better to think of end projects (like some exe that, maybe, goes in a flatpak) as a mini distro unto itself.

Anyways so back to the question, cabal2nix/haskell.nix will generate an parameter which we can then manually fill in (since there isn't a regular package that matches it per the record-pattern trick for "named parameters") and then everything works beautifully. The Rust is built, then the Haskell is is built, the builds are separate, so each language just needs to understand its own business, etc. etc.

but while I know some things in advance (in my Setup.hs script), I also need the directory full of source. Should I aim to list them all, or would a directory path be enough?

What do you need the source for? pkg-config-depends can be nice compromise where a pkg-config dependency can multiple libraries. Pkg-config, is also the the only thing that exists as a lingua franca for cross-language deps, and I generally thus support giving it more love.

[Ericson2314]

Anyways I would consider wresting with some of the prior musings on these subjects outside of Nix too, to get a few more perspectives:

• https://people.gnome.org/~federico/blog/rust-build-scripts.html Meson Cargo and build.rs
• https://nibblestew.blogspot.com/2019/02/on-possible-futures-of-meson.html language specific packages could kill Meson
• Should build-plans be deleted? rust-lang/cargo#7614 Cargo tried to make a declarative thing, but it didn't really dog food it hard enough (my take) so and so wasn't really the revolution it was meant to be, and instead has been a bit-rotting dead weight.

Meson is basically a reimaginaing of CMake. The idea is no version solving on one hand, but ergonomic / high level concepts and multi-language (CMake also is more recently) on the other hand. It's apples-to-organges vs cabal-install, but comparable and strictly better than pre-cabal-install Cabal, in original era just for Setup.hs.

For my part, I have written NixOS/rfcs#109 because the "lang2nix" things in Nixpkgs are current unpolished and only used by random downstream projects and not in Nixpkgs itself. I hope having much better, polished, and community-concensus based (rather than zillion experiments based) workflows for that will make the benefits of having Cabal/Cargo/etc. spit out complete command execution plans (vs plan.json or lockfile or something less "constructive") more enticing!

[Ericson2314]

CC @dcbaker who worked on Meson much more than me, since then also uses Nix, and might have some good musings to add on challenges for polyglot compositionality.

[dcbaker]

I'll start with this: As a build system person I really appreciate getting rid of build.rs/build.hs/build.d, etc! These really are bad build system design. As someone who has dabbled in Haskell, I'm excited to hear that these are going away!

I'll give my perspective, which is mainly from trying to mix Rust and C/C++ together, without cargo. I maintain most of the Rust support in Meson (and wrote much of it), for use in Mesa and the related free graphics ecosystem (our first public user of this was recently proposed). I've written kind of a lot, and I'm not sure how useful any of it is, lol. But here it is:

So, from a build perspective. The problem of mixing languages like this, is, for correctness and performance, you really need one build system to drive the others. In Meson, we do this with CMake, by running CMake in a "print JSON of what you're going to do" mode, parsing that, and on the fly generating Meson assembly, then interpreting that. I've taken the same approach with cargo in Meson, though the version I currently have posted as a PR has some problems and needs to be rewritten. For cargo-in-meson, this results in (sometimes much) faster builds, as there is one tool managing the build graph (to ensure ordering) and managing the number of spawned jobs (to manage system load), instead of calling cargo as a command inside of Meson. I could keep going on the performance problems with having to have one builder wait on another, if that isn't clear.

In Rust and Haskell, there is also a temptation to vendor external C libraries for FFI libraries, and this royally fucks up up distro / Nix users! So please make the libsodium vendoring optional! I will grant that in this case the Haskell and Rust are clearly part of the same "project", not downstream FFI bindings, but projects are not packages. I would argue it's better to think of end projects (like some exe that, maybe, goes in a flatpak) as a mini distro unto itself.

In Meson we try to provide the tooling for both building your own copy, or using a system copy (because on different OSes there are different expectations, of course!) We call these wraps, and dependency requirements are allowed to "fall back" to a wrap if the normal set of finders (pkg-config, CMake, various -config tools like llvm-config, and some handwritten finders), and build your own copy. This allows us to make both *nix OSes happy (who expect you to use the system copy of zlib, for example) and to build on MacOS and Windows, where you're expected to provide your own copy. Just something to think about.

I (and this is just my opinion) is that the future of building software is going to depend on having a build-plan standard, so that build systems can co-operate with each other seamlessly, there are massive C and C++ code bases that are never going to be rewritten in , meanwhile there are lots of exciting, useful projects being written in , each tightly bundling it's own package manager/build system hybrid together with the language. Meson's wrapping of cargo and CMake in Meson are fragile, and build.rs/build.hs are not good designs (Which I think most people who have dealt with such systems for any amount of time agree, including python who finally provided a mechanism to not use setup tools). Of course, that is hard, and require collaboration across a lot of project spaces.

[Kleidukos]

@Ericson2314 Thank you very much for the feedback. :)

extra-bundled-libraries is, I believe, the wrong tool for the job here. That would mean a C library that is built as part of the cabal package. But Cabal packages can only do that in simple cases, and in the case of Rust it comes from a rust package which is not the cabal package.

I am not sure how you define "simple case", can't --crate-type=staticlib generate the appropriate *.a and *.lib that their docs advertise.

What would be a more complex case?

https://nibblestew.blogspot.com/2019/02/on-possible-futures-of-meson.html language specific packages could kill Meson

I'm not sure what's the takeaway from this blog post, the conclusion seems pretty grim for Meson if Cabal doesn't support Meson-the-language as an alternative syntax. :(

In Rust and Haskell, there is also a temptation to vendor external C libraries for FFI libraries, and this royally fucks up up distro / Nix users! So please make the libsodium vendoring optional!

Okay that can be done (cc @kozross)

but projects are not packages. I would argue it's better to think of end projects (like some exe that, maybe, goes in a flatpak) as a mini distro unto itself.

Okay so static vendoring would be better to activate when building a binary, whereas pkg-config resolution would be better to activate when simply building packages as part of hydra-or-something-like-that?

What do you need the source for? pkg-config-depends can be nice compromise where a pkg-config dependency can multiple libraries. Pkg-config, is also the the only thing that exists as a lingua franca for cross-language deps, and I generally thus support giving it more love.

The piss-poor state of software packaging (and especially system dependencies) is what pushes me to adopt static vendoring for C libraries. Supporting and documenting the fantasies of five-to-six linux distros + macOS & the 3 major BSDs so that people can install a C library before compiling is not something I'm keen to do. Especially since not all distro repos are synchronized, so you have to make your end-user install a library version from official bindings when "stability" is promoted by the distribution.

Unfortunately, making the end-user learn and maintain a nix environment for the sole purpose of providing a couple of C libs is not something I wish to impose on my packages' users (and while I already do that for myself, I don't have the resources to act as Nix support as well as support for my own project).

[Kleidukos]

@dcbaker thank you for this writeup, this is invaluable. :)

I (and this is just my opinion) is that the future of building software is going to depend on having a build-plan standard, so that build systems can co-operate with each other seamlessly,

I think that such a standard could be fantastic, although I have no doubt it will be hard. But it if wasn't, we wouldn't be where we are today. :)

[Ericson2314]

Responding to @dcbaker

you really need one build system to drive the others

Just to be clear, I do agree, even though I nearly said the opposite. What I mean is I don't think any of the language-specific ones should drive the others, because it is unclear who should drive what, and we will just end up with some history-oriented (newer languages drive older) ad-hoc mes.

In Meson we try to provide the tooling for both building your own copy, or using a system copy (because on different OSes there are different expectations, of course!) We call these wraps

Yes wraps is very good! These wraps are subprojects --- Meson like Cabal has something like the *.cabal vs cabal.project distinction. I would be all for a situation where we could similarly, instead of baking vendoring logic in cabal packages, teach cabal-install how to do some sort of shim for any any not-found pkg-config-depends --- even if I never use it because Nix, it will help spread the same mentality of "don't just vendor stuff together because it's convenient, we can be 1-command easy but also compositional".

[Kleidukos]

Just to be clear, I do agree, even though I nearly said the opposite. What I mean is I don't think any of the language-specific ones should drive the others, because it is unclear who should drive what, and we will just end up with some history-oriented (newer languages drive older) ad-hoc mes.

@Ericson2314 I agree that in the context of meta-build systems it can be a problem, but on the other hand I can hardly see the Cabal team asking people to learn meson OR bazel OR nix (and these are not exclusive ORs :/).

I agree in principle with the idea of wraps, though I haven't studied how they work. I would love some guidance for Cabal to be a "good citizen" for meta-build systems.

[dcbaker]

I wasn't super clear on wraps I think, and they're one of the best features that Meson brings to the table, but they look like this:

zlib = dependency('zlib')

and then you have a "wrap file" which is an ini file (the key values might be wrong, (I'm on the top of my head):

[wrap]
source=https://path/to/source/tarball
wrap_patch=https://path/to/meson/build/definitions  ; optional, only required if upstream doesn't use meson
source_sha256=...
wrap_sha256=...

[provides]
zlib = <name of dependency object in meson build>  ; only required to support old versions of meson, newer versions declare this in the build language

Then meson will attempt to find the external dep with pkg-config, etc, and fallback if it can't. We also provide a command line option to force fallback (or error instead of falling back), to allow for finer grained control of that. Which allows distros wanting to package things ensure that they don't use wraps/subprojects, and end users who want things to Just Work™ to get that easily as well

[Ericson2314]

I ultimately do believe in the long term, it is better for language-specific build systems to say "learn Meson OR Bazel OR Nix". But I totally agree the onus is on those to get their shit together first. This is why I am pushing hard for NixOS/rfcs#109, Windows support, etc. These tools need to be really good before language-specific package managers will be comfortable not implementing features and telling their uses to use other stuff. It's a just a fact.

In the short term, for e.g. the Haskell concurrency libraries I think "either build the C library yourself, or put up with evil Setup.hs, you pick your poison" is good enough. Inconvenience + beautiful compositionality vs convenience + big ugliness. Different people will find a tradeoff they prefer, and that buys us time,

I agree in principle with the idea of wraps, though I haven't studied how they work.

So if we did have cabal project, with or without a Setup.hs, that built the thing, we could say this cabal project "provides" this foreign library, and the solver can pick one or claim ambiguity. While still ugly to do the wrapping in *.cabal which isn't really made for it, it at least it gets the stuff out the downstream crypto library itself.

I would love some guidance for Cabal to be a "good citizen" for meta-build systems.

That would be great. I am not sure where the resources would come from to make this happy, but it would be wonderful to to.

On one hand, getting non-Nix Cabal deps to look over the implementation of haskell.nix and just think, "hmm, how could this be easier" might be useful.

On the other hand, I think Meson will be for more attractive to people that have been put of with Nix's ambition to direct everything in excess of it's ability to polish things :). Meson is not "control the world", just like Cabal. and also works on Windows already, just like Cabal.

If we could get a cabal dev to work with @dcbaker or another Meson dev and doing in Meson what was done for Cargo, I think that would also be a great learning experience, make something generally useful, and greatly inform any other efforts.

[phadej]

Playing a devils advocate:

1. ghc --make is somewhat bad for external build drivers. E.g. there is no way to tell GHC to not auto-discover files: even there is a strict list of modules in .cabal description, GHC will find non-declared modules if they are referenced in the source files

• ghc doesn't have a mode to tell the build graph (Extended Dependency Generation ghc-proposals/ghc-proposals#245 is/was stalled as there is no input of what build systems (like Meson?) would need). (Note: even you know all the modules from the declarative description, their interdependencies is something you'd like to infer). This implies that ghc --make is the only really usable option for compiling arbitrary Haskell projects.

2. Another peculiarity of Haskell is that you quite often need to use GHC as the linker. That is obviously not compositional.

OTOH, if these can be fixed, the Cabal itself can be (conceptually) simplified, And then adding build-drivers or a mode for acting as proxy for meta-build-systems could be done.

(With recent Cabal-syntax split out of Cabal, making an alternative build driver for .cabal packages is almost viable. Cabal-syntax is lacking macro file generation for CPP, but that functionality is quite small and easily replicated).

[Ericson2314]

@phadej Thanks for bringing these up.

1. Great point, It does suck, and we need to move Cabal beyond having to use that. Incidentally, modules in C++ have the same issue -- the compiler is needed to uncover deps --- and https://ninja-build.org/manual.html#ref_dyndep was added to Ninja for this role. Meson already uses it too.

So a Meson integration project could actually help us gather good requirements for ghc-proposals/ghc-proposals#245 validated by the experiences of other languages!

Also, I got NixOS/rfcs#92 now proposed for Nix, which is morally the same as the dyndeps thing, so anything good for Meson will also be good for Nix. @ElvishJerricco was doing some experiments where indeed Cabal knowing nothing about this stuff was the sticking point. We can solve these issues!

2. Heheh we know this quite well from Meson. Everything major language impl is happy to link C libraries, but insists on calling the linker itself!

[Ericson2314]

I (and this is just my opinion) is that the future of building software is going to depend on having a build-plan standard, so that build systems can co-operate with each other seamlessly,

I think that such a standard could be fantastic, although I have no doubt it will be hard. But it if wasn't, we wouldn't be where we are today. :)

It's a social problem, and I do believe it has a social solution! We have major Cabal and Meson contributors in this thread, and I gotten the occasional PR into Cargo. Getting everyone in a room together to hammer out a standard might not work, but if we build out some interopt incrementally I think it could.

While the initial building trust and learning each others stuff is risky and uncertain, I do believe it's cheaper in the long run to, as less functionality is deduplicated, and barriers to people dabbling in the less popular languages (like Haskell :)) are removed.

[dcbaker]

Another peculiarity of Haskell is that you quite often need to use GHC as the linker. That is obviously not compositional.

This is actually not pecular at all. Most languages require this at some level, either because an executable must be linked with the language implementing main(), or because of mixed ABIs (we have an extensive heuristic in Meson to try to get this right for C, C++, Objc, Objc++, Fortran, and D) It's actually a hard problem and sometimes requires intermediate convenience libraries. rustc, for example, will happily slurp up object files generated from other languages (with C ABI), and link them, but gcc can't take raw rust objects (though it can take rustc's archives, IIRC).

I also happen to personally know at least one major cargo maintainer, as well as a few CMake people (build systems are a small world).

I'm happy to help build something on the meson side, either as a consumer or a producer (or both).

[fgaz]

Related, maybe duplicate: #4047

[poscat0x04]

Does the new Hooks build type solve the issue that this RFC is trying to solve?
For example if you want to call cargo you can simply do it inside PreConfPackageHook?