lib.extendMkDerivation, lib.adaptMkDerivation: init #234651

ShamrockLee · 2023-05-28T17:36:12Z

Description of changes

This PR introduces a unified approach to implement build helpers that support fixed-point arguments (lib.extendMkDerivation) and bring such support to existing build helpers (lib.adaptMkDerivation).

The fixed-point arguments support in stdenv.mkDerivation is introduced in #119942, and there are ongoing attempts to make other build helpers support such syntax (buildRustPackage refactor #194475, buildGoModule refactor #225051). The overlay style overrideAttrs brought by the stdenv.mkDerivation change can be used to implement the functionality, which is adopted by the buildRustPackage refactor and the previous version of the buildGoModule refactor. The challenge of such an approach is that the whole set pattern matching the input arguments are degenerated into a single identifier, making it hard to see from the source code which attributes to the build helper accepts.

The new Nixpkgs Library function, lib.extendMkDerivation accepts a base build helper and an attribute overlay (an overlay in the form finalAttrs: args: <attrsToUpdate>), and returns a new build helper by extending the base build helper with the attribute overlay via its <pkg>.overrideAttrs.

The following is the definition of an example build helper, mkLocalDerivation:

lib.extendMkDerivation { } stdenv.mkDerivation (
  finalAttrs:
  {
    preferLocalBuild ? true,
    allowSubstitute ? false,
    ...
  }@args:

  # No need of `// args` here.
  # The whole set of input arguments is passed in advance.
  {
    # Attributes to update
    inherit preferLocalBuild allowSubstitute;
  }
)

For existing build helpers with arguments that cannot be passed to the base build helper, another Nixpkgs library function, lib.adaptMkDerivation, is provided. Instead of an attribute overlay, it takes a function that turns the arguments of the result build helper to those of the base build helper, known as an argument set adapter. This way, developers can decide not to pass some arguments down to the base build helper.

lib.adaptMkDerivation { } stdenv.mkDerivation (
  finalAttrs:
  {
    preferLocalBuild ? true,
    allowSubstitute ? false,
    specialArg ? (_: false),
    ...
  }@args:

  removeAttrs [
    # Don't pass specialArg into mkDerivation
    "specialArg"
  ] args
  // {
    # Arguments to pass
    inherit preferLocalBuild allowSubstitute;
    # Some expressions involving specialArg
    greeting = if specialArg "hi" then "hi" else "hello";
  }
)

The reason to have two functions instead of just lib.adaptMkDerivation, is that we want to encourage (new) build helpers to pass all their input arguments properly down to stdenv.mkDerivation, and provide override functionality through the standardized <pkg>.overrideAttrs instead of custom overriders such as overridePythonAttrs, overrideNimAttrs, etc, by encouraging the use of lib.extendMkDerivation. In the meantime, providing lib.extendMkDerivation-incompatible build helpers with fixed-point arguments support (by lib.adaptMkDerivation) enables a smoother transition to a lib.extendMkDerivation-compatible style.

Fewer expression changes is another plus to convert existing build helpers with lib.adaptMkDerivation to take fixed-point arguments. Working build helper to support recursive attributes. A subsequent PR #271387 enables the fixed-point arguments support in buildPythonPackage and buildPythonApplication with slight modification against mk-python-derivation.nix with no rebuilds.

Both lib.extendMkDerivation and lib.adaptMkDerivation take a set of optional arguments to manipulate their behaviors. For example lib.adaptMkDerivation { modify = drv: toPythonModule drv; } applies toPythonModule to the derivation produced by the derived build helper.

Cc:
Python maintainers: @FRidh @mweinelt @jonringer
Author of the buildRustPackage refactor PR @amesgen
Reviewer of the buildGoModule refactor PR @zowoq
Author of the merged recursive stdenv.mkDerivation PR @roberth
People who mention 119942 in Python application definition: @LunNova

Things done

ShamrockLee · 2023-05-28T18:21:19Z

I should have checked more carefully.

There's a function called makeOverride that deprives the input builder of their recursive attributes support, and buildPythonPackage and buildPackageApplication happens to be makeOverrided. While the change keeps the current python packages unchanged, it still doesn't bring the recursive attributes support to buildPythonModule and buildPythonApplication unless we fix makeOverridable.

Updat: fixed, see below.

lib/customisation.nix

ShamrockLee · 2023-05-28T23:02:37Z

Fix lib.makeOverridable and makeOverridablePythonPackage. Now buildPython* accepts recursive attributes.
Turn a Python application pyspread into the recursive attributes style.

nixos-discourse · 2023-05-29T01:02:47Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/proposal-move-values-from-derivation-attributes-to-function-arguments/20697/21

nixos-discourse · 2023-05-29T01:20:38Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/avoid-rec-expresions-in-nixpkgs/8293/18

roberth

A mix of incremental and fundamental suggestions, because I can't decide for the nixpkgs-python maintainers.

I'm concerned about the increasing complexity, while the goal can be achieved through simplification instead: removing the python-specific level of overriding and turning it into an "overlay" on the mkDerivation arguments instead.

The alternative strategy is to

Provide this python layer, which contains the relevant mkPyton* logic in a way that works with overlay-style overriding. This can be done by reading the existing code, attribute for attribute, and adding the logic to the python layer.
Change the python packages to use that layer in combination with mkDerivation, instead of the current mkPython* functions.
Perhaps make the mkPython functions reuse the overlay so that they don't literally reimplement the same logic. I don't know if this would be worthwhile.
Eventually deprecate the mkPython* functions.

I have played around with step 1 of the alternative strategy. It is feasible, but it requires a bit of migration for each package. I'm not a nixpkgs-python maintainer, so I don't feel like I should be the one to make the decision whether to accept the complexity of this PR, or refactor to actually simplify the python logic by fitting it into a mkDerivation layer.

lib/customisation.nix

roberth · 2023-05-29T12:39:06Z

doc/builders/builders.chapter.md

+}:
+
+lib.extendRecursiveBuilder stdenv.mkDerivation [ ] (finalAttrs:
+


It's worth noting that finalAttrs contains the final arguments to mkDerivation attrs, and not the final arguments to the function you're constructing.

Now the arguments after applying the modifier function is also considered.

IMO, the finalAttrs is meant to be the final state of the attributes passing to the base builder (mkDerivation). That's how user could access finalAttrs.finalPackage and other goodies. If we just want to get the input argument set manually, the user could just lib.fix the function themselves.

Input arguments that doesn't mean to be passed to the base builder are subject to special care, and should never enter the recursion. Those not-to-pass arguments are also the reason why builder overlays are not drop-in replacement of current builders.

I tried to add those special arguments to finalAttrs, but that greatly increase the complexity of the code and makes behavior surprising (the specified result will be different from what is got through finalAttrs by design).

A better way would be to encourage passing all the arguments. When all the arguments are passed, they will all be available inside finalAttrs, and we could then switch to the overlay-based workflow specification (from the current, build-helper-based one).

lib/customisation.nix

lib/trivial.nix

pkgs/development/interpreters/python/mk-python-derivation.nix

roberth · 2023-05-29T15:20:57Z

pkgs/development/interpreters/python/python-packages-base.nix

    in
      if builtins.isAttrs result then result
      else if builtins.isFunction result then {
        overridePythonAttrs = newArgs: makeOverridablePythonPackage f (overrideWith newArgs);
        __functor = self: result;
      }
-      else result;
+      else result);


This feels too custom or repeated (not sure yet which). Does this add mkDerivation-like fixpoint logic at the python attrs level?

I believe we should merge the python attrs level into the mkDerivation attrs, so that the interface and implementation become simpler. Having multiple levels of overriding has a huge complexity cost, so getting rid of an unnecessary level would be a huge win. We'd get rid of overridePythonAttrs and all the user facing complexity, implementation complexity and bugs that come with it.

The python-specific attrs can almost be implemented as an "overlay" on the mkDerivation arguments. When I tried this, I think only like 3 attributes had the same name but a slightly different meaning. That made it a breaking change, but migrating those attributes is feasible and would vastly simplify the python/mkDerivation wiring.

With same names that is generally speaking because the Python builder will extend the lists with some "defaults". That's really the only value of the custom builder over just plain hooks.

This feels too custom or repeated (not sure yet which). Does this add mkDerivation-like fixpoint logic at the python attrs level?

I personally prefer passing most of the attributes into mkDerivation, encourage the use of overrideAttrs and gradually deprecates overridePythonAttrs as well as other builder-specific override methods.

That will be a mass rebuild, so I just work around the makeOverridablePythonPackage obstacle in order to demonstrate the possibility to add the recursive attributes support without rebuild.

infinisil · 2023-05-29T19:23:36Z

Recently the Packages Modules Working Group started investigating whether something like the module system could be used to evaluate packages. We're tracking all work in this repository, meeting weekly, but working mostly asynchronously. It would be great if you could join the Matrix room of the WG and chat with us, or even join the team yourself to work on such issues!

ShamrockLee · 2023-05-29T20:40:54Z

Thank you for taking time reviewing this!

I'm concerned about the increasing complexity, while the goal can be achieved through simplification instead: removing the python-specific level of overriding and turning it into an "overlay" on the mkDerivation arguments instead.

The idea to shift workflow-specific overlays sounds neat, and that could also be friendlier when packaging multi-language projects. Nevertheless, there are some issue on the way to the switch:

Current builders rely more or less on arguments they won't pass into mkDerivation attribute set.
Developers would need to change the way they handle the arguments / attributes, either by passing them directly or through passthru. The impl input argument of the proposed function is essentially an overlay that allows passing not-to-pass-to-mkDerivation arguments through the previous set.
Current attempt to try to implement a builder that supports the (finalAttrs: { }) through an overlay passing to .overrideAttrs, such as rustPlatform.buildRustPackage: support finalAttrs style #194475, discards the set pattern of input arguments ({ a, b ? "some default", ... }@args) completely and degenerate it into previousAttrs.
It is a pity, since the set patterns itself serves as an interface of the builder, providing information about the expected input arguments to both and the Nix interpreter and people reading the Nix expression.
The solution would be rather simple -- replace the previousAttrs with the set pattern, and then the remaining issue will be point 1.

Overall, the goal of the proposed function is to add (finalAttrs: { }) input support with minimum changes to the existing expressions, and raise the discussion about general approaches to bring such supports to builders.

fricklerhandwerk · 2023-05-30T21:39:12Z

doc/builders/builders.chapter.md

@@ -0,0 +1,42 @@
+# The builders {#chap-builders}
+
+A Builder, in the context of Nixpkgs, are functions that produces derivations. (Don't confuse it with the `builder` input argument of function `derivation`, which refers to the path to the executable that builds the derivation.) Such function is usually designed to follow a specific workflow, so that one can produce a package by specifying a few options instead of messing up with the `derivation` function.


Suggested change

A Builder, in the context of Nixpkgs, are functions that produces derivations. (Don't confuse it with the `builder` input argument of function `derivation`, which refers to the path to the executable that builds the derivation.) Such function is usually designed to follow a specific workflow, so that one can produce a package by specifying a few options instead of messing up with the `derivation` function.

A *builder*, in the context of Nixpkgs, is a function that produces derivations.

:::{.warning}

This is not to be confused with the [`builder` argument to `derivation`](https://nixos.org/manual/nix/unstable/language/derivations.html), which refers to the executable that produces the build result.

:::

Such a builder is usually designed to follow a specific workflow, so that one can produce a package by setting a limited set of options relevant to the particular use case instead of using the `derivation` function directly.

This is terrible naming. The least we can do is highlight it with red alarms (not sure about admonition syntax, it's all different everywhere...) Since you're going at great lengths to improve documentation (which is absolutely awesome!) we may as well change that while at it. I'd certainly support it by bouncing around proposals. But of course feel free to ignore requests to widen scope.

Possible candidates:

Build function

Existing use referring to the same thing

androidenv.buildApp doc/languages-frameworks/android.section.md ## Building an Android application {#building-an-android-application}

This build function is particularly useful when it is desired to use
Hydra: the Nix-based continuous integration solution
to build Android apps.

xcodeenv.composeXcodeWrapper doc/languages-frameworks/ios.section.md ## Deploying a proxy component wrapper exposing Xcode {#deploying-a-proxy-component-wrapper-exposing-xcode}

The first use case is deploying a Nix package that provides symlinks to the Xcode
installation on the host system. This package can be used as a build input to
any build function implemented in the Nix expression language that requires
Xcode.

Existing use referring to something else

Inside the build process of pkgs/build-support/rust/build-rust-crate/build-crate.nix

# configure & source common build functions

Build helper

Existing use referring to the same thing

buildIntegration pkgs/tools/networking/dd-agent/integrations-core.nix

# Build helper to build a single datadog integration package.

Existing use referring to something else

Inside the meta.description of the Haskell Hackage package hein pkgs/development/haskell-modules/hackage-packages.nix

An extensible build helper for haskell, in the vein of leiningen

The different use of both candidates doesn't seem to cause confusion. We could just choose from one of them, or come up with something else.

Great research. I like "build helper" more, because its closest to what the thing is about, and also "build function" only contains one word that adds information (almost everything is a function).

When we change section titles we should make sure URLs stay stable. Either keep the anchors or add redirects (we discussed this with @pennae somewhat recently but I can't remember why the script used in the Nix manual was not included).

we discussed this with @pennae somewhat recently but I can't remember why the script used in the Nix manual was not included

there's not technical reason it couldn't be done, it's just that nobody has done the work yet. adding first-class support for fragment redirects to nixos-render-docs would be nicer and less error-prone, but we don't have to do that as a first step.

I disagree that we should rename this from builder to something else and that "build helper" is better. Quite the contrary, the function when called actually builds something, which I'd say is exactly what a builder is. This naming has been around for a very long time.

The Nix attribute builder is a detail that no typical user would ever encounter. Those interested going through say the Pills or in detail through the Nix manual will see it, but otherwise unless you're interested in writing a custom stdenv.mkDerivation you won't do anything with it. Hence, I don't think that part should even be mentioned here.

Nix language functions don't build anything, not even the derivation primitive. What we typically mean by "build" is to run some process to produce files from other files.

Following that line of reasoning "build helper" is also incorrect, because we're solely preprocessing before deriving. So in that case "deriver" or "deriver helper" or "derivation preprocessor" or "build planner" is more correct. Though actually they invoke derivation as well so they're not solely peprocessors either.

Clearly this needs discussion, but I don't want to block this PR on a single term. I'm happy with "build helper" because it reduces ambiguity and is not obviously wrong, and I'm not opposed to further improving the terminology in a separate PR.

Maybe we could move the builder naming to another issue or PR.

BTW, "builders" also refers to "build machines" in the context of "remote builders", which is probably why the documentation of darwin.builder is misplaced under the "Builders" part in the Nixpkgs manual. It would be great to find a more suitable place for it. (#235858)

Did we ever have a user be confused about these being called builders? I at least haven't, and I don't think we should be even spending time discussing this unless there are users confused about it.

doc/builders/builders.chapter.md

fricklerhandwerk · 2023-05-30T21:59:19Z

doc/builders/builders.chapter.md

+}
+```
+
+A list of functions to modify the resulting derivation can be specified after the base builder. Modifications such as overriding and `extendDerivation` can be applied there.


Where do we find information about extendDerivation?

lib.customisation is currently not presented in the Nixpkgs manual. We'll need to add the documentation for those functions before referring to them.

Besides, the Nixpkgs Library Functions documentation automatically generated from the comment don't have anchors. So there's no way to link against them so far.

My fault. The anchor in the form function-library-<long attribute path> is also automatically generated. E. g. function-library-lib.customisation.extendMkDerivation.

This usage is very hard to discover, as the documentation is missing, and the manual provides no hyperlink in the title of each function document.

I addressed this review in the changes to add a new section about using lib.extendMkDerivatien to define build helpers. Please take a look.

doc/builders/builders.chapter.md

LunNova · 2023-05-31T22:43:51Z

Just reviewing your change to input-remapper, I like how much this cleaned up its definition. :)

ShamrockLee · 2024-07-28T21:01:57Z

I rebased the branch to resolve merge conflicts, restructured the test.overriding test cases according to (#330434), and improved the documentation.

Previously, we vendor PR NixOS#234651 because we want to keep the old behavior of filtering out packageRequires from the arguments we pass to the underling stdenv.mkDerivation. Doing so raises the concern about the complexity of PR NixOS#234651. Considering that passing packageRequires to stdenv.mkDerivation also works well, we stop filtering it out and stop vendoring PR NixOS#234651. Now, this PR only uses the existing interface of stdenv.mkDerivation. Even though the name of the build helper is still extendMkDerivation', it is nothing new and has been used in Nixpkgs, such as php.buildComposerProject[1]. [1]: https://github.com/NixOS/nixpkgs/blob/f3834de3782b82bfc666abf664f946d0e7d1f116/pkgs/build-support/php/builders/v1/build-composer-project.nix#L108

Outdated

nyabinary · 2024-11-04T12:56:39Z

Should probably target this for 25.05 (there a merge conflict anyways)

ShamrockLee · 2024-11-05T07:13:07Z

I'll rebase it before weekend.

Add functions to lib.customisation: - extendMkDerivation - adaptMkDerivation Co-authored-by: Robert Hensing <roberth@users.noreply.github.com> Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io> Co-authored-by: Lin Jian <me@linj.tech>

Add "Fixed-point arguments of build helpers" chapter in "Builde helpers" part. Co-authored-by: nicoo <nicoo@mur.at> Co-authored-by: Silvan Mosberger <github@infinisil.com> Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io> Co-authored-by: Lin Jian <me@linj.tech>

philiptaron

I'd like to review and merge these. I've done a first thorough pass through.

philiptaron · 2024-12-02T15:52:36Z

lib/customisation.nix

+    `extendMkDerivation` makes sure that the returned build helper
+    supports such first class recursion like `mkDerivation` does.
+
+    `extendMkDerivation` takes an extra attribute set to confgure its behaviour.


Suggested change

`extendMkDerivation` takes an extra attribute set to confgure its behaviour.

`extendMkDerivation` takes an extra attribute set to configure its behavior.

philiptaron · 2024-12-02T15:53:29Z

lib/customisation.nix

+    supports such first class recursion like `mkDerivation` does.
+
+    `extendMkDerivation` takes an extra attribute set to confgure its behaviour.
+    One can optionally specify *modify* to modify the result derivation,


Suggested change

One can optionally specify *modify* to modify the result derivation,

One can optionally specify `modify` to modify the result derivation,

philiptaron · 2024-12-02T15:53:57Z

lib/customisation.nix

+    or `inheritFunctionArgs` to decide
+    whether to inherit the `__functionArgs` from the base build helper.


Suggested change

or `inheritFunctionArgs` to decide

whether to inherit the `__functionArgs` from the base build helper.

or `inheritFunctionArgs` to decide whether to inherit the `__functionArgs`

from the base build helper.

philiptaron · 2024-12-02T15:55:24Z

lib/customisation.nix

+    :::
+
+    :::{.note}
+    If *modify* is specified,


Suggested change

If *modify* is specified,

If `modify` is specified,

philiptaron · 2024-12-02T16:16:33Z

lib/customisation.nix

+
+    `extendMkDerivation`-specific configurations
+    : `inheritFunctionArgs`: Whether to inherit `__functionArgs` from the base build helper (default to `true`)
+    : `modify`: Function to modify the result derivation (default to `lib.id`)


Bikeshedding: I'd name this transform or derivationTransformFn. map calls this function apply, and so does lib.modules and lib.options, and both of those are fine names with much history as well. Other places call this mapFn or mergeFn.

The only thing in lib that calls this function modify is a helper in lib.debug.traceSeqN.

Not blocking feedback, but I do like derivationTransformFn because it's quite descriptive and shows up in the output.

philiptaron · 2024-12-02T16:47:10Z

doc/build-helpers/fixed-point-arguments.chapter.md

+lib.extendMkDerivation { } stdenv.mkDerivation (
+  finalAttrs:
+  {
+    preferLocalBuild ? true,
+    allowSubstitute ? false,
+    ...
+  }@args:
+
+  # No need of `args //` here.
+  # The whole set of input arguments is passed in advance.
+  {
+    # Attributes to update
+    inherit preferLocalBuild allowSubstitute;
+  }
+)


Is this formatted with nixfmt? Please do so if not.

IIRC, they were formatted with nixfmt-rfc-style months ago. The formatter may have changed a bit. I'll format it again.

philiptaron · 2024-12-02T16:48:17Z

doc/build-helpers/fixed-point-arguments.chapter.md

+
+As a wrapper around [`overrideAttrs`](#sec-pkg-overrideAttrs), `lib.extendMkDerivation` passes the whole set of arguments directly to the base build helper before extending them, making it incompatible with build helpers that only pass part of their input arguments down the base build helper.
+
+To workaround this issue, [`lib.customisation.adaptMkDerivation`](#function-library-lib.customisation.adaptMkDerivation) is introduced. Instead of taking an attribute overlay that returns a subset of attributes to update, it takes an argument set adapter that returns the whole set of arguments to pass to the base build helper, allowing the removal of some arguments. The expression change needed to adopt `lib.adaptMkDerivation` is also smaller, enabling a smooth transition toward fixed-point arguments.


Suggested change

To workaround this issue, [`lib.customisation.adaptMkDerivation`](#function-library-lib.customisation.adaptMkDerivation) is introduced. Instead of taking an attribute overlay that returns a subset of attributes to update, it takes an argument set adapter that returns the whole set of arguments to pass to the base build helper, allowing the removal of some arguments. The expression change needed to adopt `lib.adaptMkDerivation` is also smaller, enabling a smooth transition toward fixed-point arguments.

To work around this issue, [`lib.customisation.adaptMkDerivation`](#function-library-lib.customisation.adaptMkDerivation) is introduced. Instead of taking an attribute overlay that returns a subset of attributes to update, it takes an argument set adapter that returns the whole set of arguments to pass to the base build helper, allowing the removal of some arguments. The change needed to adopt `lib.adaptMkDerivation` is also smaller, enabling a smooth transition to fixed-point arguments.

philiptaron · 2024-12-02T16:49:05Z

doc/build-helpers/fixed-point-arguments.chapter.md

+
+# Example definition of `mkLocalDerivation` extended from `stdenv.mkDerivation` with `lib.adaptMkDerivation`
+
+Should the original definition of build helper `mkLocalDerivation` take an argument `specialArg` that cannot be passed to `sdenv.mkDerivation`,


Suggested change

Should the original definition of build helper `mkLocalDerivation` take an argument `specialArg` that cannot be passed to `sdenv.mkDerivation`,

Let's take as our example a build helper `mkLocalDerivation`, which requires an argument `specialArg` that cannot be passed to `stdenv.mkDerivation`.

philiptaron · 2024-12-02T16:51:24Z

doc/build-helpers/fixed-point-arguments.chapter.md

+)
+```
+
+wrap around the original definition with `lib.adaptMkDerivation` to make the result build helper accept fixed-point arguments.


Suggested change

wrap around the original definition with `lib.adaptMkDerivation` to make the result build helper accept fixed-point arguments.

`lib.adaptMkDerivation` is able to wrap the original definition to make the resulting build helper accept fixed-point arguments.

philiptaron · 2024-12-02T16:52:56Z

doc/build-helpers/fixed-point-arguments.chapter.md

+lib.adaptMkDerivation { } stdenv.mkDerivation (
+  finalAttrs:
+  {
+    preferLocalBuild ? true,
+    allowSubstitute ? false,
+    specialArg ? (_: false),
+    ...
+  }@args:
+
+  removeAttrs [
+    # Don't pass specialArg into mkDerivation.
+    "specialArg"
+  ] args
+  // {
+    # Arguments to pass
+    inherit preferLocalBuild allowSubstitute;
+    # Some expressions involving specialArg
+    greeting = if specialArg "hi" then "hi" else "hello";
+  }
+)


Please format with nixfmt if it isn't already.

ShamrockLee requested review from FRidh, fricklerhandwerk, edolstra and infinisil as code owners May 28, 2023 17:36

github-actions bot added 6.topic: python 8.has: documentation labels May 28, 2023

ShamrockLee force-pushed the extend-builder branch from a2bd999 to 301bd5b Compare May 28, 2023 17:41

Artturin requested a review from roberth May 28, 2023 17:41

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels May 28, 2023

otavio suggested changes May 28, 2023

View reviewed changes

lib/customisation.nix Outdated Show resolved Hide resolved

ShamrockLee force-pushed the extend-builder branch from 301bd5b to 4f7af7a Compare May 28, 2023 23:02

ShamrockLee mentioned this pull request May 28, 2023

buildGoModule: format #234511

Merged

12 tasks

ofborg bot requested a review from LunNova May 29, 2023 01:18

infinisil added the significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. label May 29, 2023

ShamrockLee changed the title ~~lib.extendRecursiveBuilder: init; mkPythonPackage / mkPythonApplication: support recursive attributes~~ lib.extendRecursiveBuilder: init; buildPython*: support recursive attributes May 29, 2023

roberth reviewed May 29, 2023

View reviewed changes

fricklerhandwerk requested changes May 30, 2023

View reviewed changes

ShamrockLee force-pushed the extend-builder branch from 954f48e to 2ee39a7 Compare June 2, 2023 22:43

ShamrockLee changed the title ~~lib.extendRecursiveBuilder: init; buildPython*: support recursive attributes~~ lib.extendMkDerivation: init; buildPython*: support function-based attribute recursion Jun 2, 2023

ShamrockLee force-pushed the extend-builder branch 3 times, most recently from f3c34fb to d868837 Compare June 3, 2023 00:49

jian-lin mentioned this pull request Jul 28, 2024

emacs: teach elisp builders the finalAttrs pattern #330589

Merged

13 tasks

ShamrockLee force-pushed the extend-builder branch 2 times, most recently from 886a96a to b30a617 Compare July 28, 2024 20:59

ShamrockLee mentioned this pull request Jul 29, 2024

Undocumented / impossible: override go modules #86349

Closed

ShamrockLee force-pushed the extend-builder branch from b30a617 to 1f0fa69 Compare July 30, 2024 10:37

jian-lin mentioned this pull request Jul 30, 2024

factor-lang: Restructure package for easier extension #287852

Open

13 tasks

emilazy mentioned this pull request Sep 9, 2024

python312Packages.humann: init at 3.0.0 #340539

Open

13 tasks

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Sep 10, 2024

nbraud mentioned this pull request Sep 17, 2024

[Tracking] Review package examples from manual #253291

Open

9 tasks

emilazy mentioned this pull request Oct 5, 2024

darwin: change the SDK pattern, update the SDKs, and update source releases #346043

Merged

52 tasks

nyabinary added the 2.status: wait for branch‐off Waiting for the next Nixpkgs branch‐off label Nov 4, 2024

ShamrockLee force-pushed the extend-builder branch from 1f0fa69 to 7ed535a Compare November 6, 2024 07:33

ShamrockLee and others added 3 commits November 6, 2024 15:44

lib.extendMkDerivation: add a release note entry

506a1be

ShamrockLee force-pushed the extend-builder branch from 7ed535a to 506a1be Compare November 6, 2024 07:48

ofborg bot added 10.rebuild-darwin: 1-10 and removed 2.status: merge conflict This PR has merge conflicts with the target branch 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin labels Nov 6, 2024

emilazy mentioned this pull request Nov 18, 2024

Nixpkgs committers list #321665

Open

This was referenced Nov 28, 2024

buildGoModule: Use the env attribute to pass environment variables #359641

Open

buildGoModule: no longer filter out vendorSha256 #359798

Merged

philiptaron requested changes Dec 2, 2024

View reviewed changes

		}:

		lib.extendRecursiveBuilder stdenv.mkDerivation [ ] (finalAttrs:

		@@ -0,0 +1,42 @@
		# The builders {#chap-builders}

		A Builder, in the context of Nixpkgs, are functions that produces derivations. (Don't confuse it with the `builder` input argument of function `derivation`, which refers to the path to the executable that builds the derivation.) Such function is usually designed to follow a specific workflow, so that one can produce a package by specifying a few options instead of messing up with the `derivation` function.

-A Builder, in the context of Nixpkgs, are functions that produces derivations. (Don't confuse it with the `builder` input argument of function `derivation`, which refers to the path to the executable that builds the derivation.) Such function is usually designed to follow a specific workflow, so that one can produce a package by specifying a few options instead of messing up with the `derivation` function.
+A *builder*, in the context of Nixpkgs, is a function that produces derivations.
+:::{.warning}
+This is not to be confused with the [`builder` argument to `derivation`](https://nixos.org/manual/nix/unstable/language/derivations.html), which refers to the executable that produces the build result.
+:::
+Such a builder is usually designed to follow a specific workflow, so that one can produce a package by setting a limited set of options relevant to the particular use case instead of using the `derivation` function directly.

	`extendMkDerivation` takes an extra attribute set to confgure its behaviour.
	`extendMkDerivation` takes an extra attribute set to configure its behavior.

	One can optionally specify modify to modify the result derivation,
	One can optionally specify `modify` to modify the result derivation,

		or `inheritFunctionArgs` to decide
		whether to inherit the `__functionArgs` from the base build helper.


		As a wrapper around [`overrideAttrs`](#sec-pkg-overrideAttrs), `lib.extendMkDerivation` passes the whole set of arguments directly to the base build helper before extending them, making it incompatible with build helpers that only pass part of their input arguments down the base build helper.

		To workaround this issue, [`lib.customisation.adaptMkDerivation`](#function-library-lib.customisation.adaptMkDerivation) is introduced. Instead of taking an attribute overlay that returns a subset of attributes to update, it takes an argument set adapter that returns the whole set of arguments to pass to the base build helper, allowing the removal of some arguments. The expression change needed to adopt `lib.adaptMkDerivation` is also smaller, enabling a smooth transition toward fixed-point arguments.


		# Example definition of `mkLocalDerivation` extended from `stdenv.mkDerivation` with `lib.adaptMkDerivation`

		Should the original definition of build helper `mkLocalDerivation` take an argument `specialArg` that cannot be passed to `sdenv.mkDerivation`,

	Should the original definition of build helper `mkLocalDerivation` take an argument `specialArg` that cannot be passed to `sdenv.mkDerivation`,
	Let's take as our example a build helper `mkLocalDerivation`, which requires an argument `specialArg` that cannot be passed to `stdenv.mkDerivation`.

	wrap around the original definition with `lib.adaptMkDerivation` to make the result build helper accept fixed-point arguments.
	`lib.adaptMkDerivation` is able to wrap the original definition to make the resulting build helper accept fixed-point arguments.

lib.extendMkDerivation, lib.adaptMkDerivation: init #234651

Are you sure you want to change the base?

lib.extendMkDerivation, lib.adaptMkDerivation: init #234651

Conversation

ShamrockLee commented May 28, 2023 • edited Loading

Description of changes

Things done

ShamrockLee commented May 28, 2023 • edited Loading

ShamrockLee commented May 28, 2023 • edited Loading

nixos-discourse commented May 29, 2023

nixos-discourse commented May 29, 2023

roberth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShamrockLee May 29, 2023 • edited Loading

Choose a reason for hiding this comment

infinisil commented May 29, 2023 • edited Loading

ShamrockLee commented May 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FRidh Jun 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShamrockLee Jun 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShamrockLee Jan 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LunNova commented May 31, 2023

ShamrockLee commented Jul 28, 2024

nyabinary commented Nov 4, 2024

ShamrockLee commented Nov 5, 2024

philiptaron left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShamrockLee Dec 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShamrockLee commented May 28, 2023 •

edited

Loading

ShamrockLee commented May 28, 2023 •

edited

Loading

ShamrockLee commented May 28, 2023 •

edited

Loading

ShamrockLee May 29, 2023 •

edited

Loading

infinisil commented May 29, 2023 •

edited

Loading

ShamrockLee commented May 29, 2023 •

edited

Loading

FRidh Jun 14, 2023 •

edited

Loading

ShamrockLee Jun 14, 2023 •

edited

Loading

ShamrockLee Jan 3, 2024 •

edited

Loading

ShamrockLee Dec 3, 2024 •

edited

Loading