[RFC 0014] Improve import from derivation #14

Ericson2314 · 2017-06-03T19:53:28Z

New description

Build on #40 to allow pausing and resuming evaluation with derivations, such that we always have derivations for are end goals. This allows hydra and various nix-env queries to work with their current architecture. Therefore, if this RFC is implemented, hydra.nixos.org should allow importing from derivations.

Original description

Fix some issues with importing from derivations so it is nicer to use both manually and with CI. If this RFC is implemented, hydra.nixos.org should allow importing from derivations.

Rendered

shlevy

🙌

Happy to help out in any way with this!

zimbatm · 2017-06-06T09:46:34Z

@Ericson2314 did you get my review by email? Let me know what I can do to help.

zimbatm · 2017-06-09T10:14:14Z

rfcs/0000-import-from-derivation.md

+There's currently a bug with `nix-repl` where if computation is interrupted (say with `SIGINT`), and the same computation is begun again, one will get an infinite recursion error despite there being no infinite recursion.
+This is because the thunks stay black-holed—the indication that they are currently evaluated—despite the computation being aborted.
+To fix this, we want aborted computations to instead unwind their stack, un-black-holing any thunks back the way they were before.
+Mathematically, black-hole unwinding ensures that evaluation is idempotent.


Just a thought: this seems useful on it's own. Maybe split it in it's own RFC, I suspect smaller RFCs are more likely to make progress. Or just implement it?

Seeing that that's just a bug fix, I might just open an issue. That issue can be linked from any tracking the RFC as a whole if it is accepted.

Opened NixOS/nix#1407

zimbatm · 2017-06-30T16:24:58Z

rfcs/0000-import-from-derivation.md

+# Drawbacks
+[drawbacks]: #drawbacks
+
+Overlap with recursive Nix.


Do you have a link to what recursive Nix is? Just so the reader can compare.

0xABAB · 2017-06-30T22:04:53Z

The technical writing could be a lot better, but I get the gist of it.

It appears that this is just an instance of "recursive make considered harmful", though. My longterm concern is that this will lead to a new workflow where people will start making use of the feature, which means that the system as a whole becomes dependent on more dynamic features while they could also be implemented with more static features (as it is now).

In short, I would say this is a controversial feature. It's not like I can stop you, but I probably wouldn't add this feature if I had complete control over the project.

Ericson2314 · 2017-07-01T15:54:51Z

Yes, dynamism should be the escape the escape hatch of last resort. I hope the burden of writing concrete syntax with an external program is enough to dissuade anyone from using it more.

Ericson2314 · 2017-07-01T17:28:45Z

I should edit this RFC to also mention readFile. ReadFile of a derivation needs analogous improvements.

domenkozar · 2017-07-19T14:52:09Z

So if I understand correctly, to summarize:

Hydra would evaluate an expression until it hits IFD and halt there
it would use build slaves to build the IFD paths
start with 1) using the new built paths to continue the eval

Ericson2314 · 2017-07-20T17:36:09Z

@domenkozar do you mean " start 1) again..."? If so yes. It could also just let nix-instantiate do that automatically, but avoid keeping the eval process around it's probably best to do that by hand with --dry-run. [That remainds me, I should make clear that with these changes --dry-run is now meaningful for nix-instantiation too, whereas today we are stuck without it.]

edolstra · 2019-01-17T11:33:47Z

@Ericson2314 We're doing triage on the RFC backlog, are you still interested in pursuing this RFC?

The issue I see with this RFC is that it doesn't really work given Hydra's current design, specifically, the strict split between the evaluator and the queue runner. The evaluator is supposed to add rows to the Builds table, so it can't delay building IFDs - it has to produce a top-level derivation. Likewise the queue runner doesn't have access to the jobset inputs, so it can't resume evaluation.

This could be solved by having an integrated evaluator/queue runner, but then we don't need any partial evaluation tricks: we can just build IFDs as normal build steps during evaluation. (The only issue is that during evaluation there is no Build yet to account the BuildStep to, but that can be hacked around by having a temporary Build or whatever.)

The downside of integrating the evaluator and queue runner is that it reduces security a bit, since (for example) the jobset Nix expression could try to read SSH credentials of the queue runner during evaluation using readFile.

Ericson2314 · 2019-01-17T17:00:59Z

I'm going to close for the time being. I need to fix a few details, and then I have a new idea using derivations that build derivation trees to do this better still.

Ericson2314 · 2019-01-17T21:39:16Z

@edolstra Also, thank you for that description of the hydra status quo. I'll try to better separate the Nix itself and hydra concerns from this PR, but on the hydra side I'm thinking that top-level derivations must not be auto-generated, but their dependencies may be. My new idea already supported that on the Nix side, and from your description, that would be allow Hydra to keep its existing division of labor. [If we had to refactor hydra that much, I'd declare it "totalled", i.e. it would be better to just use a rewrite that would also fix a slew of other issues.]

edolstra · 2019-01-18T08:32:58Z

In Nix, you can't have a .drv file without knowing all its dependencies. This is independent from any evaluation issues.

Ericson2314 · 2019-01-18T20:07:04Z

@edolstra yes this would be a recursive Nix thing, where drv1 would build drv2 (and drv2's transitive dependencies' drvs). drv1 would be delayed evaluation of nix expressions, which we may or may not wish to special-case purely. That special case would purely be an optimization.

) * unindent inherit semicolon, reshuffle binding/inherit sections * fixup! Stuff

@piegamesde

* 166: Nix formatting Create 0101-nix-formatting.md WIP Go through a large part and agree on it Co-Authored-By: @piegamesde Update 0101-nix-formatting.md Rework a lot of things Update 0101-nix-formatting.md Move around some sections Reword the detailed section Minor updates Slight header changes again Updates Update 0101-nix-formatting.md Update after today's meeting Update 0101-nix-formatting.md Further updates in the meeting Update 0101-nix-formatting.md After todays meeting Update after meeting Rename to probably the right number Only use anchor links Improvements and additions - The sub-expression rule is now reworded and its own section with examples and rationale - Line length limit is now specified as we agreed-upon in the meeting - The operator section is rewritten to align more with the consensus Redo and explain operator special case Also remove the special case for non-chainable operators, barely any benefit in Nixpkgs * Operator chains outside bindings can also have a compact form * Make the operator compact form specific to binders * Fix accidentally formatted semicolons in alternatives * Minor changes * Light copy editing * Fix .git-blame-ignore-revs * Improve assert/with wording * Be more flexible with single-line element count * binder -> binding * unindent inherit semicolon, reshuffle binding/inherit sections (#14) * unindent inherit semicolon, reshuffle binding/inherit sections * fixup! Stuff * Give alternatives to `in` formatting * Expand on line break preservation * Add editorconfig * Expand argumentation against leading commas * Add @dasJ to the formatter team * Add shepherd team Co-authored-by: Linus Heckemann <git@sphalerite.org> * Various improvements (#15) * Various improvements - Remove unnecessary **Description** headers - Rename **Rationale and Alternatives** to just **Alternatives** - Insert must/may/should more diligently - Add some TODOs where things are unclear - Remove numberings from examples when not needed - Minor clarity improvements and simplifications throughout * Apply suggestions from code review Co-authored-by: Ryan Hendrickson <ryan.hendrickson@alum.mit.edu> --------- Co-authored-by: Ryan Hendrickson <ryan.hendrickson@alum.mit.edu> * Address TODOs and rework with/assert * Minor adjustments * Mention formatting Nix code in documentation * Working towards finalization (#16) - Defined absorption and absorbable terms - Adapted the existing RFC text to make use of these definitions, resulting in simplications of the text in many cases. - Updated `with` section to match the implementation - Updated the function declaration section to match the implementation - Sometimes, the function body may get absorbed - This used to be a special case scoped to bindings only, so it got removed there - Updated the operators section to match the implementation - Specify the format of non-chainable operators (somehow those got lost in the past) - Reworked bindings section. It should now be clear and specific enough. - Minor wording fixes * String section * Specify assert conditions * More absorption for multi-line arguments * How to update the standard format * Fix minor typos * Less lines for common function call patterns * Specify comments * Specify that the formatter should be as pure as possible With some exceptions * nit: fix list concatenation example (#17) * Update rfcs/0166-nix-formatting.md Co-authored-by: Doron Behar <doron.behar@gmail.com> * Add good indentation examples (#18) * Add another chainable operators example * justify difference in semicolon placement * Allow different parenthesized argument style * Clarify non-vertical alignment rule * Improved clarity of bindings rule * Improve bindings semicolon alternatives section --------- Co-authored-by: Silvan Mosberger <github@infinisil.com> Co-authored-by: Silvan Mosberger <silvan.mosberger@tweag.io> Co-authored-by: Ryan Hendrickson <ryan.hendrickson@alum.mit.edu> Co-authored-by: Yuriy Taraday <yorik.sar@gmail.com> Co-authored-by: Linus Heckemann <git@sphalerite.org> Co-authored-by: Janne Heß <janne@hess.ooo> Co-authored-by: Doron Behar <doron.behar@gmail.com>

Ericson2314 and others added 10 commits May 24, 2017 17:21

import-from-derivation: Copy template

e837055

import-from-derivation: WIP draft

aa324c3

import-from-derivation: Add @expipiplus1 as coauthor!

d2b9b55

import-from-derivation: Add note about hydra.nixos.org to summary

0c17f04

import-from-derivation: Mention that layering is preserved

8736c3f

import-from-derivation: Add mention of the ultimate goal of less churn

7311ed9

import-from-derivation: Tweak wording

43caba7

import-from-derivation: Fix misc typos, wording, formatting

16517a2

import-from-derivation: Fix spelling of "shlevy", sorry!

ec99095

import-from-derivation: Fill in missing issue number

a8a5007

shlevy approved these changes Jun 3, 2017

View reviewed changes

Ericson2314 mentioned this pull request Jun 3, 2017

[RFC 0009] Nix rapid release #9

Closed

zimbatm reviewed Jun 9, 2017

View reviewed changes

Ericson2314 mentioned this pull request Jun 12, 2017

Infinite recursion false-positives in nix-repl NixOS/nix#1407

Open

zimbatm reviewed Jun 30, 2017

View reviewed changes

shlevy mentioned this pull request Sep 5, 2017

[Discussion] On updating Leksah — and packages that define nontrivial default.nix in their repo NixOS/nixpkgs#28931

Closed

Ericson2314 closed this Jan 17, 2019

piegamesde pushed a commit to piegamesde/rfcs-1 that referenced this pull request Jan 31, 2024

unindent inherit semicolon, reshuffle binding/inherit sections (NixOS#14

299c16f

) * unindent inherit semicolon, reshuffle binding/inherit sections * fixup! Stuff

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC 0014] Improve import from derivation #14

[RFC 0014] Improve import from derivation #14

Ericson2314 commented Jun 3, 2017 •

edited

Loading

shlevy left a comment

zimbatm commented Jun 6, 2017

zimbatm Jun 9, 2017

Ericson2314 Jun 9, 2017

Ericson2314 Jun 12, 2017

zimbatm Jun 30, 2017

0xABAB commented Jun 30, 2017

Ericson2314 commented Jul 1, 2017

Ericson2314 commented Jul 1, 2017

domenkozar commented Jul 19, 2017 •

edited

Loading

Ericson2314 commented Jul 20, 2017 •

edited

Loading

edolstra commented Jan 17, 2019

Ericson2314 commented Jan 17, 2019

Ericson2314 commented Jan 17, 2019

edolstra commented Jan 18, 2019

Ericson2314 commented Jan 18, 2019 •

edited

Loading

[RFC 0014] Improve import from derivation #14

[RFC 0014] Improve import from derivation #14

Conversation

Ericson2314 commented Jun 3, 2017 • edited Loading

New description

Original description

shlevy left a comment

Choose a reason for hiding this comment

zimbatm commented Jun 6, 2017

zimbatm Jun 9, 2017

Choose a reason for hiding this comment

Ericson2314 Jun 9, 2017

Choose a reason for hiding this comment

Ericson2314 Jun 12, 2017

Choose a reason for hiding this comment

zimbatm Jun 30, 2017

Choose a reason for hiding this comment

0xABAB commented Jun 30, 2017

Ericson2314 commented Jul 1, 2017

Ericson2314 commented Jul 1, 2017

domenkozar commented Jul 19, 2017 • edited Loading

Ericson2314 commented Jul 20, 2017 • edited Loading

edolstra commented Jan 17, 2019

Ericson2314 commented Jan 17, 2019

Ericson2314 commented Jan 17, 2019

edolstra commented Jan 18, 2019

Ericson2314 commented Jan 18, 2019 • edited Loading

Ericson2314 commented Jun 3, 2017 •

edited

Loading

domenkozar commented Jul 19, 2017 •

edited

Loading

Ericson2314 commented Jul 20, 2017 •

edited

Loading

Ericson2314 commented Jan 18, 2019 •

edited

Loading