Configuration file formats for JSON, INI, YAML and TOML #75584

infinisil · 2019-12-12T23:38:11Z

Motivation for this change

This is a main part of the implementation of NixOS/rfcs#42 (see https://github.com/Infinisil/rfcs/blob/config-option/rfcs/0042-config-option.md#part-1-1).

Since that part of the RFC is fairly uncontroversial, I think we can merge this before the RFC is eventually accepted. This allows people to use such types in their NixOS options already.

Ping @Profpatsch @rycee @aanderse @kampka

Things done

Tested that these types and generator combinations indeed can be used in modules.
Write some tests
Wrote documentation on how to use them..

Here's a simple example module though:

{ lib, config, pkgs, ... }:
let
  format = pkgs.formats.json {};
in {
  options.settings = lib.mkOption {
    type = format.type;
    default = {};
  };

  config = {
    settings.foo = 10;
    settings.bar.baz = "Hello";
    environment.etc.test.source = format.generate "config.json" config.settings;
  };
}

infinisil · 2019-12-12T23:44:22Z

Potentially/Eventually these formats might be implemented using lazyAttrsOf (or what becomes of it) from #70138, such that self-dependent assignments of them will be possible. Edit: This is now done in this PR :)

aanderse · 2019-12-15T02:22:22Z

@infinisil awesome stuff!

lib/settings-formats.nix

infinisil · 2019-12-22T13:31:52Z

(This still needs a bit of docs probably and a history cleanup at least btw, before anybody merges)

infinisil · 2020-01-09T01:32:09Z

I've now rebased this ontop of #70138 to implement types where values can depend on other values. E.g. this previously threw an infinite recursion error, but now it doesn't anymore:

{ lib, config, ... }:
let format = lib.formats.json;
in {
  options.settings = lib.mkOption {
    type = format.type;
    default = {};
  };

  config = {
    settings.foo = true;
    settings.bar = lib.mkIf config.settings.foo "Foo is true!";

    environment.etc.test.text = format.generate config.settings;
  };
}

Due to how the implementation works, it's however a bit more cumbersome if you aren't sure whether an attribute exists at all. This is how that can now be handled gracefully:

{
  # To make sure the `.name` attribute exists
  settings.name = format.empty;
  # To handle the case of `.name` being empty
  settings.greeting = "Hello, ${format.get "nobody" config.settings.name}!";
}

This will generate { greeting = "Hello, nobody!"; }

Alternatively if you don't mind name being set:

{
  settings.name = lib.mkDefault "nobody";
  settings.greeting = "Hello, ${config.settings.name}!";
}

Which will generate { name = "nobody"; greeting = "Hello, nobody!"; }

With JSONValue from: * NixOS/nixpkgs#75584

infinisil · 2020-01-15T14:29:34Z

lib/settings-formats.nix

+
+    empty = { _nixos_empty_json_value =
+      throw ("You tried to access an option value of type json that is empty. "
+        + "Use `lib.formats.json.get` on the value before using it.");


It might be a good idea to only have a single empty value for all

infinisil · 2020-01-15T16:37:21Z

On IRC @adisbladis brought to my attention that to support more formats like TOML, we could use builtins.toJSON and convert that to toml with a build-time tool, meaning the generated strings won't be available to nix at eval time anymore and the generate functions have the signature generate :: Value -> Derivation. This doesn't seem like a problem: Nobody should need the raw Nix strings anyways, and this will speed up evaluation too.

domenkozar · 2020-07-29T12:52:35Z

I'd review it if it was written with terminal length in mind, but currently it's not feasible via github ui.

roberth

It should be noted in the docs that some of these functions may write to the store, which is not safe when Nix is handling secrets. This is in contrast to builtins.toJSON etc, which operate entirely in memory.

lib/generators.nix

roberth · 2020-07-29T13:02:32Z

pkgs/pkgs-lib/default.nix

+# they depend on some packages. This notably is *not* for supporting package
+# building, instead pkgs/build-support is the place for that.
+{ lib, pkgs }: {
+  # setting format types and generators. These would fit in lib/types.nix, but


Suggested change

# setting format types and generators. These would fit in lib/types.nix, but

# setting format types and generators. These do not fit in lib/types.nix, because

Done with https://github.com/NixOS/nixpkgs/compare/76e8e4a26fa451bbc1832e0cd599080795227593..83b16885f526d6ab7b39e98159e2b48024f3238c#diff-fa126a2182114f76a8e8df7ed256395d

roberth · 2020-07-29T13:03:24Z

pkgs/pkgs-lib/default.nix

@@ -0,0 +1,11 @@
+# pkgs-lib is for things that should be in lib, but can't beceause


Suggested change

# pkgs-lib is for things that should be in lib, but can't beceause

# pkgs-lib is for functions that can't be in lib, because

Done with adjustments in https://github.com/NixOS/nixpkgs/compare/76e8e4a26fa451bbc1832e0cd599080795227593..83b16885f526d6ab7b39e98159e2b48024f3238c#diff-fa126a2182114f76a8e8df7ed256395d

roberth · 2020-07-29T13:13:00Z

pkgs/top-level/stage.nix

@@ -75,6 +75,11 @@ let
      inherit (self) runtimeShell;
    };

+  pkgsLib = self: super: import ../pkgs-lib {


An overlay is unnecessary because the same could be achieved by putting formats in all-packages.nix.

Categorizing functions separately from packages is beyond the scope of this change. It can be discussed in a separate issue/pr, although I doubt its usefulness.

Doesn't have to be just functions though, can be arbitrary values, and other functions like fetchgit wouldn't fit in a pkgs-lib. I guess it won't be a problem to just put formats in all-packages.nix, but imo having a separation between "things needed to create packages" and "things that just happen to depend on pkgs" would make sense. Could still introduce that in the future though

Not using an overlay anymore now

All of formats, fetchgit, dockerTools depend on Nixpkgs, take arguments, produce derivations, are maintained and released in the same way. What makes formats so architecturally different that it needs to be treated differently?

imo having a separation between "things needed to create packages" and "things that just happen to depend on pkgs" would make sense. Could still introduce that in the future though

If only one attribute is separate it doesn't make sense though. For it to make sense it must be applied to the whole of Nixpkgs and it must have practical benefits. The prior can be done in the future and I'm skeptical about the benefits, but would love to be proven wrong in that same future.

For now please follow the example of, say, dockerTools.

Looks like my view of the PR was outdated (maybe caused by the current service disruption?) anyway, thanks for removing the overlay. Currently build-support has the role of pkgs-lib so you may want to move the files to build-support/formats. I don't really like that name either so I'm going to leave this change up to you to decide.

The difference between e.g. fetchgit and formats.json is that fetchgit is needed for building many of the pkgs.* derivations, whereas formats.json doesn't have any use for any pkgs.*. Instead formats are mostly just useful for NixOS modules (really, the module system in general, so it wouldn't fit in nixos/lib)

This is also why I don't think build-support would make sense, because formats.json doesn't support any package building.

worldofpeace · 2020-07-29T15:23:26Z

I'd review it if it was written with terminal length in mind, but currently it's not feasible via github ui.

this is useful https://github.com/StylishThemes/GitHub-code-wrap

rycee · 2020-07-29T15:29:49Z

An alternative that doesn't need any installations is to use the "Split" diff option. For whatever reason, it seems to wrap lines while "Unified" doesn't.

infinisil · 2020-07-29T15:39:43Z

It should be noted in the docs that some of these functions may write to the store, which is not safe when Nix is handling secrets. This is in contrast to builtins.toJSON etc, which operate entirely in memory.

For NixOS modules at least it doesn't make much of a difference, because the only way for a string (which toJSON returns) to influence a system build is by putting it in a derivation. With the interface this PR adds, this step is just done a little earlier. But yeah it can't hurt to point this out, I'll add a note

infinisil · 2020-07-31T15:28:32Z

@roberth Looking good now?

Ericson2314 · 2020-08-02T14:59:01Z

I wonder if pkgs-lib should be pkgs/lib to match nixos/lib.

infinisil · 2020-08-02T15:05:32Z

Awesome, thanks for merging!

@Ericson2314 I intentionally didn't name it lib because it could easily be confused with pkgs.lib.

Foremost, the message was discarding double quotes on one side of the diff, which was super-confusing to me, as I thought that the format convertor broke that when in fact only whitespace was changed. I thought I'd cat the files, but then... switching to `diff -u` seemed self-sufficient. It felt sufficiently non-controversial to push directly, but certainly feel free to improve further.

fzakaria · 2020-11-30T20:20:23Z

@infinisil you might want to update this documentation section as well https://nixos.org/manual/nixpkgs/unstable/#sec-generators
It seems out of date considering the new NixOS doc you added.

fzakaria · 2020-11-30T20:22:13Z

Actually reading lib/generators.nix it looks like it should be consolidated with the new formats option introduced here.

infinisil requested review from edolstra, nbp and Profpatsch as code owners December 12, 2019 23:38

infinisil mentioned this pull request Dec 12, 2019

[RFC 0042] NixOS settings options NixOS/rfcs#42

Merged

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild 10.rebuild-linux: 0 This PR does not cause any packages to rebuild labels Dec 12, 2019

kampka mentioned this pull request Dec 13, 2019

lib: Implement settings formats YAML infinisil/nixpkgs#1

Merged

Profpatsch reviewed Dec 20, 2019

View reviewed changes

lib/settings-formats.nix Outdated Show resolved Hide resolved

Profpatsch approved these changes Dec 22, 2019

View reviewed changes

lib/settings-formats.nix Outdated Show resolved Hide resolved

infinisil mentioned this pull request Jan 1, 2020

Change types.attrs to recursively merge #76760

Closed

10 tasks

infinisil force-pushed the settings-formats branch from e5661da to 3974642 Compare January 9, 2020 01:10

ofborg bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: documentation labels Jan 9, 2020

infinisil changed the title ~~Configuration file formats for JSON and INI~~ Configuration file formats for JSON, INI and YAML Jan 9, 2020

ryantm added 2.status: blocked by pr/issue Another PR or issue is preventing this from being completed and removed 2.status: blocked by pr/issue Another PR or issue is preventing this from being completed labels Jan 9, 2020

infinisil force-pushed the settings-formats branch from 3974642 to 3501efa Compare January 10, 2020 15:43

ofborg bot removed 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: documentation labels Jan 10, 2020

samueldr added a commit to samueldr-wip/mobile-nixos-wip that referenced this pull request Jan 13, 2020

initrd: make bootConfig deeply mergeable

6ba9521

With JSONValue from: * NixOS/nixpkgs#75584

samueldr added a commit to samueldr-wip/mobile-nixos-wip that referenced this pull request Jan 13, 2020

initrd: make bootConfig deeply mergeable

b5e6064

With JSONValue from: * NixOS/nixpkgs#75584

samueldr added a commit to samueldr-wip/mobile-nixos-wip that referenced this pull request Jan 13, 2020

initrd: make bootConfig deeply mergeable

5ef4198

With JSONValue from: * NixOS/nixpkgs#75584

samueldr added a commit to samueldr-wip/mobile-nixos-wip that referenced this pull request Jan 14, 2020

initrd: make bootConfig deeply mergeable

8b40feb

With JSONValue from: * NixOS/nixpkgs#75584

Profpatsch self-requested a review January 15, 2020 12:25

infinisil commented Jan 15, 2020

View reviewed changes

roberth requested changes Jul 29, 2020

View reviewed changes

infinisil added 5 commits July 29, 2020 18:06

lib/generators: Extend mkValueStringDefault with float support

9df69cb

pkgs: Add pkgs-lib structure

888c923

pkgs-lib: Implement settings formats for JSON, INI, YAML and TOML

b6c540a

pkgs-lib: Add tests for formats

9c1565a

nixos/docs: Add documentation for settings options

83b1688

infinisil force-pushed the settings-formats branch from 76e8e4a to 83b1688 Compare July 29, 2020 16:23

domenkozar approved these changes Aug 2, 2020

View reviewed changes

roberth approved these changes Aug 2, 2020

View reviewed changes

roberth merged commit 150bf4f into NixOS:master Aug 2, 2020

infinisil deleted the settings-formats branch August 2, 2020 15:11

infinisil mentioned this pull request Aug 3, 2020

mirakurun: init at 3.3.0 #48740

Merged

9 tasks

infinisil mentioned this pull request Aug 22, 2020

nixos/redshift: rework to use config file #50979

Closed

10 tasks

midchildan mentioned this pull request Sep 1, 2020

epgstation: init at 1.7.4 #49413

Merged

9 tasks

infinisil mentioned this pull request Sep 3, 2020

transmission: use freeformType on settings and various fixes #96655

Merged

20 tasks

infinisil mentioned this pull request Oct 2, 2020

Use cases supported by the existing module system and overlays/overrides tweag/nix-ux#20

Open

Sohalt mentioned this pull request Oct 10, 2020

nixos: added navidrome module #92621

Closed

infinisil mentioned this pull request Aug 5, 2021

nixos/matrix-synapse: migrate to generators.toYAML and other cleanup #120260

Closed

10 tasks

infinisil mentioned this pull request Apr 6, 2023

CODEOWNERS: Add @infinisil to pkgs/pkgs-lib #225003

Merged

infinisil added the significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. label Apr 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configuration file formats for JSON, INI, YAML and TOML #75584

Configuration file formats for JSON, INI, YAML and TOML #75584

infinisil commented Dec 12, 2019 •

edited

Loading

infinisil commented Dec 12, 2019 •

edited

Loading

aanderse commented Dec 15, 2019

infinisil commented Dec 22, 2019

infinisil commented Jan 9, 2020

infinisil Jan 15, 2020

infinisil commented Jan 15, 2020

domenkozar commented Jul 29, 2020

roberth left a comment

roberth Jul 29, 2020

infinisil Jul 29, 2020

roberth Jul 29, 2020

infinisil Jul 29, 2020

roberth Jul 29, 2020

infinisil Jul 29, 2020

infinisil Jul 29, 2020

roberth Jul 29, 2020

roberth Jul 29, 2020

infinisil Jul 29, 2020

worldofpeace commented Jul 29, 2020

rycee commented Jul 29, 2020 •

edited

Loading

infinisil commented Jul 29, 2020

infinisil commented Jul 31, 2020

Ericson2314 commented Aug 2, 2020

infinisil commented Aug 2, 2020

fzakaria commented Nov 30, 2020

fzakaria commented Nov 30, 2020

	# setting format types and generators. These would fit in lib/types.nix, but
	# setting format types and generators. These do not fit in lib/types.nix, because

		@@ -0,0 +1,11 @@
		# pkgs-lib is for things that should be in lib, but can't beceause

	# pkgs-lib is for things that should be in lib, but can't beceause
	# pkgs-lib is for functions that can't be in lib, because

Configuration file formats for JSON, INI, YAML and TOML #75584

Configuration file formats for JSON, INI, YAML and TOML #75584

Conversation

infinisil commented Dec 12, 2019 • edited Loading

Motivation for this change

Things done

infinisil commented Dec 12, 2019 • edited Loading

aanderse commented Dec 15, 2019

infinisil commented Dec 22, 2019

infinisil commented Jan 9, 2020

Choose a reason for hiding this comment

infinisil commented Jan 15, 2020

domenkozar commented Jul 29, 2020

roberth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

worldofpeace commented Jul 29, 2020

rycee commented Jul 29, 2020 • edited Loading

infinisil commented Jul 29, 2020

infinisil commented Jul 31, 2020

Ericson2314 commented Aug 2, 2020

infinisil commented Aug 2, 2020

fzakaria commented Nov 30, 2020

fzakaria commented Nov 30, 2020

infinisil commented Dec 12, 2019 •

edited

Loading

infinisil commented Dec 12, 2019 •

edited

Loading

rycee commented Jul 29, 2020 •

edited

Loading