New API Porting Guide

[Pacman99]

Hey everyone! We just did a lot of refactoring in the core branch and if you are updating to the latest changes I would recommend you read this guide on how to update.

This is a pretty quick explanation, to get more detail take a look at the doc folder which carefully goes over all the new API arguments - I also linked relevant doc sections to the headers.

The main things to notice are the changes to extern, suites and overrides.

Suites

Because of how important suites are to the configuration we decided it would be good to just define them in the flake.nix itself. You will notice a suites argument under the nixos section. You should copy all the content from your suites/default.nix to that argument.

Extern

The cool part about extern is that you can see everything a devos system depends on in one file. We believe that this information should actually be in the flake.nix too. So you can now just use different API arguments to pass external module and overlays.

Overlays

You can just put overlays in the overlays argument for the nixos channel. Where necessary, all external overlays will be automatically filtered out by inspecting your inputs.

Modules

There is a dedicated externalModules argument for this purpose.

specialArgs

We currently no longer allow passing specialArgs to host - its possible but quite difficult. The reasoning behind this was that you should add modules in the flake.nix directly instead of passing them to hosts and adding them. You can add modules to different hosts by adding lines to the hosts argument.

Overrides

Module overrides are no longer directly demonstrated by default, because of how finicky they are. It is possible by making use of the disabledModules key and the latestModulesPath special argument.

Package overrides, on the other hand, can be done in the overlays/overrides.nix file. There are plenty of examples there.

---

Generally as your merging, make sure to drop extern, overrides, and suites and move the code from there either to your flake.nix or the relevant files.
If you would like to split those things up into separate files, you can always just import files into the api arguments. Bonus: to de-clutter, you can leave out any import statements, where you would otherwise put them. Most items are piped through a maybeImport.

If you have any issues or comments about the API please comment here or make a dedicated issue. We would love to hear feedback!

[benneti]

Hey when pulling in the latest core into my config I get:

• error: gnome3.corePackages is removed since 2019-08-25: please use services.gnome3.core-shell.enable.
• I have desktopManager.gnome3.enable = true;, but rg corePackages gives me no results.

As the only change from my last working config was trying to switch to the new layout (twice the first try I did not see there was this guide) I guess there might be a name clash or something similar from this.
(it could be that this is due to another commit, too as my last merge was three weeks ago 82b73cf .)

EDIT: also happens for the direnv where I think no gnome modules and or packages should be used. Additionally show-trace only traces it back to the /nix/store/2y8xq739cy0klq7mh92fxziahda70ryd-source/pkgs/desktops/gnome/default.nix:332:3 in nixpkgs

[Pacman99]

error: gnome3.corePackages is removed since 2019-08-25: please use services.gnome3.core-shell.enable.

This is very interesting. At first look I would think it was because of using an option and then updating nixpkgs. But considering you never set corePackages and the option is from over a year ago, that doesn't really make sense.

Can you post the full stack trace output of the configuration building, that should help with debugging. Also if possible, seeing your repository's code could be helpful - if its private then just the stack trace should be okay.

[benneti]

thank you for taking the time, here is the link to the broken config https://cloud.tissot.de/gitea/benneti/nixos/src/branch/broken and here is the traceback:

error: gnome.corePackages is removed since 2019-08-25: please use `services.gnome.core-shell.enable`

       … while evaluating the attribute 'corePackages'

       at /nix/store/2y8xq739cy0klq7mh92fxziahda70ryd-source/pkgs/desktops/gnome/default.nix:332:3:

          331|   # added 2019-08-25
          332|   corePackages = throw "gnome.corePackages is removed since 2019-08-25: please use `services.gnome.core-shell.enable`";
             |   ^
          333|   optionalPackages = throw "gnome.optionalPackages is removed since 2019-08-25: please use `services.gnome.core-utilities.enable`";

       … while evaluating 'op'

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:3:19:

            2| let
            3|   op = sum: path: val:
             |                   ^
            4|     let

       … from call site

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:30:18:

           29|     builtins.foldl'
           30|       (sum: key: op sum (path ++ [ key ]) val.${key})
             |                  ^
           31|       sum

       … while evaluating anonymous lambda

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:30:13:

           29|     builtins.foldl'
           30|       (sum: key: op sum (path ++ [ key ]) val.${key})
             |             ^
           31|       sum

       … from call site

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:29:5:

           28|   recurse = sum: path: val:
           29|     builtins.foldl'
             |     ^
           30|       (sum: key: op sum (path ++ [ key ]) val.${key})

       … while evaluating 'recurse'

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:28:24:

           27|
           28|   recurse = sum: path: val:
             |                        ^
           29|     builtins.foldl'

       … from call site

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:21:8:

           20|     # recurse into that attribute set
           21|       (recurse sum path val)
             |        ^
           22|     else

       … while evaluating 'op'

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:3:19:

            2| let
            3|   op = sum: path: val:
             |                   ^
            4|     let

       … from call site

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:30:18:

           29|     builtins.foldl'
           30|       (sum: key: op sum (path ++ [ key ]) val.${key})
             |                  ^
           31|       sum

       … while evaluating anonymous lambda

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:30:13:

           29|     builtins.foldl'
           30|       (sum: key: op sum (path ++ [ key ]) val.${key})
             |             ^
           31|       sum

       … from call site

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:29:5:

           28|   recurse = sum: path: val:
           29|     builtins.foldl'
             |     ^
           30|       (sum: key: op sum (path ++ [ key ]) val.${key})

       … while evaluating 'recurse'

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:28:24:

           27|
           28|   recurse = sum: path: val:
             |                        ^
           29|     builtins.foldl'

       … from call site

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:35:1:

           34| in
           35| recurse { } [ ] tree
             | ^

       … while evaluating anonymous lambda

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/flattenTree.nix:1:1:

            1| tree:
             | ^
            2| let

       … from call site

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/default.nix:101:23:

          100|   #   }
          101|   flattenTree = tree: import ./flattenTree.nix tree;
             |                       ^
          102|

       … while evaluating 'flattenTree'

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/default.nix:101:17:

          100|   #   }
          101|   flattenTree = tree: import ./flattenTree.nix tree;
             |                 ^
          102|

       … from call site

       at /nix/store/a5kj99c62gd9ybk0rx61vj0gk227i9l0-source/packagesFromOverlaysBuilderConstructor.nix:74:32:

           73|         # flatten nested sets with "/" delimiter then drop disallowed packages
           74|         filterPackages system (flattenTree exportPackages);
             |                                ^
           75|

       … while evaluating 'filterAttrs'

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/filterPackages.nix:7:23:

            6|   nameValuePair = name: value: { inherit name value; };
            7|   filterAttrs = pred: set:
             |                       ^
            8|     listToAttrs (

       … from call site

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/filterPackages.nix:30:1:

           29| in
           30| filterAttrs sieve packages
             | ^

       … while evaluating anonymous lambda

       at /nix/store/2d4rhqdx6jcm8i747aar73rr80dqg9xi-source/filterPackages.nix:2:9:

            1| { allSystems }:
            2| system: packages:
             |         ^
            3| let

       … from call site

       at /nix/store/a5kj99c62gd9ybk0rx61vj0gk227i9l0-source/packagesFromOverlaysBuilderConstructor.nix:74:9:

           73|         # flatten nested sets with "/" delimiter then drop disallowed packages
           74|         filterPackages system (flattenTree exportPackages);
             |         ^
           75|

       … while evaluating 'packagesFromOverlaysBuilder'

       at /nix/store/a5kj99c62gd9ybk0rx61vj0gk227i9l0-source/packagesFromOverlaysBuilderConstructor.nix:8:37:

            7|
            8|       packagesFromOverlaysBuilder = channels:
             |                                     ^
            9|         /**

       … from call site

       at /nix/store/a5kj99c62gd9ybk0rx61vj0gk227i9l0-source/systemFlake.nix:195:29:

          194|             (optionalAttrs (args ? ${builder})
          195|               { ${output} = args.${builder} pkgs; });
             |                             ^
          196|       in

       … while evaluating anonymous lambda

       at /nix/store/a5kj99c62gd9ybk0rx61vj0gk227i9l0-source/flake.nix:38:20:

           37|           lhs // mapAttrs
           38|             (name: value:
             |                    ^
           39|               if isAttrs value then lhs.${name} or { } // value

       … from call site

       … while evaluating the attribute 'packages'
sudo: /tmp/nixos-rebuild.n1fALU/result/bin/switch-to-configuration: command not found
warning: error(s) occurred while switching to the new configuration
/etc/nixos

[Pacman99]

Thank you for the detailed report! I think I know exactly what is going on now. I opened an issue to explain the problem in digga: #13.

Although this might actually a problem that is being triggered by a function in flake-utils-plus. So we likely have to go there to fix it.

As a workaround you can add the __dontExport property to any overlay where you override gnome. And that should prevent that error from being triggered.

[benneti]

thank you very much, overlays were basically the only thing I did not consider (as I forgot I had one for nautilus and thereby gnome admittedly)!
I think some of these new paradigms are a bit weird (at least for me) as they are counterintuitive to the lazy evaluation of nix.

[Pacman99]

I think some of these new paradigms are a bit weird (at least for me) as they are counterintuitive to the lazy evaluation of nix.

Not sure if your talking about overlays or devos. One of the goals for devos is to try and intuitively export as many of your creations as possible. We tried to create a sharing model for overlays and packages for anyone that uses digga. So its a bit counter-intuitive in terms of lazy evaluation, but fits well with flakes and the sharing goal.

[benneti]

Yes thats what I meant.

[benneti]

I think there is another issue due to exporting everything, it seems like packages in overlays that are currently broken are tried to be built when entering the direnv currently.
An example for this is emacsPgtkGcc from the emacs-overlay which is built when entering my nixos config folder but is currently broken, making it unnecessary hard to enter the direnv.

[benneti]

Sorry for kinda spamming this thread, but I think there is another small problem.
I think overlays, that change custom pkgs are not applied anymore.
I guess this is not a hard problem but sometimes it is cleaner to have a package close to upstream and then apply some changes in an overlay.

[Pacman99]

I think there is another issue due to exporting everything, it seems like packages in overlays that are currently broken are tried to be built when entering the direnv currently.

This could be because we try to build all packages defined in your overlay when you enter the devShell. This follows the same rules as package exporting, so if you don't want a package to be built you can set __dontExport in the overlay or mark packages as broken. devos/digga can't actually tell which packages are broken unless you explicitly mark it as such and its also good practice to do so. Now if you have done either, set __dontExport or marked the package, that is probably a bug, they shouldn't be built when entering the devShell. Or if you disagree with the practice feel free to explain why and we can discuss how a change would look.

I think overlays, that change custom pkgs are not applied anymore.
I guess this is not a hard problem but sometimes it is cleaner to have a package close to upstream and then apply some changes in an overlay

Now this is an issue, all overlays should be applied to the channels they are passed to. Can you include more details about this? Which overlays weren't applied, and where did you use the package that didn't get changed?

Sorry for kinda spamming this thread, but I think there is another small problem.

Please keep spamming, the more information here the more stable devos is for all of us.

[benneti]

How would I add __dontExport to emacs-overlay (thats not part of my config but an external overlay from another flake, for me that is https://github.com/nix-community/emacs-overlay ).

The package that had the overlay not applied was schildichat and it is installed via homemanager, here the link to the commit where i move the contents of the overlay to the package which makes things work again, but is not that clean anymore https://cloud.tissot.de/gitea/benneti/nixos/commit/09b80c090dab64fb4f7636807eb07d4736d6a620

[Pacman99]

And I realized I disagree with my old self: #14

I don't think building all packages from overlays was the best idea.

[benneti]

I am happy about the opinion of your new self 😅

[Pacman99]

The package that had the overlay not applied was schildichat and it is installed via homemanager, here the link to the commit where i move the contents of the overlay to the package which makes things work again, but is not that clean anymore https://cloud.tissot.de/gitea/benneti/nixos/commit/09b80c090dab64fb4f7636807eb07d4736d6a620

This could be fixed by nix flake lock --update-input pkgs. Otherwise I'm not so sure whats up with the overlay not being applied.

[benneti]

The package that had the overlay not applied was schildichat and it is installed via homemanager, here the link to the commit where i move the contents of the overlay to the package which makes things work again, but is not that clean anymore https://cloud.tissot.de/gitea/benneti/nixos/commit/09b80c090dab64fb4f7636807eb07d4736d6a620

This could be fixed by nix flake lock --update-input pkgs. Otherwise I'm not so sure whats up with the overlay not being applied.

I just checked again to make sure that the lock for pkgs is up to date and even with an just updated pkgs the overlay is not applied (just to make it clear, before the merge of the new branch it was working).
Do I maybe need to namespace overlays for my custom pkgs correctly?

[Pacman99]

No namespacing shouldn't be necessary.

I just pulled your repo and it looks like schildi-desktop does get exported in packages. We export packages based on overlays, so it means the overlay should have been applied.

This could be a problem where home-manager gets the wrong pkgs, maybe something about channels.

[blaggacao]

I don't like that you have been forced to move the overlay into the package definition. This might be caused by the circumstances that the order of.overlays matter and the pkgs.overlay might have got applied afterwards. I don't know at this moment, how imports influences the order. We should validate that.

[Pacman99]

I think it does make sense in that case to define everything in pkgs because schildi-desktop is a new package.

But yeah I have been thinking about overlay order with listOf merging. I believe modules are evaluated in the reverse order. So the overlays defined from imports will actually be first, even though they are "later" in the module set.

[Pacman99]

This could be a good reason for someone to make use of the old method of just importing all the overlays after ./pkgs/default.nix in the same overlays list.

[Pacman99]

Just updated core with the digga fix. But you can just do a nix flake lock --update-input digga for the fix.

[danielphan2003]

Where should I put disabledModules? Is it a separate overlay or I need to define it in flake.nix?

[blaggacao]

disabledModules actually is a plain nixos module attribute, that was previously exposed in the overrides.nix file.

Now, you would set it the same way, yet in a (dedicated) module. For example you could have a modules/disble-foo.nix with:

{
    disabledModules = [ "path/to/module.nix" ];
}

Does that help?

[stites]

I've been adding secrets to a specialArgs.secrets and using this for sensitive information in profiles/ -- could someone recommend a different workflow for this? alternatively, is it still possible to add specialArgs to profiles?

btw it could be nice to have this issue added to the changelog/release notes.

[Pacman99]

We did just add agenix support to the develop branch in devos, so that will be coming in the next release.
I don't think specialArgs is a great method to do this, because it seems like your secrets would end up getting passed around in plain-text. Or maybe you have some other method to keep it secure.
You could make use of _module.args, which lets you pass arguments to modules but it is an option that can be set within a module. So instead of passing specialArgs.secrets you can just set _module.args.secrets in a module.
specialArgs should only be used for arguments that you plan to use in imports lines in the module system, otherwise _module.args is the best method to pass arguments because of portability.