Reduce closure size of executable #392

srid · 2022-12-06T15:57:34Z

Resoves #295

Port of @edrex 's NixOS/nixpkgs#204675

srid · 2022-12-06T15:58:24Z

flake.nix

+                # Ditch data dependencies that are not needed at runtime.
+                # cf. https://github.com/NixOS/nixpkgs/pull/204675
+                postInstall = (old.postInstall or "") + ''
+                  ${lib.concatStrings (map (e: "remove-references-to -t ${e} $out/bin/emanote\n") disallowedReferences)}


Is there a way to not hardcode $out/bin/emanote here?

Because I'm wondering if this can be upstreamed to https://github.com/srid/haskell-flake (cc @roberth) in some form to make it useful for other projects.

$out/bin/* perhaps? However I don't think we should need it. Why are unused references created in the first place?

If this is about data-files, maybe you need extra-source-files instead?

$out/bin/* perhaps? However I don't think we should need it. Why are unused references created in the first place?

Good question. Per NixOS/nixpkgs#204675 (comment) we won't need this after haskell/cabal#8534

With the new changes, I suppose another idea is to just upstream removeReferencesTo to nixpkgs (in pkgs.haskell.lib). That or wait for that cabal PR to go through.

You'd still have to manually specify all the deps, and look at this example to understand context. I figure at that point it's not much harder to copy + edit.

srid · 2022-12-06T16:02:04Z

Makes little difference compared to #295 (comment) ? @edrex Any ideas?

❯ nix path-info -rSh . | sort -rhk2        
warning: Using saved setting for 'extra-substituters = https://cache.garnix.io' from ~/.local/share/nix/trusted-settings.json.
warning: Using saved setting for 'extra-trusted-public-keys = cache.garnix.io:CTFPyKSLcx5RMJKfLo5EEPUObbA78b0YQ2DTCJXqr9g=' from ~/.local/share/nix/trusted-settings.json.
/nix/store/lrki677qy2vpa1y4889h7vjxmik1wydj-emanote-1.0.1.0                                5.0G
/nix/store/2m3zbpinahd83ki2fdscrlwhqqfsnv8r-pandoc-2.17.1.1                                4.5G
/nix/store/8jq127z1bfbs84p80ynpqx6j314wvz93-citeproc-0.6.0.1                               3.3G
/nix/store/jzmijwsh6v28pawigacyx84xabjm0xra-skylighting-0.12.3.1                           3.2G
/nix/store/zpzf6dq7dzf10balbs48b9z8iwryqniq-tree-diff-0.2.2                                3.1G
/nix/store/avb55ygi7g3yw2z2b8fkg602fss48hjp-skylighting-core-0.12.3.1                      3.1G
/nix/store/a52ijkcxx8nqdif7y0541z9vhr48wq8x-http-client-tls-0.3.6.1                        3.1G
/nix/store/0dmn5n5icw9xzwrjld8s8f52x16xqc07-haddock-library-1.10.0                         3.1G
[..]

roberth · 2022-12-06T16:11:55Z

Any ideas?

Did you try nix why-depends <emanote output> <dep output>?

srid · 2022-12-06T16:26:03Z

Looks like there should be a remove-references-to on the pandoc "data" package, too? How do we get a handle on that? pkgs.haskellPackages.pandoc vs ...

❯ nix why-depends -a ./result /nix/store/vz5ayadjlxql78xfw1p2hswprr1wc955-pandoc-2.17.1.1-data
/nix/store/lm25bv6wkl17l40avd9fx03bf2cl15r9-emanote-1.0.1.0
├───/nix/store/vz5ayadjlxql78xfw1p2hswprr1wc955-pandoc-2.17.1.1-data
└───/nix/store/2m3zbpinahd83ki2fdscrlwhqqfsnv8r-pandoc-2.17.1.1
    ├───/nix/store/vz5ayadjlxql78xfw1p2hswprr1wc955-pandoc-2.17.1.1-data
    └───/nix/store/h6b272kfq4r1y8r0zqanfzgh0axzmxr6-pandoc-2.17.1.1-doc
        └───/nix/store/vz5ayadjlxql78xfw1p2hswprr1wc955-pandoc-2.17.1.1-data

srid · 2022-12-06T16:26:56Z

Oh wait, remove-references-to happened on /nix/store/25vbf295m39lnvv8nmhiv1j340i254aj-pandoc-2.17.1.1 but I see /nix/store/2m3zbpinahd83ki2fdscrlwhqqfsnv8r-pandoc-2.17.1.1 in why-depends.

srid · 2022-12-06T16:35:05Z

Fix: ddde55a (had to put it in overrides and use self as the package set to locate those references)

Looking good now,

❯ nix path-info -rSh .#default | sort -rhk2 | head
warning: Git tree '/Users/srid/code/emanote' is dirty
warning: Using saved setting for 'extra-substituters = https://cache.garnix.io' from ~/.local/share/nix/trusted-settings.json.
warning: Using saved setting for 'extra-trusted-public-keys = cache.garnix.io:CTFPyKSLcx5RMJKfLo5EEPUObbA78b0YQ2DTCJXqr9g=' from ~/.local/share/nix/trusted-settings.json.
/nix/store/harhdmbq712flmnpk5gds4kl9mkfxqfn-emanote-1.0.1.0                              517.6M
/nix/store/ypg0hh3ll5yxmqpiq9kqbw9mw55rv591-tailwindcss-3.2.2                            271.0M
/nix/store/v206l0yasw29544linxg1fq0dvqrg2sh-_at_tailwindcss_slash_typography-0.5.7       237.7M
/nix/store/zc25rdbhiva9zybj4c46fnfm84m4sjyy-_at_tailwindcss_slash_line-clamp-0.4.2       237.6M
/nix/store/xbbhfnxdj6d1qf4jl3aq3jk6wik11irp-_at_tailwindcss_slash_aspect-ratio-0.4.2     237.6M
/nix/store/51wg9ywhm7fq01wwnca32qrlp53jq8si-_at_tailwindcss_slash_forms-0.5.3            237.6M
/nix/store/863ragxqwjs33zjdjrlxa5vnyr5ayrf6-postcss-8.4.18                               229.7M
/nix/store/7542hda9k5jy4z4fp5qhzwcd51y8mfq3-nodejs-18.12.1                               229.3M
/nix/store/ns32408gw33gfjvkgnrfbv2h3l0vnzd6-python3-3.10.8                                80.1M
/nix/store/8b4nmpwz1jvhf9wgf4asan1jzmzh68fi-openssl-3.0.7-dev                             66.6M

Also log it in CI.

Reduce closure size of executable

9e406d2

srid commented Dec 6, 2022

View reviewed changes

srid added 2 commits December 6, 2022 11:05

remove newlines

3982dae

Make docs use the final emanote exe

09531b1

srid added 3 commits December 6, 2022 11:29

Log what got removed

0c4c24b

Move the bloat fixing to overrides

f027efd

Use the overriden package set

ddde55a

srid added 7 commits December 6, 2022 11:36

Simplify

139e2ac

Make general

9776ebb

move around

25fba1b

Add path-info log to CI

051188f

make the function do one thing only

4d7aac2

refactor using lib.pipe

b3d0134

rename for clarity

24c7e5e

srid marked this pull request as ready for review December 6, 2022 16:50

srid merged commit 5693a0e into master Dec 6, 2022

srid deleted the closure-size branch December 6, 2022 16:58

This was referenced Dec 6, 2022

Meta: Minimize Emanote's Nix runtime closure size #295

Closed

Docker image is huge (6.5 GB) #326

Closed

srid added a commit that referenced this pull request Nov 28, 2023

ci: Reintroduce change from #392

85b50d6

srid mentioned this pull request Nov 28, 2023

Nix closure size regression #462

Closed

shivaraj-bh pushed a commit to shivaraj-bh/emanote that referenced this pull request Dec 18, 2023

Reduce closure size of executable (srid#392)

5169c72

Also log it in CI.

shivaraj-bh pushed a commit to shivaraj-bh/emanote that referenced this pull request Dec 18, 2023

ci: Reintroduce change from srid#392

203539c

srid mentioned this pull request Dec 24, 2024

nix: Update all inputs #564

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce closure size of executable #392

Reduce closure size of executable #392

srid commented Dec 6, 2022

srid Dec 6, 2022

srid Dec 6, 2022

roberth Dec 6, 2022

roberth Dec 6, 2022

srid Dec 6, 2022

srid Dec 6, 2022 •

edited

Loading

edrex Dec 14, 2022 •

edited

Loading

srid commented Dec 6, 2022

roberth commented Dec 6, 2022

srid commented Dec 6, 2022

srid commented Dec 6, 2022

srid commented Dec 6, 2022

Reduce closure size of executable #392

Reduce closure size of executable #392

Conversation

srid commented Dec 6, 2022

srid Dec 6, 2022

Choose a reason for hiding this comment

srid Dec 6, 2022

Choose a reason for hiding this comment

roberth Dec 6, 2022

Choose a reason for hiding this comment

roberth Dec 6, 2022

Choose a reason for hiding this comment

srid Dec 6, 2022

Choose a reason for hiding this comment

srid Dec 6, 2022 • edited Loading

Choose a reason for hiding this comment

edrex Dec 14, 2022 • edited Loading

Choose a reason for hiding this comment

srid commented Dec 6, 2022

roberth commented Dec 6, 2022

srid commented Dec 6, 2022

srid commented Dec 6, 2022

srid commented Dec 6, 2022

srid Dec 6, 2022 •

edited

Loading

edrex Dec 14, 2022 •

edited

Loading