hash mismatch in downloaded path #1885

domenkozar · 2018-02-20T15:11:24Z

TLDR; keep all of your binary cache history or don't use binary-cache-v3.sqlite:

hash mismatch in downloaded path '/nix/store/nmhvs4pihm56qnhpb6j09ajcwxjx9h07-digest-0.0.1.2': expected 8afe6105f80feb8f8b9e176bace053da40714598125b1201f91775fc9b1c22f4, got 772e334daa6beff675bee87adaaa2d38e45c0ccd15f065dad32ac88c22cd6637

Turns out what happened:

Hydra built /nix/store/nmhvs4pihm56qnhpb6j09ajcwxjx9h07-digest-0.0.1.2
Build slave substituted it and wrote the hash to binary-cache-v3.sqlite
Hydra GC-ed /nix/store/nmhvs4pihm56qnhpb6j09ajcwxjx9h07-digest-0.0.1.2
Months later someone wanted the same old package and Hydra rebuilt /nix/store/nmhvs4pihm56qnhpb6j09ajcwxjx9h07-digest-0.0.1.2
Build slave downloaded the new narinfo, but binary-cache-v3.sqlite had the old hash

I presume this is due to binary non-determinism?

The fix is: rm /nix/var/nix/binary-cache-v3.sqlite*

The text was updated successfully, but these errors were encountered:

cleverca22 · 2018-02-20T15:15:42Z

sqlite> .mode line
sqlite> select * from NARs where url = 'nar/0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6';
      cache = 2
  storePath = 0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6
        url = nar/0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6
compression = xz
   fileHash = 
   fileSize = 
    narHash = sha256:0yifcx0975amb366q6nshb5qm2rrppy2f7h7k4q66hlblxr8m3rp
    narSize = 4655744
       refs = 0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6 6bw3rq0xhayyj9dl158rqwi8ap395aa0-transformers-compat-0.5.1.4 73n08rwpzjqwvyizwphybcjfd85pkvqh-ghc-8.0.2 fjn0g0q5wrphysg3ijydijcpi274nwi8-gmp-6.1.2 l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25
    deriver = 8f35pf501k765mdz6vjwg1pibmgxckwp-deriving-compat-0.3.6.drv
   signedBy = hydra.iohk.io
  timestamp = 1514632193

$ curl https://hydra.iohk.io/0p7x4l2zab62bghdhc5ssbqi7kpwnblj.narinfo
StorePath: /nix/store/0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6
URL: nar/0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6
Compression: xz
NarHash: sha256:0qsf2vki8g6p3lhn1jqn1vpq8s4lqanjmnfqmg88m9pc7ycgbjmm
NarSize: 4655736
References: 0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6 6bw3rq0xhayyj9dl158rqwi8ap395aa0-transformers-compat-0.5.1.4 73n08rwpzjqwvyizwphybcjfd85pkvqh-ghc-8.0.2 fjn0g0q5wrphysg3ijydijcpi274nwi8-gmp-6.1.2 l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25
Deriver: 8f35pf501k765mdz6vjwg1pibmgxckwp-deriving-compat-0.3.6.drv
System: x86_64-linux
Sig: hydra.iohk.io:fNL1+eP/1CVem0xispVMChbktbEjujeAI0Eu4Te8ka21GRg2zUu+lKnp30Eb6lQjiqC83YVc7DEkEJ5M26B9BQ==

domenkozar · 2018-02-20T15:25:00Z

lesson learned: don't GC your binary cache

AmineChikhaoui · 2018-02-20T15:30:24Z

Or probably introduce a config for reducing the disk cache ttl for both positive/negative lookups. Which should be useful for binary caches that are GCed.

cleverca22 · 2018-02-20T15:52:16Z

https://github.com/NixOS/nix/blob/1.11.15/scripts/download-from-binary-cache.pl.in

i think it can only expire negative hits, so positives stick around forever

nix 2.0 will expire things better

domenkozar · 2018-02-20T17:36:35Z

So

nix/src/libstore/nar-info-disk-cache.cc

Line 52 in c0015e8

const int ttlPositive = 30 * 24 * 3600;

needs a config setting to tweak it.

Gabriella439 · 2018-02-28T19:44:16Z

There is another fix which doesn't require rm /nix/var/nix/binary-cache-v3.sqlite. You can instead add --fallback to your nix-build command or (better) add --option build-fallback true. The latter is more robust because it works for import-from-derivation, too.

See this other issue, which is related to this one: #1148

disk cache lookup for example by doing: nix copy --from <binary-cahe> <store-path> --option \ positive-disk-cache-ttl 0 Issues: NixOS#1885 NixOS#2035

blitz · 2019-12-10T12:13:20Z

@domenkozar We just hit this in our system as well. Was there some activity on this issue since last year? Or is the solution still to not expire things on the cache server?

domenkozar · 2019-12-10T12:23:06Z

See narinfo-cache-positive-ttl in man nix.conf

roberth · 2019-12-10T13:18:48Z

Looks like a duplicate of #2035 - or the other way around I guess

domenkozar · 2020-06-09T12:54:21Z

I really think the correct fix is for binary cache to advertise this setting and user min(user_setting, cache_setting).

roberth · 2020-06-09T13:53:34Z

Can't we just invalidate the entry when it turns out to be unusable and retry the substitution?

That's what @volth suggested in #2035 (comment) and it appears to solve this issue too.

domenkozar · 2020-06-09T13:56:29Z

Sure, that would be most reliable. Note that in IFD we'd have to make sure it works too.

roberth · 2020-06-09T14:01:44Z

Why do you think IFD is different? Afaik, it builds and substitutes normally.

domenkozar · 2020-06-09T14:03:07Z

I don't know the code base, I want to be sure IFD is tested.

domenkozar · 2020-06-10T14:00:55Z

It seems like this would be easily added at https://github.com/NixOS/nix/blob/master/src/libstore/binary-cache-store.cc#L89 to fetch narinfo without using the cache and getting the nar file again.

coderfromhere · 2020-08-17T12:52:07Z

I've stumbled upon the same error message today. The original tar I depend on is https://github.com/NixOS/nixpkgs-channels/archive/f9eba87bf03318587df8356a933f20cfbc81c6ee.tar.gz, when I run:

nix-prefetch-url --unpack https://github.com/NixOS/nixpkgs-channels/archive/6e1393654240cca71efe1c7712c870aad1ab0792.tar.gz
unpacking...
[18.5 MiB DL]
path is '/nix/store/wcmdxyqq4m2761aqyrhz1v28mj5a0wdj-6e1393654240cca71efe1c7712c870aad1ab0792.tar.gz'
0vq5z3pmz5vf4637fhyg4ina3nhg29l94pnf962nw0b5lvwn9mxr

But when instantiating a nix-shell, I get the error:

% nix-shell             
unpacking 'https://github.com/NixOS/nixpkgs-channels/archive/f9eba87bf03318587df8356a933f20cfbc81c6ee.tar.gz'...
error: hash mismatch in file downloaded from 'https://github.com/NixOS/nixpkgs-channels/archive/f9eba87bf03318587df8356a933f20cfbc81c6ee.tar.gz':
  wanted: sha256:0vq5z3pmz5vf4637fhyg4ina3nhg29l94pnf962nw0b5lvwn9mxr
  got:    sha256:15cfqd3hargafkhfq5p9hw75dibnbajh06ha69k18scg9l1p1hnl
(use '--show-trace' to show detailed location information)

how can I keep the same references without breaking checksums in the future?

domenkozar · 2020-08-30T19:29:33Z

I've tried to reproduce this in #3969 but it seems to just work? Maybe my test is not testing the right thing?

domenkozar · 2020-08-30T19:43:22Z

I've found the problem, Nix will say it's copying it, but it will actually fallback to building it.

avanov · 2020-09-03T20:29:03Z

not sure if it helps, but I've just discovered this one:

building '/nix/store/j8z3r8nzig402vwdqww3f2a6bd05zc3q-source.drv'...

trying https://github.com/hashicorp/packer/archive/v1.6.2.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   124  100   124    0     0    558      0 --:--:-- --:--:-- --:--:--   558
100 16.5M    0 16.5M    0     0  3803k      0 --:--:--  0:00:04 --:--:-- 4288k
unpacking source archive /private/var/folders/7s/_k9r0mwd15b4pr4h78v5x2km0000gn/T/nix-build-source.drv-0/v1.6.2.tar.gz
hash mismatch in fixed-output derivation '/nix/store/1x9w8wf4qfgf2xnmafjmqd2mshzk47hr-source':
  wanted: sha256:104jw2jcshzds74d7m4yqn6mbk7lgps6qnqmp6h395b1mdyjhink
  got:    sha256:0kr9whv3s0f8866yjmwg311j3kcj29bp5xwpnv43ama4m1mq3bm7
cannot build derivation '/nix/store/f8idr16gzbh6nbh546ym1d2qyiyi3907-packer-1.6.2.drv': 1 dependencies couldn't be built
error: build of '/nix/store/f8idr16gzbh6nbh546ym1d2qyiyi3907-packer-1.6.2.drv', '/nix/store/wp749d94mr8zx1cpjm388k9ggnbc2aay-ghc-8.10.2-with-packages.drv' failed

MacOS, the pinned nixpgs are:

builtins.fetchTarball {
        name = "nixpkgs-unstable-2020-09-01";
        url = "https://github.com/NixOS/nixpkgs-channels/archive/780da9407e6da081b505f299681c78833357ebe0.tar.gz";
        sha256 = "0l78bgsil1h5i390d34x5qh3raji0wdxqhsqbqs3ihyi6i6c2403";
    };

but then it works well with a more recent NixOS/nixpkgs-channels@a0b3103

builtins.fetchTarball {
        name = "nixpkgs-unstable-2020-09-01";
        url = "https://github.com/NixOS/nixpkgs-channels/archive/a0b3103d81bcd5049dc51598314fc437f6f317d1.tar.gz";
        # Hash obtained using `nix-prefetch-url --unpack <url>`
        sha256 = "0fcsc4qkqd3m4m24wyypppqiv5z06b6wsvsfa90x4qfzbadgdc0x";
    }

Both checkouts were marked as "verifed" on github.

Maybe my reasoning is not right and it is possible to have hash mismatches for some packages in nixpkgs-unstable? Does CI make sure that the entire package set is consistent with their checksums?

roberth · 2020-09-03T20:57:47Z

@coderfromhere @avanov The issues you're seeing seem to be distinct from the original issue discussed here, because your mismatches occur in fetchers and fixed-output derivations, not during substitution.

These are usually caused by changes to the contents backing those URLs you're fetching from. For example, maintainers can change git tags or re-upload an "improved" release tarball and cause a hash mismatch that way. The reason it works on the recent commit, is because it was fixed four days ago.

@coderfromhere In your case, a prefetching command should be able to produce the correct hash. Please make sure that you're using the same parameters in your command and expression and that you're using a fixed-output derivation, not an evaluation-time fetcher like fetchurl. If the issue remains, please open a separate issue.

nixos-discourse · 2021-01-19T05:50:37Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/hash-mismatch-in-downloaded-path/11056/1

domenkozar · 2021-01-19T10:02:10Z

The minimum actionable here is to improve the error message.

stale · 2021-07-19T18:23:38Z

I marked this as stale due to inactivity. → More info

lockejan · 2022-06-27T11:04:37Z

I honestly wonder if this is the improved error message:

darwin-rebuild switch
building the system configuration...
error: hash mismatch in file downloaded from 'https://api.github.com/repos/nixos/nixpkgs/tarball/f2537a505d45c31fe5d9c27ea9829b6f4c4e6ac5':
         specified: sha256:1z28a3gqbv62sxahlssc5a722kh46f26f5ss3arbxpv7a1272vf1
         got:       sha256:1n2z95321nccy3y9h8zw3gsgzwkaigypkybqjva31qd6xb4dl3d1
(use '--show-trace' to show detailed location information)

I tried rm $HOME/.cache/nix/binary-cache-v*.sqlite* as well as narinfo-cache-negative-ttl = 0 without success.

Don't know what to do about it.

thefloweringash · 2022-06-27T11:58:00Z

I honestly wonder if this is the improved error message:
darwin-rebuild switch
building the system configuration...
error: hash mismatch in file downloaded from 'https://api.github.com/repos/nixos/nixpkgs/tarball/f2537a505d45c31fe5d9c27ea9829b6f4c4e6ac5':
         specified: sha256:1z28a3gqbv62sxahlssc5a722kh46f26f5ss3arbxpv7a1272vf1
         got:       sha256:1n2z95321nccy3y9h8zw3gsgzwkaigypkybqjva31qd6xb4dl3d1
(use '--show-trace' to show detailed location information)
I tried rm $HOME/.cache/nix/binary-cache-v*.sqlite* as well as narinfo-cache-negative-ttl = 0 without success.

Don't know what to do about it.

This might be NixOS/nixpkgs#179272. For the URL in the log, I get a the "specified:" hash on (case-sensitive) linux, and the "got:" hash on (case-insensitive) macOS. Was the "specified:" hash computed on a case-sensitive filesystem?

lockejan · 2022-06-29T22:10:56Z

I honestly wonder if this is the improved error message:
darwin-rebuild switch
building the system configuration...
error: hash mismatch in file downloaded from 'https://api.github.com/repos/nixos/nixpkgs/tarball/f2537a505d45c31fe5d9c27ea9829b6f4c4e6ac5':
         specified: sha256:1z28a3gqbv62sxahlssc5a722kh46f26f5ss3arbxpv7a1272vf1
         got:       sha256:1n2z95321nccy3y9h8zw3gsgzwkaigypkybqjva31qd6xb4dl3d1
(use '--show-trace' to show detailed location information)
I tried rm $HOME/.cache/nix/binary-cache-v*.sqlite* as well as narinfo-cache-negative-ttl = 0 without success.
Don't know what to do about it.
This might be NixOS/nixpkgs#179272. For the URL in the log, I get a the "specified:" hash on (case-sensitive) linux, and the "got:" hash on (case-insensitive) macOS. Was the "specified:" hash computed on a case-sensitive filesystem?

Well, at first I thought it's related but it's actually not. I'm following 22.05-release-channel.
The mentioned commit never made it to my channel.

Steps To Reproduce

Steps to reproduce the behavior:

use any recent channel, e.g. https://nixos.org/channels/nixpkgs-unstable or https://nixos.org/channels/nixpkgs-22.05-darwin or even nixpkgs-22.05-nixos (it happens on all of them)
use nix 2.8.1
sudo -i nix-channel --update
darwin-rebuild switch

I tried imperatively install nix-2.9.1 and then tested why darwin-rebuild switch again, but I'm not sure if it really picked up the nix-version from nix-env. At least it failed exactly the same way.

Additional context

sudo -i nix-channel --list

darwin https://github.com/LnL7/nix-darwin/archive/master.tar.gz
home-manager https://github.com/nix-community/home-manager/archive/release-22.05.tar.gz
nixpkgs https://nixos.org/channels/nixpkgs-22.05-darwin

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 21.5.0, macOS 12.4`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.8.1`
 - channels(lockejan): `""`
 - channels(root): `"darwin, home-manager-22.05.tar.gz, nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

domenkozar · 2022-06-29T22:42:20Z

I wonder if we could catch the bug with a sanity script. Someone with good knowledge of find?

thefloweringash · 2022-06-30T07:05:18Z

One way would be to canonicalize the list of filenames and check for duplicates:

❯ curl -fsSL https://api.github.com/repos/nixos/nixpkgs/tarball/f2537a505d45c31fe5d9c27ea9829b6f4c4e6ac5 \
    | tar tz | tr A-Z a-z | sort | uniq -d
nixos-nixpkgs-f2537a5/pkgs/tools/x11/

zimbatm · 2022-06-30T16:33:08Z

nixpkgs was fixed in NixOS/nixpkgs@aae0476

lockejan · 2022-06-30T16:55:40Z

nixpkgs was fixed in NixOS/nixpkgs@aae0476

Indeed, but because I used the neovim-nightly-overlay it remained broken until its flake.lock file got updated to a point of nixpkgs where the fix got merged.
I don't really understand it yet, since I don't use flakes, but @thefloweringash brilliantly figured this out and pointed to https://github.com/nix-community/neovim-nightly-overlay/blob/e82a3b43fd30507b8de1e11384282f0479926656/flake.lock#L58-L65 as the trouble making section.

I only updated my channels today and the problem finally disappeared.

domenkozar · 2022-06-30T21:43:22Z

One way would be to canonicalize the list of filenames and check for duplicates:

❯ curl -fsSL https://api.github.com/repos/nixos/nixpkgs/tarball/f2537a505d45c31fe5d9c27ea9829b6f4c4e6ac5 \
    | tar tz | tr A-Z a-z | sort | uniq -d
nixos-nixpkgs-f2537a5/pkgs/tools/x11/

Do you want to open a PR to add that to tarball validation steps?

shlevy added the backlog label Apr 1, 2018

shlevy self-assigned this Apr 1, 2018

domenkozar removed the backlog label Apr 30, 2020

domenkozar mentioned this issue Jun 10, 2020

Nix caches (locally) 404 responses received from (remote) binary caches #781

Closed

domenkozar added the bug label Jul 29, 2020

domenkozar mentioned this issue Aug 30, 2020

fallback to refreshing narinfo when nar is missing #3969

Closed

domenkozar added the error-messages label Jan 19, 2021

stale bot added the stale label Jul 19, 2021

stale bot removed the stale label Jun 27, 2022

lovesegfault mentioned this issue Jun 28, 2022

System dependent narHash in flake.lock #6731

Closed

lockejan mentioned this issue Jun 29, 2022

hash mismatch in downloaded file NixOS/nixpkgs#179601

Closed

thefloweringash mentioned this issue Jul 4, 2022

nixpkgs-basic-release-checks: check for case-insensitive path conflicts NixOS/nixpkgs#180091

Merged

13 tasks

stale bot added the stale label Jan 8, 2023

vielmetti mentioned this issue Feb 15, 2023

SHA256 hash mismatches (deno cache and curl | sha256sum disagree) SnO2WMaN/deno2nix#27

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

hash mismatch in downloaded path #1885

hash mismatch in downloaded path #1885

domenkozar commented Feb 20, 2018 •

edited

Loading

cleverca22 commented Feb 20, 2018 •

edited

Loading

domenkozar commented Feb 20, 2018

AmineChikhaoui commented Feb 20, 2018

cleverca22 commented Feb 20, 2018 •

edited

Loading

domenkozar commented Feb 20, 2018

Gabriella439 commented Feb 28, 2018

blitz commented Dec 10, 2019

domenkozar commented Dec 10, 2019

roberth commented Dec 10, 2019 •

edited

Loading

domenkozar commented Jun 9, 2020 •

edited

Loading

roberth commented Jun 9, 2020

domenkozar commented Jun 9, 2020

roberth commented Jun 9, 2020

domenkozar commented Jun 9, 2020 •

edited

Loading

domenkozar commented Jun 10, 2020 •

edited

Loading

coderfromhere commented Aug 17, 2020 •

edited

Loading

domenkozar commented Aug 30, 2020

domenkozar commented Aug 30, 2020

avanov commented Sep 3, 2020 •

edited

Loading

roberth commented Sep 3, 2020 •

edited

Loading

nixos-discourse commented Jan 19, 2021

domenkozar commented Jan 19, 2021

stale bot commented Jul 19, 2021

lockejan commented Jun 27, 2022 •

edited

Loading

thefloweringash commented Jun 27, 2022

lockejan commented Jun 29, 2022 •

edited

Loading

domenkozar commented Jun 29, 2022

thefloweringash commented Jun 30, 2022

zimbatm commented Jun 30, 2022

lockejan commented Jun 30, 2022

domenkozar commented Jun 30, 2022

hash mismatch in downloaded path #1885

hash mismatch in downloaded path #1885

Comments

domenkozar commented Feb 20, 2018 • edited Loading

cleverca22 commented Feb 20, 2018 • edited Loading

domenkozar commented Feb 20, 2018

AmineChikhaoui commented Feb 20, 2018

cleverca22 commented Feb 20, 2018 • edited Loading

domenkozar commented Feb 20, 2018

Gabriella439 commented Feb 28, 2018

blitz commented Dec 10, 2019

domenkozar commented Dec 10, 2019

roberth commented Dec 10, 2019 • edited Loading

domenkozar commented Jun 9, 2020 • edited Loading

roberth commented Jun 9, 2020

domenkozar commented Jun 9, 2020

roberth commented Jun 9, 2020

domenkozar commented Jun 9, 2020 • edited Loading

domenkozar commented Jun 10, 2020 • edited Loading

coderfromhere commented Aug 17, 2020 • edited Loading

domenkozar commented Aug 30, 2020

domenkozar commented Aug 30, 2020

avanov commented Sep 3, 2020 • edited Loading

roberth commented Sep 3, 2020 • edited Loading

nixos-discourse commented Jan 19, 2021

domenkozar commented Jan 19, 2021

stale bot commented Jul 19, 2021

lockejan commented Jun 27, 2022 • edited Loading

thefloweringash commented Jun 27, 2022

lockejan commented Jun 29, 2022 • edited Loading

Steps To Reproduce

Additional context

Metadata

domenkozar commented Jun 29, 2022

thefloweringash commented Jun 30, 2022

zimbatm commented Jun 30, 2022

lockejan commented Jun 30, 2022

domenkozar commented Jun 30, 2022

domenkozar commented Feb 20, 2018 •

edited

Loading

cleverca22 commented Feb 20, 2018 •

edited

Loading

cleverca22 commented Feb 20, 2018 •

edited

Loading

roberth commented Dec 10, 2019 •

edited

Loading

domenkozar commented Jun 9, 2020 •

edited

Loading

domenkozar commented Jun 9, 2020 •

edited

Loading

domenkozar commented Jun 10, 2020 •

edited

Loading

coderfromhere commented Aug 17, 2020 •

edited

Loading

avanov commented Sep 3, 2020 •

edited

Loading

roberth commented Sep 3, 2020 •

edited

Loading

lockejan commented Jun 27, 2022 •

edited

Loading

lockejan commented Jun 29, 2022 •

edited

Loading