By default, new-build should freeze the Hackage index revision, so "cabal update" does not cause rebuilds

[jaccokrijnen]

See #3832 (comment) for how to avoid user confusion.

Original report below.

Not sure if this is expected behaviour, I am new-building leksah after running a cabal update. It seems to be installing many of my dependencies again and duplicating them in the global database (probably having a different hash?) ghc-pkg gives me this for example:

ghc-7.10.3/package.db/ | grep gi-soup
    gi-soup-2.4.6
    gi-soup-2.4.6

I'm not sure how I can reproduce this to see what caused it, because the build plan is now dependent on the hackage index right?

By the way, shouldn't there be hashes in the package id's? This is with cabal 1.24.

[ezyang]

This is expected behavior, but perhaps we can make the reasons why more transparent to the user.

The principle of new-build is that builds should be deterministic with respect to the Hackage index. That means that new-build will never attempt to prefer existing packages in the database. It just does dependency solving as if there was nothing preinstalled, and if it happens to pick the same dependency resolution which lets you reuse something, great. But it will always act the same.

CC'ing @hvr, I actually think it is a bit bad that despite all we are for determinism, the Hackage index is still one big honking source of non-reproducibility. If we were to automatically freeze the Hackage index (let's add another config file, cabal.project.snapshot) and then just let the user know when their global index is newer than their local index, with a cabal new-update to actually advance the index forward, I think that would be much better user experience. Then @jaccokrijnen can run cabal update but still use the old index for leksah until he feels like updating the dependencies.

Some other matters:

• ghc-pkg by default does NOT display IPIDs. I guess this is not so clever in the age of Nix-style local builds. You could probably do ghc-pkg field gi-soup id to get the ids?
• Since we don't have GC ([nix-local-build] Garbage collecting the store #3333) the old install files will stick around forever. But it's safe to delete the store, you'll just end up rebuilding a bunch of stuff.
• Would you have liked it if we diffed the package against existing ones in the store, and told you why they needed to be rebuilt? A difficult problem of not putting in too much info to overload the user. Maybe some sort of hysteresis just in the UI would be helpful (last time you called new-build, we decided to build these packages; this time around, this changed...)

[23Skidoo]

If we were to automatically freeze the Hackage index (let's add another config file, cabal.project.snapshot) and then just let the user know when their global index is newer than their local index, with a cabal new-update to actually advance the index forward, I think that would be much better user experience. Then @jaccokrijnen can run cabal update but still use the old index for leksah until he feels like updating the dependencies.

How will this work when new-update becomes update?

[ezyang]

How about, update from within a cabal.project writes out a snapshot config file, outside it just downloads a newer index. (Ugh, do we need an offline mode haha.)

[jaccokrijnen]

Sounds like a nice plan! A project-local snapshat would make a lot of sense and sounds like another big improvement for reproducibility.

Would you have liked it if we diffed the package against existing ones in the store, and told you why they needed to be rebuilt? A difficult problem of not putting in too much info to overload the user. Maybe some sort of hysteresis just in the UI would be helpful (last time you called new-build, we decided to build these packages; this time around, this changed...)

The local snapshot would already solve a lot of stuff. But for cases like this where it's installing packages with an already present version number (probably some other flag assignment) it would be very nice to have :)

[hvr]

@ezyang fwiw, I have slightly other plans (especially because I like the current behaviour and don't want to loose it; as I said elsewhere, we can still implement that per-project update feature, but make it configurable as I wouldn't want it to be default for me), and plan to hook up index-snapshot to cabal new-freeze (where it actually belongs into and is actually a bug that it's missing currently IMO) asap... let's talk about this again when I'm finally done with that.. ;-)

[dcoutts]

I was also thinking we might do a semi-freeze by default, ie first time you run (new-)configure/build it'd stick with the current snapshot, and then not be affected by running update in other projects/places until you run update again in this project.

[ezyang]

Pinging @hvr: what is the status update on index snapshot and freezing?

[gbaz]

snapshot and freezing are both in afaik. I guess this ticket is worth keeping in for the discussion over if if people want to consider making a semi-freeze by default?

[hanshoglund]

@hvr If I understand the above correctly, the proposed design is to make cabal v2-freeze write the index version to the freeze file, which would make source trees with freeze files reproducible.

Has this been implemented recently (Cabal 3.0.0 does not seem to do it)?

[phadej]

No it isn't. The underlying problem is that the ~/.cabal/config (or cabal.project) may have other than hackage.haskell.org repositories (and the set might change).

So whatever is written to .freeze file should freeze the repositories as well. Yet, currently the constraints are already quite rigid, updating 01-index file shouldn't change the plan if there's only Hackage in use. Thus, this isn't really high priority as proper solution doesn't have that much ROI.

[hanshoglund]

Hi Oleg, thanks for replying,

No it isn't

Do you mean "That is not the design", or "it is not implemented"?

updating 01-index file shouldn't change the plan if there's only Hackage in use

Surely new revisions can breaks builds by e.g. pushing unsolvable constraints. Do you mean that this is not done in practice?

So whatever is written to .freeze file should freeze the repositories as well.

Surely freeze could just store the state of each repo, similar to #5965

[hanshoglund]

OK I now see I can get the behavior I want by setting index-state in cabal.project (though only for Hackage as per #5965). It would be nice if this was handled by freeze though.

[phadej]

v2-freeze will write index-state. Implemented in #6597