[Feature] remote freeze files

[hasufell]

If you want to use the same freeze file for a plethora of internal projects, then a "global" freeze file might make sense. Currently this would require copying and synchronising such a file across repositories, or having a mono-repo. Being able to set a remote location as a freeze file in cabal.project would solve this use case.

Cli options could allow to say what to do in case fetching fails:

• abort resolution
• continue without a freeze file

[Kleidukos]

This is a great idea and will certainly help bringing cabal to feature parity with stack in enterprise environments.

[merijn]

@hvr spent considerable time to keep the critical path to produce a plan.json fast, so that a no-op cabal build and cabal run latency would be <100ms in order to keep unconditional cabal build and cabal run usable in interactive settings or when integrating with other build tools.

Such a remote fetch for a freeze file would be and almost certainly ruin this criticial path.

[cdsmith]

IMO, it's very reasonable to be concerned about re-fetching a remote freeze file every time you run cabal build, and this is a convincing reason not to do it that way.

There are other ways to do it:

1. A cache that would only re-fetch a freeze file occasionally. Taking an extra couple seconds once a day isn't anything like the same barrier. But the unpredictable timing of picking up changes would be unfortunate.
2. My favorite: Give names to packaged constraint sets, and fetch them as part of cabal update. This opens the door to future work mirroring them in case one source is down, migrating hosting or even hosting on Hackage, integrating with signing and other security measures, etc.

It's not like the latency concern is a dead end for the concept. It's just an argument against one of many possible implementations.

[hasufell]

Wrt 2.: Sounds like a controversial choice to me, since remote freeze files very likely would be per project (although there might be a global one too). cabal update so far has no project specific semantics afaik, so this would be a pretty drastic change in workflow.

As for the latency concerns: I do not share them.

1. This is an optional feature
2. Users who rely on it will care more about the feature than latency. It has zero impact on hackage and no impact on repos you use through source-repository-package. If you don't want it, it doesn't impact you.
3. There are a bunch of reasons cabal today is already pretty slow. Fetching of source-repository-package is one of them. I have never seen people complain about that.
4. Fetch strategies could be added to the configuration, to allow for immutable remote freeze files (uniquely identified by the name). Then you would only fetch once.

I believe that configure would be the correct place for this, since it also fetches remote repos at that stage.

[gbaz]

Recent discussions have convinced me this would be a great feature to have. I think we should give the current freeze workflows a once-over as a first pass to make sure they're sufficiently ergonomic, then just tweak them to allow http fetch with caching. There are at least three features to reduce repeated fetch latency -- cache-control, etag, and last-modified checks. Our current transport backends should have decent support for them, last I checked :-)

[cdsmith]

I believe that configure would be the correct place for this, since it also fetches remote repos at that stage.

I agree with this. I was thinking too complex and missing a simpler answer here. I think it needs to be possible to do cabal build and cabal run without internet access, but if network fetching can already happen during configure, then adding remote freeze files at the same time sounds good to me.

[Kleidukos]

I too agree with this.

[Mikolaj]

@ptkato: does v2-configure still fetch remote repos or did we focus it exclusively on translating build flags to caba.project syntax and overwriting project files?

Even if it still does, given that the use of cabal configure is deprecated in normal workflows and that fetching remote repos is a part of cabal build, wouldn't it be more natural to call cabal build --fetch-remote-only or something similar and include repos and freeze files in that? [Edit: the same as for cabal build applies to cabal install.]

I didn't read closely about your use cases in this issue, so pardon me if I don't make sense.

[hasufell]

Even if it still does, given that the use of cabal configure is deprecated in normal workflows and that fetching remote repos is a part of cabal build, wouldn't it be more natural to call cabal build --fetch-remote-only or something similar and include repos and freeze files in that?

I was under the impression that configure is an internal phase in cabal-install that is run by both cabal configure and cabal build, while the former seems to do additional work.

[Mikolaj]

I was under the impression that configure is an internal phase in cabal-install that is run by both cabal configure and cabal build, while the former seems to do additional work.

Yes, that was a misguided attempt at backward compatibilty with v1-configure, which was a mandatory first step in the standard v1- workflow and which was not duplicated in v1-install. Now the overlap of cabal v2-configure and cabal v2-build (and the same in the alternative cabal v2-install) is gone (in cabal 3.6, I think), which makes it easier to tell which action fits into which command and so I'm just trying to establish exactly that. :)

[hasufell]

Well in that case, the user definitely needs a way to:

1. pre-fetch everything before running cabal build, even if that's a completely different command
2. tell cabal build to not fetch anything ever (I believe --offline exists for that purpose)

[Mikolaj]

I think if cabal build consists of stage1 and stage2, it makes perfect sense to have cabal build --stage1-only and cabal build-stage2-only. The remaining work it delineating the stages sensibly (re-ordering some things, if needed), agreeing on self-explanatory names, porting the flags to cabal install and possibly elsewhere. I'm guessing this is quite tangential to your original issue and your needs and can be done in a separate bike-shedding ticket. However, if you can refine what precise stage1 would fit your use cases best, that would fit in here, certainly.

[fgaz]

pre-fetch everything before running cabal build, even if that's a completely different command

That'd be the new --only-download flag

[hasufell]

I think if cabal build consists of stage1 and stage2, it makes perfect sense to have cabal build --stage1-only and cabal build-stage2-only. The remaining work it delineating the stages sensibly (re-ordering some things, if needed), agreeing on self-explanatory names, porting the flags to cabal install and possibly elsewhere. I'm guessing this is quite tangential to your original issue and your needs and can be done in a separate bike-shedding ticket. However, if you can refine what precise stage1 would fit your use cases best, that would fit in here, certainly.

Well, I don't have too much of a clear picture how current or future cabal-install are structured. But I'd argue that internal phases and external cli interface is a different matter entirely.

Internally, especially in the light of #7394 I'd argue that an approach similar to classic linux distro package managers is desirable. See for example https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-850009.1

So cabal would have at least phases, such as:

• fetch
• unpack
• configure
• build
• install

and possibly more (again, see the link to the gentoo PMS).

My impression was that configure is basically doing fetch, unpack and configure. If it doesn't anymore, I'd argue the only sensible way forward is to have more fine-grained phases, that will also allow users to hook into them. That would also make it trivial to decide on when remote freeze file fetching ought to happen.

The user might only be exposed to the name build and install, while cabal will clearly indicate in its output which phase is currently running.

[Mikolaj]

more fine-grained phases, that will also allow users to hook into them

I'm very much in favour, even as separate explicit plumbing commands, from which porcelain commands for the usual simple workflows are put together. For now, flags to pick the phases and some output indicating which phases are selected and which runs ATM, would probably be fastest to implement (see bike-shedding) and a good start. So, possibly, downloading the freeze files in the phase selected with --only-download makes sense and we'd also give feedback somewhere the phase is started/finished.

Q: is what is currently being done with --only-download, plus freeze files downloading, exactly what you need in your use cases? More than that? Less?

[ptkato]

@ptkato: does v2-configure still fetch remote repos or did we focus it exclusively on translating build flags to caba.project syntax and overwriting project files?

No it doesn't; right now it translates the given flags into a config file. I believe there's some confusion due to the naming of the command, which does give the impression that it does more. The naming ain't the best, I concur, try to see "configure" as in setting up a configuration file.

[hasufell]

Q: is what is currently being done with --only-download, plus freeze files downloading, exactly what you need in your use cases? More than that? Less?

I guess that's fine

[jneira]

Afaiu this could be closed as #7783 is already merged, feel free to reopen if there is any pending question

[cdsmith]

An important use case for this was being able to build with cabal using versions from stackage. What would that look like? Is there work needed to publish stackage package versions in some accessible location with the right format? I guess that can be continued in #7556

[hasufell]

An important use case for this was being able to build with cabal using versions from stackage. What would that look like? Is there work needed to publish stackage package versions in some accessible location with the right format? I guess that can be continued in #7556

You just add this to your cabal.project:

import: https://www.stackage.org/lts-18.26/cabal.config

[fgaz]

Is there work needed to publish stackage package versions in some accessible location with the right format?

The freeze files are already there at https://www.stackage.org/$snapshot/cabal.config since the inception of stackage, though now they are hidden

An important use case for this was being able to build with cabal using versions from stackage. What would that look like?

So the syntax would be (I didn't try it):

import: https://www.stackage.org/lts-xx.yy/cabal.config

[jneira]

For completeness we are about to add the example in docs: #8083

[andreabedini]

I guess one point of disagrement (from commercialhaskell/stackage-server#232) was that "versions from stackage" would still be subject to revisions.

I think with the remote config definitely make a step forward but I am left wondering (because I don't know stack&stackage much): a stackage lts does fix the revisions right? how do we know which revisions are in the lts? (I amit I did have a play with pantry and I think I found a way to get that info out).

[hasufell]

Wrt revisions: #7833

[fgaz]

a stackage lts does fix the revisions right?

On its own it doesn't, because unlike cabal.projec.tfreeze it doesn't include index-state. if index-state is added next to the import, then package versions are completely fixed, including revisions.

[hasufell]

a stackage lts does fix the revisions right?

On its own it doesn't, because unlike cabal.projec.tfreeze it doesn't include index-state. if index-state is added next to the import, then package versions are completely fixed, including revisions.

Freezing index state freezes the entire set of revisions. You can't specify individual revisions.

[andreabedini]

I understand what cabal does, I am just wondering what stack does. E.g. I would expect a package to build with stack and lts-18.26 IFF it builds with cabal and import: https://www.stackage.org/lts-18.26/cabal.config. But this might not be true if cabal follow revisions and stack doesn't.

I was under the impression that since the pantry reform, stackage snapshots will fix all the cabal files too, but I might be wrong here.

[hasufell]

I understand what cabal does, I am just wondering what stack does. E.g. I would expect a package to build with stack and lts-18.26 IFF it builds with cabal and import: https://www.stackage.org/lts-18.26/cabal.config. But this might not be true if cabal follow revisions and stack doesn't.

I was under the impression that since the pantry reform, stackage snapshots will fix all the cabal files too, but I might be wrong here.

Stack has 3 forms:

• acme-missiles-0.3: uses the most recent revision
• acme-missiles-0.3@rev:1 uses revision 1
• acme-missiles-0.3@sha256:2ba66a092a32593880a87fb00f3213762d7bca65a687d45965778deb8694c5d1,631: uses the revision with the given hash (this is discovered automatically)

stackage snapshots usually specify hashes and as such implicitly pin revisions: https://github.com/commercialhaskell/stackage-snapshots/blob/6b1cf107d88d0ac78573d843c26a075d6bfff9c3/lts/19/2.yaml#L62

Also see https://docs.haskellstack.org/en/stable/pantry/

[gbaz]

I don't think the revisions stuff is important. I do think that we're missing another thing still -- the logic for combining constraints remains totally additive -- so if there is a constraint from some imported freeze file, there's no way to override it.

We need a PR that says "within processing project files, constraints on versions should be combined under Last, not under union.

[hasufell]

We need a PR that says "within processing project files, constraints on versions should be combined under Last, not under union.

Wasn't duncans idea to write a nice algebra with combinators?

[gbaz]

yes but we're not doing that.

[hasufell]

yes but we're not doing that.

Why not?

[gbaz]

Because as discussed here #7556 specifying a package algebra is a lot of work and new code which may be subtle and just changing the behavior this way will handle the use cases people are concerned about in a much more straightforward fashion

[hasufell]

Because as discussed here #7556 specifying a package algebra is a lot of work and new code which may be subtle and just changing the behavior this way will handle the use cases people are concerned about in a much more straightforward fashion

How can you specify non-overridable constraints with this approach (e.g. security masks)? Wasn't that one of the major use cases?

[gbaz]

you can't specify non-overridable constraints, but if someone wants to deliberately override a security mask that's their right, prerogative, and problem.

[gbaz]

As an aside, I'm starting to come to the view that the "correct" way to handle security masks is a proper notion of a "vulnerability-warning-list" either orthogonal to or integrated into the main package index, rather than just trying to beef up existing tools to cover this sort of situation.

[hasufell]

I still think an algebra is cooler and more powerful. Is this left open as a future option?

[gbaz]

Sure, there's nothing in the plan that would rule such a thing out.

[hasufell]

I added cabal pre-releases, so people can test this feature on current HEAD comfortably:

ghcup config add-release-channel https://raw.githubusercontent.com/haskell/ghcup-metadata/master/ghcup-prereleases-0.0.7.yaml
ghcup install cabal 3.7.0.0

[andreabedini]

Stack has 3 forms:

Thank you @hasufell, exactly the summary I needed!