Shall we use head.hackage in CI for the newest GHC (or all GHCs?)

[Mikolaj]

We discussed

#9610 (comment)

on a cabal fortnightly chat and during the discussion @ulysses4ever proposed a somehow related idea to move our CI (at least for the newest GHC) to https://ghc.gitlab.haskell.org/head.hackage/. I think this should have a higher priority than the original proposal to fix --index-state in our github CI. Let's discuss (and volunteer for subtasks). E.g.,

1. Whether to fix the index state of head.hackage or just follow the HEAD of head.hackage, which is quite a bit curated and fixed often.
2. Whether to use head.hackage for other GHCs and, if so, exclusively or in addition to newest Hackage and/or fixed index state.
3. Maybe add to the mixture --dry-run or some other policy, e.g., to run short tests in many configurations and long ones only in one (but we don't only worry about CI time, but also information overload, especially for new contributors that have to read CI results).
4. The same on our release CI that is located on Haskell gitlab instance.

In addition, let's discuss the original proposal, which included the --index-state proposal for the whole github CI and more. Here's are some remarks about the proposal from the cabal fortnightly chat:

Agree with Bryan's points from the discussion that fixing the index state is useful for a more robust CI.

There is no argument against it, but it is not high priority. Accomodating head.hackage would be a higher priority. Fixing the index state wouldn't save you from dealing with failures due to Hackage updates, it just puts you on a more predictable schedule of when you deal with those failures (points where you update the state).

Unless the failures are already fixed by the point we update our fixed index. But by then we may be inundated by user reports that we can't reproduce in our CI, which is yet another way in which we can't escape the failures some of the time, but they don't disrupt our regular PR merging process at least. There is also the related issue of isolating ourselves from GHA (via containers or custom ghcup invocations). Similar considerations apply. I estimate each has bitten us ~5 times in the last couple of years. That's not many times, but that kind of firefighting tends to burn out maintainers (however, nothing fully isolates us from this, because stack and cabal are the go-to bug trackers for affected users).

CC: @chreekat @philderbeast

[geekosaur]

One point that occurred to me after the call: periodically we bump bounds, and those PRs need to be run without an index-state if we do otherwise freeze it for CI.

[chreekat]

Without putting a lot of thought into it, my immediate feeling is that head.hackage is not stable enough to be used as the basis for CI. After putting a few more seconds of thought into it, I do think it is possible, if done with care.

Some of the questions in the OP don't make sense to me, so I wonder if we lack a clear shared understanding of what head.hackage is. For instance, I understand that head.hackage is only suitable for use with GHC HEAD, i.e. the tip of GHC's master branch. Is that what this ticket is about?

[Mikolaj]

@chreekat: thank you for your thoughts. I think you are right head.hackage is likely to break with old GHCs, even though it often works, I think (it's an overlay, so old packages are still visible, right?). In any case, this was intended to be used for testing the GHC version that is not yet released, but that a cabal major release needs to sync with despite it not being released. Also, it would let us test against the versions of package prepared for the newest GHC, which we can't test from normal Hackage. It would also often catch problems with newest versions of package on Hackage not forced by a GHC release process, because the build plans for that pipeline will normally not exclude any packages as too new.

That CI would be the high-breakage part of our CI, maybe even run overnight and not with every PR. When it's in place, we could make the rest of CI conservative, e.g., sharing the same index state pin as the release CI. Does this clarify things?

[chreekat]

In any case, this was intended to be used for testing the GHC version that is not yet released, but that a cabal major release needs to sync with despite it not being released

Ok! head.hackage is not only good but necessary for this use case.

But I would not recommend trying this use case until https://gitlab.haskell.org/ghc/ghc/-/issues/24000 is solved. I mean, you can try, but I don't know if the cost-benefit tradeoff would be worth it.

[chreekat]

Here's my GHC Nightly status dashboard: https://grafana.gitlab.haskell.org/d/ab109e66-a8a1-4ae9-b976-40e2dfe281ab/availability-of-ghc-nightlies-via-ghcup?orgId=2&refresh=1d

[philderbeast]

$ curl https://ghc.gitlab.haskell.org/head.hackage/cabal.project >> cabal.project.local
$ cabal update

I followed the above head.hackage instructions with master. The default cabal.project builds but not at first;

$ cabal --version
cabal-install version 3.11.0.0
compiled using version 3.11.0.0 of the Cabal library

$ git rev-parse HEAD
46e822152c9b3f38044870115d7a198b3a509d66

$ cabal build all --enable-tests --enable-benchmarks --dry-run
Warning: this is a debug build of cabal-install with assertions enabled.
Resolving dependencies...
Error: [Cabal-7107]
Could not resolve dependencies:
[__0] next goal: Cabal (user goal)
[__0] rejecting: Cabal-3.11.0.0
      (cabal.project.local requires ==2.4.1.0 || ==3.0.2.0 || ==3.2.1.0)
[__0] rejecting: Cabal-3.10.2.1, Cabal-3.10.2.0/installed-fd40, Cabal-3.10.2.0, Cabal-3.10.1.0, Cabal-3.8.1.0, Cabal-3.6.3.0, Cabal-3.6.2.0, Cabal-3.6.1.0, Cabal-3.6.0.0, Cabal-3.4.1.0, Cabal-3.4.0.0, Cabal-3.2.1.0, Cabal-3.2.0.0, Cabal-3.0.2.0, Cabal-3.0.1.0, Cabal-3.0.0.0, Cabal-2.4.1.0, Cabal-2.4.0.1, Cabal-2.4.0.0, Cabal-2.2.0.1, Cabal-2.2.0.0, Cabal-2.0.1.1, Cabal-2.0.1.0, Cabal-2.0.0.2, Cabal-1.24.2.0, Cabal-1.24.0.0, Cabal-1.22.8.0, Cabal-1.22.7.0, Cabal-1.22.6.0, Cabal-1.22.5.0, Cabal-1.22.4.0, Cabal-1.22.3.0, Cabal-1.22.2.0, Cabal-1.22.1.1, Cabal-1.22.1.0, Cabal-1.22.0.0, Cabal-1.20.0.4, Cabal-1.20.0.3, Cabal-1.20.0.2, Cabal-1.20.0.1, Cabal-1.20.0.0, Cabal-1.18.1.7, Cabal-1.18.1.6, Cabal-1.18.1.5, Cabal-1.18.1.4, Cabal-1.18.1.3, Cabal-1.18.1.2, Cabal-1.18.1.1, Cabal-1.18.1, Cabal-1.18.0, Cabal-1.16.0.3, Cabal-1.16.0.2, Cabal-1.16.0.1, Cabal-1.16.0, Cabal-1.14.0, Cabal-1.12.0, Cabal-1.10.2.0, Cabal-1.10.1.0, Cabal-1.10.0.0, Cabal-1.8.0.6, Cabal-1.8.0.4, Cabal-1.8.0.2, Cabal-1.6.0.3, Cabal-1.6.0.2, Cabal-1.6.0.1, Cabal-1.4.0.2, Cabal-1.4.0.1, Cabal-1.4.0.0, Cabal-1.2.4.0, Cabal-1.2.3.0, Cabal-1.2.2.0, Cabal-1.2.1, Cabal-1.1.6, Cabal-1.24.1.0 (constraint from user target requires ==3.11.0.0)
[__0] fail (backjumping, conflict set: Cabal)
After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: Cabal

I can fix this by commenting out the constraint conflict on Cabal in cabal.project.local;

$ git diff
diff --git a/cabal.project.local b/cabal.project.local
index 576224433..9b6fb23cb 100644
--- a/cabal.project.local
+++ b/cabal.project.local
@@ -31,7 +31,7 @@ constraints:
     template-haskell installed
 
 constraints:
-    Cabal ==2.4.1.0 || ==3.0.2.0 || ==3.2.1.0,
+ -- Cabal ==2.4.1.0 || ==3.0.2.0 || ==3.2.1.0,
     Cabal-syntax ==3.8.1.0,
     FPretty ==1.1,
     JuicyPixels ==3.3.8,

Try again and there's another conflict for Cabal-syntax that I can fix the same way;

$ cabal build all --enable-tests --enable-benchmarks --dry-run
Warning: this is a debug build of cabal-install with assertions enabled.
Resolving dependencies...
Error: [Cabal-7107]
Could not resolve dependencies:
[__0] next goal: Cabal-syntax (user goal)
[__0] rejecting: Cabal-syntax-3.11.0.0
      (constraint from cabal.project.local requires ==3.8.1.0)
[__0] rejecting: Cabal-syntax-3.10.2.0/installed-8eb4, Cabal-syntax-3.10.2.0, Cabal-syntax-3.10.1.0, Cabal-syntax-3.8.1.0, Cabal-syntax-3.6.0.0 (constraint from user target requires ==3.11.0.0)
[__0] fail (backjumping, conflict set: Cabal-syntax)
After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: Cabal-syntax

$ git diff
diff --git a/cabal.project.local b/cabal.project.local
index 576224433..c92471375 100644
--- a/cabal.project.local
+++ b/cabal.project.local
@@ -31,8 +31,8 @@ constraints:
     template-haskell installed
 
 constraints:
-    Cabal ==2.4.1.0 || ==3.0.2.0 || ==3.2.1.0,
-    Cabal-syntax ==3.8.1.0,
+ -- Cabal ==2.4.1.0 || ==3.0.2.0 || ==3.2.1.0,
+ -- Cabal-syntax ==3.8.1.0,
     FPretty ==1.1,
     JuicyPixels ==3.3.8,
     ansi-pretty ==0.1.2.2,

Now cabal build ... works.

[philderbeast]

Who maintains https://ghc.gitlab.haskell.org/head.hackage/? It would be nice to have 4 space indenting in the project to allow more room for -- commenting inplace.

[philderbeast]

• Why does https://ghc.gitlab.haskell.org/head.hackage/cabal.project have an allow-newer exception for Cabal but also equality version constraints on Cabal?

• Why is there an allow-newer exception for Cabal but not for Cabal-syntax? They're both boot libraries aren't they, looking at the ghc-9.8.1 included libraries?

[andreabedini]

I'm 👎 on adding head.hackage to the CI. I admit I have been out of the loop and missed some conversations but I fail to see what problems using head.hackage will solve.

[geekosaur]

The main one is that, if you are making a Cabal release intended for an unreleased ghc, you generally need to use head.hackage to get compatible dependencies.

[philderbeast]

There'll be quite a lot of churn won't there if we're going to keep up?

I had a stale cabal.project.local from a week ago and with it there, the project no longer compiles;

$ cabal build all --enable-tests --enable-benchmarks
...
[137 of 137] Compiling Distribution.InstalledPackageInfo
Warning: this is a debug build of cabal-install with assertions enabled.
Error: [Cabal-7125]
Failed to download text-short-0.1.5 (which is required by test:unit-tests from cabal-install-3.11.0.0, test:parser-tests from Cabal-tests-3 and others). The exception was:
  Invalid hash for <repo>/package/text-short-0.1.5.tar.gz
Failed to download th-abstraction-0.6.0.0 (which is required by test:unit-tests from cabal-install-3.11.0.0, test:parser-tests from Cabal-tests-3 and others). The exception was:
  file returned by server too large: <repo>/package/th-abstraction-0.6.0.0.tar.gz (expected exactly 42944 bytes)
Failed to download vector-stream-0.1.0.0 (which is required by test:unit-tests from cabal-install-3.11.0.0, test:parser-tests from Cabal-tests-3 and others). The exception was:
  Unexpected response 404 for https://ghc.gitlab.haskell.org/head.hackage/package/vector-stream-0.1.0.0.tar.gz

Aside from my manual edits for Cabal and Cabal-syntax, vector-stream is now gone from the latest head.hackage (but I'm not using that as cabal.project.local yet - the errors above were from last week's curl of head.hackage);

$  curl https://ghc.gitlab.haskell.org/head.hackage/cabal.project >> cabal.project.cmp

$ diff cabal.project.local cabal.project.cmp
6,7d5
<        7541f32a4ccca4f97aea3b22f5e593ba2c0267546016b992dfadcd2fe944e55d
<        26021a13b401500c8eb2761ca95c61f2d625bfef951b939a8124ed12ecf07329
8a7,8
>        26021a13b401500c8eb2761ca95c61f2d625bfef951b939a8124ed12ecf07329
>        7541f32a4ccca4f97aea3b22f5e593ba2c0267546016b992dfadcd2fe944e55d
34,35c34,35
<  -- Cabal ==2.4.1.0 || ==3.0.2.0 || ==3.2.1.0,
<  -- Cabal-syntax ==3.8.1.0,
---
>     Cabal ==2.4.1.0 || ==3.0.2.0 || ==3.2.1.0,
>     Cabal-syntax ==3.8.1.0,
67a68,71
>     ghc-tcplugins-extra ==0.4.5,
>     ghc-typelits-extra ==0.4.6,
>     ghc-typelits-knownnat ==0.7.10,
>     ghc-typelits-natnormalise ==0.7.9,
111d114
<     vector-stream ==0.1.0.0,

Also I am hitting #9829 after an update;

$ cabal update --ignore-project
Warning: this is a debug build of cabal-install with assertions enabled.
Downloading the latest package list from hackage.haskell.org
Package list of hackage.haskell.org has been updated.
The index-state is set to 2024-03-26T12:05:53Z.
To revert to previous state run:
    cabal v2-update 'hackage.haskell.org,2024-03-21T00:41:16Z'

$ cabal build all --enable-tests --enable-benchmarks
Warning: this is a debug build of cabal-install with assertions enabled.
Warning: Parsing the index cache failed (Data.Binary.Get.runGet at position
16: Non-matching structured hashes: f46da61e7afa58a5e8fd1d2b6fb79899;
expected: d81bdd513f41b5d7ee4cd28455adadbe). Trying to regenerate the index
cache...
~/.cabal/packages/head.hackage.ghc.haskell.org/01-index.tar: withFile: resource busy (file is locked)

To update head.hackage, after first deleting ~/.cabal/packages/head.hackage.ghc.haskell.org I can't --ignore-project as the project holds the repository head.hackage.ghc.haskell.org and other settings we need for head.hackage;

$ cabal build all --enable-tests --enable-benchmarks
Warning: this is a debug build of cabal-install with assertions enabled.
Error: [Cabal-7160]
The package list for 'head.hackage.ghc.haskell.org' does not exist. Run 'cabal update' to download it.

$ cabal build all --enable-tests --enable-benchmarks^C

$ cabal update
Warning: this is a debug build of cabal-install with assertions enabled.
Downloading the latest package lists from:
- hackage.haskell.org
- head.hackage.ghc.haskell.org
Package list of hackage.haskell.org is up to date.
The index-state is set to 2024-03-26T12:05:53Z.
Package list of head.hackage.ghc.haskell.org has been updated.
The index-state is set to 2024-03-26T08:20:48Z.

Using the newest head.hackage and repeating my manual edits to exclude Cabal and Cabal-syntax the project builds again.

[angerman]

Please do not even remotely consider using head.hackage. It is a wart that must stay internal to GHC. If this proliferates outside of GHC we have absolutely lost.

Head.hackage exists only to paper over the fact that GHC head might be in flux in a way that is incompatible with the ecosystem. This in itself should signal strong enough that head.hackage should not be used outside of GHC.

Ontop of that is head.hackage not even particularly stable. You can't pin it properly.

No, just no, don't use it. It shouldn't even exist. The fact that it does is bad enough, if it now also leaves the confines of GHC only, we can just all throw our hand up in despair.

[angerman]

The main one is that, if you are making a Cabal release intended for an unreleased ghc, you generally need to use head.hackage to get compatible dependencies.

Fair. Why would you want to do this though?

(a) in preparation for a GHC release? If so head.hackage for that GHC release should already be the identity (empty!)
(b) for some other in-flight reason? This should not be the responsibility of Cabal. If someone so wants to do something with an inflight, unstable, patches-needed, GHC, this should maybe done exclusively in the GHC repository?