When shall we update the pinned index state in gitlab release CI?

[Mikolaj]

I'm opening a new separate ticket for this question, because this has cost us a serious problem in the 3.10.3.0 release process: an urgent serious upstream fix has not been included in a nearly released deemed-final binaries: haskell/tar#89 (comment)

Older related discussion:

#9610 (comment)

#9808

[Mikolaj]

@Kleidukos: in case of release 3.10.3.0 (and 3.12.0.0), due to haskell/tar#89, I guess we have to update the index right now?

[Kleidukos]

Yes we have to restart the whole process. I'll submit a PR for 3.10

[hasufell]

Just fyi: the binaries that you test in the validate pipeline are potentially not the same as the ones you release (cabal.project.validate does not use index state pinning).

[Kleidukos]

@hasufell ok I'm addressing this in #9820

[Mikolaj]

@hasufell: yes, we are discussing this for the last week. For now, we --dry-run build the release project file with the pinned index state on GHA. There's a plan to run full tests on gitlab. There is a plan to use head.hackage to test against bleeding edge and then we could fix the index in the normal validate tests and not worry that our CI can't reproduce problems users (that compile from source) are having. I don't think we should make radical changes without a proper discussion, except for the disastrous too old index state on release pipeline.

[hasufell]

Well, I have no skin in this game. I have my own way of building cabal release binaries.

[Kleidukos]

That looks like a lot of moving parts from a glance

[chreekat]

Ouch. :(

What bit Cabal here is not the question of when to do semiautomatic freshness bumps, but the fact that the validation CI and the release CI don't share a pin. I think we can just call that a longstanding bug unrelated to the title of this issue.

Whether a pin is updated on a semiregular basis to keep deps fresh, or whether it's updated explicitly to pull in a specific required dep, all CI pipelines should use the same pin, and ideally it is only defined once!

[Kleidukos]

Isn't Phil trying to get a PR in that would substantially increase sharing between cabal project files?

[Kleidukos]

Anyhoo, I'll proceed with my PR for 3.10.

[Mikolaj]

What bit Cabal here is not the question of when to do semiautomatic freshness bumps, but the fact that the validation CI and the release CI don't share a pin. I think we can just call that a longstanding bug unrelated to the title of this issue.

I don't follow. How sharing a pin would make us automatically use the fixed instead of the broken version of package tar?

Whether a pin is updated on a semiregular basis to keep deps fresh, or whether it's updated explicitly to pull in a specific required dep, all CI pipelines should use the same pin, and ideally it is only defined once!

Thank you for your voice in the discussion. Actually it's already cited in #9808. How do you respond to the counter-arguments from the discussion, mostly the one that if we fix index state in all of our CI, we won't be seeing in our CI the errors users experience and report (some users compile cabal from source, others use Cabal as a library and so compile with new or even newest Hackage)? E.g., if newest tar was causing a visible error in our CI (it didn't) just as it did for the reporter, with the pin in place we wouldn't be seeing that error until after the release and so we would include a faulty tar (which had the deeper error for many releases, but not manifested and reported until the last one).

[philderbeast]

Isn't Phil trying to get a PR in that would substantially increase sharing between cabal project files?

@Kleidukos #9565 is approved but I have some squashing to do before adding the merge label and one last look over my shoulder to see what has changed with projects on the master branch.

[hasufell]

mostly the one that if we fix index state in all of our CI

Who said "all"?

[chreekat]

I confess I'm talking about strategy rather than tactics. From a strategic perspective, there is a tension between ensuring CI works reliably and ensuring any random use case of the package "just works". You can do both, to a degree, but it takes a lot of work. Which one do you prioritize?

How do you respond to the counter-arguments from the discussion, mostly the one that if we fix index state in all of our CI, we won't be seeing in our CI the errors users experience and report (some users compile cabal from source, others use Cabal as a library and so compile with new or even newest Hackage)?

How much work do you want to do, and when do you want to do it?

From my perspective, you should worry about having reliable CI first, so that new and existing contributors are able to contribute. Then you can start thinking about making more and more use cases "just work". For example, if people want to build cabal-install from source, you could try recommending a known-good dependency set. (Of course, this requires publishing the "recommended dependency set" somewhere.) If Cabal-the-library stops working with a newer index-state, then maybe Cabal's dependency bounds need revision.

Maybe this isn't useful advice, because I don't have any visibility into the kinds of issues users face when trying to build Cabal themselves. But again, which do you prioritize: reliable CI, or ensuring all possible use cases of the package "just work"? Right now Cabal seems to have neither, and focusing on one at a time might be wise.

mostly the one that if we fix index state in all of our CI

Who said "all"?

I did, for one.

[hasufell]

Well, GHCup doesn't need Cabal's release CI anymore. It would make no difference to me if you drop it altogether.

Of course, you might wish to release your own binaries regardless.

[Kleidukos]

@chreekat @Mikolaj starting with cabal.project.release and cabal.project.validate: what do you think?

[chreekat]

@Kleidukos seems fine, but I will leave it to the project maintainers' decision. :) I have other decisions I need to focus my limited energy on at this time.

[Mikolaj]

@Kleidukos: we'd done some bit of discussion since yesterday, nobody violently objected (nor were there very strong voices in the previous discussion in the devs chat) so I'm no longer opposed to including the CI setup experiment in your release commit. Please go ahead. Would you also kindly add an entry to the release checklist about when to update the index state (in both project files)? [Edit; maybe with a note that whenever we are informed about a grave bug in a used version of our dependency, the index state has to be updated ASAP to a snapshot that contains the corresponding fix.]

@chreekat: I don't think there'd be any discussion "having reliable CI" is the foremost priority and we struggle in delivering even that. Thank you again for sharing your experience.

Of course, you might wish to release your own binaries regardless.

@hasufell: thank you for taking that burden and that failure mode off our shoulders. We will still want nightlies and internal release candidates on many architectures, but it's no longer a critical issue, so we can adjust priorities. I wonder how to start communicating to users that have scripts set up to download our binaries, etc., that they are welcome to switch over to ghcup. I hope that's not controversial.

[ffaf1]

I am still going through the discussion, but let me add this:

I am helping with releasing Cabal 3.12. During the release process, there is a step where you ask — after thorough testing — hackage-security maintainers to relax their bounds on Cabal and Cabal-syntax, and to release it on Hackage.

Of course after hackage-security was released on Hackage, I had to update index-state: in cabal-project.release.

[Kleidukos]

@Mikolaj

I wonder how to start communicating to users that have scripts set up to download our binaries, etc., that they are welcome to switch over to ghcup. I hope that's not controversial.

It is controversial from a product perspective. ghcup is the entrypoint to the toolchain for individuals who don't rely on other distributors. We produce these binaries on downloads.haskell.org for distributors, although in my experience it is better to let the distributors compile their own binaries, and I will show an example why:

Distributions can switch "on" hardening modes at their discretion for their downstream users. When they do it's because they made sure that their whole ecosystem can be transitioned to such a mode (usually it's with gcc/clang flags).

However it's quite hard for the cabal dev team to provide such a life-cycle management for our releases. I fully expect Jens or Julian to tweak our releases with patches to best serve their users, but that's out of scope of us by necessity.

The binaries we provide — while we want to them to actually be usable – are ultimately one reference point for distributors.

I feel like I need to repeat it: we do not have the capacity to be full-time distributors ourselves.

[Mikolaj]

To finalize this single issue at least: I think we are actually forced to update the pinned index state at each major release start, just as @ffaf1 says, in order to include a new release of hackage-security that customarily has strict upper bounds and so needs a new release or a new revision to accept a new cabal major version. Additionally, the case with a tar problem shows that it may be prudent to update at release start to make sure we include all last minute fixes, especially that there may be more fixes than usual due to the testing with the new GHC version the cabal version synces with.

Let me close the ticket with this conclusion. However, please add more comments if you have them and please re-open if you disagree with the recommendation. I've updated https://github.com/haskell/cabal/wiki/Making-a-release accordingly, but this is subject to change by release crew without opening/re-opening tickets.

[chreekat]

I just want to note, for the record, that none of my suggestions are incompatible with removing the pin altogether. That's simply another point on the churn spectrum.