Cabal and Cabal-syntax API checking

[geekosaur]

Not checking cabal-install or cabal-install-solver currently because they don't have public APIs. This may be incorrect for cabal-install-solver, so that may be added later.

Template B: This PR does not modify behaviour or interface

E.g. the PR only touches documentation or tests, does refactorings, etc.

Include the following checklist in your PR:

• Patches conform to the coding conventions.
• Is this a PR that fixes CI? If so, it will need to be backported to older cabal release branches (ask maintainers for directions).

There should be no need for manual QA, because it's already been tested by multiple rebases on master which changed API (for example, #10270).

[ulysses4ever]

Wow, this is awesome! 🤩

[geekosaur]

Do you happen to know whether cabal-install-solver needs to be included here, since it's a library? My assumption is that it's internal and as such has no public API to be checked.

[geekosaur]

Also, this is very much WIP as I sort out what's needed (and deal with weirdshit like the fact that I somehow got a byte order mark the first time, and subsequent runs don't like it much).

[ulysses4ever]

Cabal and Cabal-syntax are mission critical. The solver package is not (e.g. reverse dependencies count from Hackage: 259 for Cabal vs 1 for solver). So it's fine to start with those.

[geekosaur]

Just made the current API dumps into artifacts, since not everyone will have an Ubuntu box to run print-api on and sufficiently large changes would be a PITA working from diff output.

[Kleidukos]

@geekosaur why not use GHC 9.10.1? :)

[geekosaur]

Originally I was thinking 9.4.8 for consistency with GHC_FOR_RELEASE, then discovered that wasn't supported so I bumped it. I could do it again with 9.10.1 if that's preferred.

[Kleidukos]

Yes, let's use the highest supported GHC now and bump it once it falls out of the support window

[mpickering]

Please can we provide a means to easily update these files with a single command locally?

Ideally it should be possible to run all cabal tests locally with a single command and update the output within the same framework. This is already an issue with the structured hash test, which is difficult to update and dependent on GHC version, and in practice, just more annoying busywork than informative for contributors.

Otherwise, I don't think this job should be required by contributors as it introduces a significant amount of friction. It could be run as part of CI to track how the API has changed over time rather than requiring a golden file to update or perhaps this test could just be run on release branches (where the API is not expected to change).

[sheaf]

I think having such an API test is going to be very valuable, as it allows us to confidently assess the API impact of PRs under review. This will give us much more confidence and allow us to write more accurate changelogs, and avoid accidental changes in minor Cabal version etc. I'm grateful that the Cabal project is taking such a step.

However, I share Matthew's concern about the workflow for accepting the test. At the moment one has to provision a docker container to run with the appropriate environment for the test. Either that, or one ends up with a situation like the structured hash test where one has to wait for CI to run, let it fail because of the test, then change the stdout locally, then push again. If one needs one more CI run than one did before then I think it is too onerous; I would much rather that the structured hash test and this test be separate tests which don't cause the whole CI pipeline to fail if they fail, so that they can be updated in parallel with working on the PR without cancelling the CI which might otherwise have valuable information.

I also thing the surface area we are testing is a bit too large (I don't think the Cabal project has any guarantees about the entire set of exposed modules; that seems too much). As a result, I'm worried that we will get unreadable diffs as a result whenever there is a significant refactor. But I suppose we can wait and see how things pan out in that regard.

[Kleidukos]

I think having such an API test is going to be very valuable, as it allows us to confidently assess the API impact of PRs under review. This will give us much more confidence and allow us to write more accurate changelogs, and avoid accidental changes in minor Cabal version etc. I'm grateful that the Cabal project is taking such a step.

👍

However, I share Matthew's concern about the workflow for accepting the test

We all do, I don't think anyone has objected publicly or privately.

I also thing the surface area we are testing is a bit too large (I don't think the Cabal project has any guarantees about the entire set of exposed modules; that seems too much). As a result, I'm worried that we will get unreadable diffs as a result whenever there is a significant refactor. But I suppose we can wait and see how things pan out in that regard.

Yep, configuring the tool with an "ignore" list is in the plans. Cabal is just an early adopter and the needs of adopters will help determining how print-api & diff-package-api can provide the most value. Example: @geekosaur had to bypass the pre-made Workflow in order to test multiple packages at once.

[geekosaur]

However, I share Matthew's concern about the workflow for accepting the test. At the moment one has to provision a docker container to run with the appropriate environment for the test.

Actually no: as a first cut, the new API files are artifacts which can be downloaded and copied over the existing ones, specifically for this case. If there's some friendly way to make it completely automatic, I'm all ears, but I did specifically address that "contributors need to replicate the CI environment" issue. (And will shortly be using it myself, since I'll be redoing the golden API files for 9.10.1.)

[geekosaur]

Otherwise, I don't think this job should be required by contributors as it introduces a significant amount of friction. It could be run as part of CI to track how the API has changed over time rather than requiring a golden file to update or perhaps this test could just be run on release branches (where the API is not expected to change).

As semi-official release manager for the 3.12 branch, I would like this to run on every incoming PR. I do need to tune that though, as I don't want API changes to prevent merges; I want them to inform me whether a given PR is a backport candidate or not. As I'm still learning the details of workflows, I should probably put this back in draft even though I'm soliciting reviews.

[geekosaur]

Uh, let me correct that: the test should always pass, this is accomplished by updating the golden tests by copying the new API files over the old ones as described above. Then the presence of such updates tells me as release manager that the PR shouldn't be backported.

I think my ideal for both this and the structured hashes is that they'd be checkboxes in the PR that the tests could inspect in order to determine whether to report a mismatch or do an update. But

1. AFAICT there's no way to inspect PR checkboxes from workflows;
2. the multiple templates would make this difficult anyway.

[philderbeast]

@geekosaur do you want to split this into two PRs? It would be useful to run the make targets locally if we suspect an API change.

[geekosaur]

@philderbeast, I'm not sure what a split would be worth. The only split that makes any sense to me would be to make the API files a separate commit in this PR, facilitating cherry-picking the rest into a test branch.

[philderbeast]

@geekosaur, I'm trying this out. I was able to install print-api with make print-api-install:

PRINT_API_VERSION ?= eedf83e6f828217ba7946ca8bfaf4ab4062c2363
PRINT_API_URL := https://github.com/Kleidukos/print-api/archive/${PRINT_API_VERSION}.tar.gz

print-api/print-api.cabal:
	rm -rf print-api
	curl -sSL ${PRINT_API_URL} | tar -xz
	mv print-api-* print-api

print-api-install: print-api/print-api.cabal
	cabal install print-api:exe:print-api --project-dir=print-api --overwrite-policy=always

[philderbeast]

@geekosaur I was thinking of a split into two, one with the makefile changes (so we could use it locally) and the other with CI changes.

[philderbeast]

When I run it I'm seeing an error:

$ make generate-api-cabal-syntax
...
ghcup run --ghc 9.10.1 -- print-api --package-name Cabal-syntax | \
   sed 's/\([( ]\)[Cc]abal-[-0-9.][-0-9]*:/\1/g' >Cabal-syntax-9.10.1.api
Detected GHC version: 9.10.1
print-api: print-api-9.10.1: can't find a package database at /.../dist-newstyle/packagedb/ghc-9.10.1
HasCallStack backtrace:
  finally, called at compiler/GHC/Utils/Panic.hs:303:7 in ghc-9.10.1-69c3:GHC.Utils.Panic

Note

The fix for this error is to build Cabal-syntax itself first.

[geekosaur]

Huh. I guess I'll have to add that to the rules. (I seem to have hit the same problem when trying to sanity check 3.12 branch against its base.)

[geekosaur]

Uhhhhh… @Kleidukos, why is print-api trying to build cabal-testsuite? (This is on 3.12.1.0, and it looks like something got bumped inappropriately.)

ghcup run --ghc 9.10.1 -- print-api --package-name cabal-install-solver | sed 's/\([( ]\)[Cc]abal-[-0-9.][-0-9]*:/\1/g' >cabal-install-solver-9.10.1.api
print-api: Error: [Cabal-7107]
Could not resolve dependencies:
[__0] trying: cabal-testsuite-3 (user goal)
[__1] next goal: cabal-testsuite:setup.Cabal (dependency of cabal-testsuite)
[__1] rejecting: cabal-testsuite:setup.Cabal-3.14.0.0 (conflict: cabal-testsuite => cabal-testsuite:setup.Cabal>=3.10 && <3.11)
[__1] skipping: cabal-testsuite:setup.Cabal; 3.12.1.0, 3.12.0.0/installed-be90, 3.12.0.0 (has the same characteristics that caused the previous version to fail: excluded by constraint '>=3.10 && <3.11' from 'cabal-testsuite')
[__1] rejecting: cabal-testsuite:setup.Cabal; 3.10.3.0, 3.10.2.1, 3.10.2.0, 3.10.1.0, 3.8.1.0, 3.6.3.0, 3.6.2.0, 3.6.1.0, 3.6.0.0, 3.4.1.0, 3.4.0.0, 3.2.1.0, 3.2.0.0, 3.0.2.0, 3.0.1.0, 3.0.0.0, 2.4.1.0, 2.4.0.1, 2.4.0.0, 2.2.0.1, 2.2.0.0, 2.0.1.1, 2.0.1.0, 2.0.0.2, 1.24.2.0, 1.24.0.0, 1.22.8.0, 1.22.7.0, 1.22.6.0, 1.22.5.0, 1.22.4.0, 1.22.3.0, 1.22.2.0, 1.22.1.1, 1.22.1.0, 1.22.0.0, 1.20.0.4, 1.20.0.3, 1.20.0.2, 1.20.0.1, 1.20.0.0, 1.18.1.7, 1.18.1.6, 1.18.1.5, 1.18.1.4, 1.18.1.3, 1.18.1.2, 1.18.1.1, 1.18.1, 1.18.0, 1.16.0.3, 1.16.0.2, 1.16.0.1, 1.16.0, 1.14.0, 1.12.0, 1.10.2.0, 1.10.1.0, 1.10.0.0, 1.8.0.6, 1.8.0.4, 1.8.0.2, 1.6.0.3, 1.6.0.2, 1.6.0.1, 1.4.0.2, 1.4.0.1, 1.4.0.0, 1.2.4.0, 1.2.3.0, 1.2.2.0, 1.2.1, 1.1.6, 1.24.1.0 (constraint from minimum version of Cabal used by Setup.hs requires >=3.12)
[__1] fail (backjumping, conflict set: cabal-testsuite, cabal-testsuite:setup.Cabal)
After searching the rest of the dependency tree exhaustively, these were the goals I've had most trouble fulfilling: cabal-testsuite:setup.Cabal, cabal-testsuite


CallStack (from HasCallStack):
  error, called at src/PrintApi/CLI/Types.hs:34:7 in print-api-0.1.0.1-inplace:PrintApi.CLI.Types
HasCallStack backtrace:
  collectBacktraces, called at libraries/ghc-internal/src/GHC/Internal/Exception.hs:92:13 in ghc-internal:GHC.Internal.Exception
  toExceptionWithBacktrace, called at libraries/ghc-internal/src/GHC/Internal/Exception.hs:128:3 in ghc-internal:GHC.Internal.Exception

---

ETA: no, this was our forgetting to bump the custom-setup on release (#10172). I still would like to know why print-api cares, though.

[geekosaur]

Welp. With this fix it appears that a few API changes leaked into 3.12 after all. That said, they're slight and I think they might be safe because there's backward compatibility code.

• Cabal-syntax: new pattern synonym, which provides backward compatibility to changes to SCC, plus a Foldable1 instance for it
• Cabal: reexports of the preceding
• cabal-install-solver: reexporting the above Foldable1 instance

For the record:

diff -c Cabal-syntax/Cabal-syntax-9.10.1.api Cabal-syntax-9.10.1.api
*** Cabal-syntax/Cabal-syntax-9.10.1.api	2024-12-03 20:22:52.606058049 -0500
--- Cabal-syntax-9.10.1.api	2024-12-03 20:47:59.628820415 -0500
***************
*** 162,167 ****
--- 162,168 ----
    tryIO :: forall a. GHC.Types.IO a -> GHC.Types.IO (GHC.Internal.Data.Either.Either GHC.Internal.IO.Exception.IOException a)
  
  module Distribution.Compat.Graph where
+   pattern CyclicSCC :: forall vertex. [vertex] -> SCC vertex
    type role Graph nominal
    type Graph :: * -> *
    data Graph a = ...
***************
*** 175,181 ****
    type Node :: * -> * -> *
    data Node k a = N a k [k]
    type SCC :: * -> *
!   data SCC vertex = AcyclicSCC vertex | CyclicSCC [vertex]
    broken :: forall a. Graph a -> [(a, [Key a])]
    closure :: forall a. Graph a -> [Key a] -> GHC.Internal.Maybe.Maybe [a]
    cycles :: forall a. Graph a -> [[a]]
--- 176,182 ----
    type Node :: * -> * -> *
    data Node k a = N a k [k]
    type SCC :: * -> *
!   data SCC vertex = AcyclicSCC vertex | NECyclicSCC {-# UNPACK #-}(GHC.Internal.Base.NonEmpty vertex)
    broken :: forall a. Graph a -> [(a, [Key a])]
    closure :: forall a. Graph a -> [Key a] -> GHC.Internal.Maybe.Maybe [a]
    cycles :: forall a. Graph a -> [[a]]
***************
*** 5246,5251 ****
--- 5247,5253 ----
  instance Cabal-syntax-3.12.1.0:Distribution.Utils.Structured.Structured Cabal-syntax-3.12.1.0:Language.Haskell.Extension.KnownExtension -- Defined in ‘Cabal-syntax-3.12.1.0:Language.Haskell.Extension’
  instance Cabal-syntax-3.12.1.0:Distribution.Utils.Structured.Structured Cabal-syntax-3.12.1.0:Language.Haskell.Extension.Language -- Defined in ‘Cabal-syntax-3.12.1.0:Language.Haskell.Extension’
  instance Data.Bifoldable.Bifoldable Data.Map.Internal.Map -- Defined in ‘Data.Map.Internal’
+ instance [safe] Data.Foldable1.Foldable1 Data.Graph.SCC -- Defined in ‘Data.Graph’
  instance Data.Foldable1.Foldable1 Cabal-syntax-3.12.1.0:Distribution.Fields.Field.Field -- Defined in ‘Cabal-syntax-3.12.1.0:Distribution.Fields.Field’
  instance Data.Foldable1.Foldable1 Cabal-syntax-3.12.1.0:Distribution.Fields.Field.FieldLine -- Defined in ‘Cabal-syntax-3.12.1.0:Distribution.Fields.Field’
  instance Data.Foldable1.Foldable1 Cabal-syntax-3.12.1.0:Distribution.Fields.Field.Name -- Defined in ‘Cabal-syntax-3.12.1.0:Distribution.Fields.Field’

diff -c Cabal/Cabal-9.10.1.api Cabal-9.10.1.api
*** Cabal/Cabal-9.10.1.api	2024-12-03 20:22:52.610058071 -0500
--- Cabal-9.10.1.api	2024-12-03 20:50:49.281418522 -0500
***************
*** 271,276 ****
--- 271,277 ----
    stripExtension :: GHC.Internal.Base.String -> GHC.Internal.IO.FilePath -> GHC.Internal.Maybe.Maybe GHC.Internal.IO.FilePath
  
  module Distribution.Compat.Graph where
+   pattern CyclicSCC :: forall vertex. [vertex] -> SCC vertex
    type role Graph nominal
    type Graph :: * -> *
    data Graph a = ...
***************
*** 284,290 ****
    type Node :: * -> * -> *
    data Node k a = N a k [k]
    type SCC :: * -> *
!   data SCC vertex = AcyclicSCC vertex | CyclicSCC [vertex]
    broken :: forall a. Graph a -> [(a, [Key a])]
    closure :: forall a. Graph a -> [Key a] -> GHC.Internal.Maybe.Maybe [a]
    cycles :: forall a. Graph a -> [[a]]
--- 285,291 ----
    type Node :: * -> * -> *
    data Node k a = N a k [k]
    type SCC :: * -> *
!   data SCC vertex = AcyclicSCC vertex | NECyclicSCC {-# UNPACK #-}(GHC.Internal.Base.NonEmpty vertex)
    broken :: forall a. Graph a -> [(a, [Key a])]
    closure :: forall a. Graph a -> [Key a] -> GHC.Internal.Maybe.Maybe [a]
    cycles :: forall a. Graph a -> [[a]]
***************
*** 8625,8630 ****
--- 8626,8632 ----
  instance [safe] Cabal-syntax-3.12.1.0:Distribution.Utils.Structured.Structured Cabal-3.12.1.0:Distribution.Verbosity.Internal.VerbosityFlag -- Defined in ‘Cabal-3.12.1.0:Distribution.Verbosity.Internal’
  instance [safe] Cabal-syntax-3.12.1.0:Distribution.Utils.Structured.Structured Cabal-3.12.1.0:Distribution.Verbosity.Internal.VerbosityLevel -- Defined in ‘Cabal-3.12.1.0:Distribution.Verbosity.Internal’
  instance Data.Bifoldable.Bifoldable Data.Map.Internal.Map -- Defined in ‘Data.Map.Internal’
+ instance [safe] Data.Foldable1.Foldable1 Data.Graph.SCC -- Defined in ‘Data.Graph’
  instance Data.Foldable1.Foldable1 Cabal-syntax-3.12.1.0:Distribution.Fields.Field.Field -- Defined in ‘Cabal-syntax-3.12.1.0:Distribution.Fields.Field’
  instance Data.Foldable1.Foldable1 Cabal-syntax-3.12.1.0:Distribution.Fields.Field.FieldLine -- Defined in ‘Cabal-syntax-3.12.1.0:Distribution.Fields.Field’
  instance Data.Foldable1.Foldable1 Cabal-syntax-3.12.1.0:Distribution.Fields.Field.Name -- Defined in ‘Cabal-syntax-3.12.1.0:Distribution.Fields.Field’

diff -c cabal-install-solver/cabal-install-solver-9.10.1.api cabal-install-solver-9.10.1.api
*** cabal-install-solver/cabal-install-solver-9.10.1.api	2024-12-03 20:22:52.610058071 -0500
--- cabal-install-solver-9.10.1.api	2024-12-03 20:52:39.792002104 -0500
***************
*** 1256,1261 ****
--- 1256,1262 ----
  instance forall loc. Distribution.Utils.Structured.Structured loc => Distribution.Utils.Structured.Structured (Distribution.Solver.Types.SolverPackage.SolverPackage loc) -- Defined in ‘Distribution.Solver.Types.SolverPackage’
  instance forall loc. Distribution.Utils.Structured.Structured loc => Distribution.Utils.Structured.Structured (Distribution.Solver.Types.SourcePackage.SourcePackage loc) -- Defined in ‘Distribution.Solver.Types.SourcePackage’
  instance Data.Bifoldable.Bifoldable Data.Map.Internal.Map -- Defined in ‘Data.Map.Internal’
+ instance [safe] Data.Foldable1.Foldable1 Data.Graph.SCC -- Defined in ‘Data.Graph’
  instance forall k. GHC.Classes.Eq k => Data.Functor.Classes.Eq1 (Data.Map.Internal.Map k) -- Defined in ‘Data.Map.Internal’
  instance Data.Functor.Classes.Eq1 Data.Set.Internal.Set -- Defined in ‘Data.Set.Internal’
  instance [safe] Data.Functor.Classes.Eq1 Data.Graph.SCC -- Defined in ‘Data.Graph’

Notes for future me: as of cabal-install-v3.12.1.0 API_PRE_FLAGS must be edited.

API_PRE_FLAGS:=-w ghc-$(API_GHC) --disable-documentation --project-file=cabal.release.project --allow-newer=rere:base --allow-newer=hackage-security:template-haskell

Additionally, as mentioned above the custom-setup at the end of cabal-testsuite/cabal-testsuite.cabal must be updated to use 3.12.* instead of 3.10.*. Both of these are unnecessary with HEAD of 3.12 branch.

[ulysses4ever]

new pattern synonym, which provides backward compatibility to changes to SCC, plus a Foldable1 instance for it

Note PVP:

A package version number SHOULD have the form A.B.C ... if only new bindings, types, classes, non-orphan instances or modules (but see below) were added to the interface, then A.B MAY remain the same but the new C MUST be greater than the old C

https://github.com/haskell/pvp/blob/master/v1.0/pvp-specification.md

[geekosaur]

Yes, but this would be 3.14.2.0 (C increase). The patsyn shouldn't count since it is maintaining the 3.14.1.0 API; the addition of the replacement constructor to SCC with the backward compatibility patsyn should then also be a minor version bump (C), I hope. If not, I'll have to track down what commit added the change and revert it.

[geekosaur]

Oh, this is in a dependency (Data.Graph, used by Cabal-syntax:Distribution.Compat.Graph). Does that mean tracking down a version that doesn't have these, or can we just eat the change? (I had assumed that SCC meant GHC, whoops.) It appears to be part of Backpack support.

[Kleidukos]

(I had assumed that SCC meant GHC, whoops.)

Ah, Strongly Connected Components are not Stack Cost Centres :P

[Kleidukos]

Does that mean tracking down a version that doesn't have these, or can we just eat the change?

The deal with compat modules somewhat eludes me, we need to either ask people who set them in place, or actually ask people who use them (hopefully hackage-search can help us), and ask the end-users for their expectations.

[geekosaur]

Yeh, I was wondering why we'd be looking at that level of detail when we only care about enabling profiling. 😀

[geekosaur]

I'm not sure we even care about the change as such; I bet if we search the codebase it'll still be using the compatibility patsyn. ezyang hasn't made any commits since adding Backpack support, has he? Certainly not since 3.12.

[geekosaur]

Right, nothing uses the new constructor, or even the old one.

[geekosaur]

At this point I am thinking this is simply Not Our Problem; it will depend on the index-state and solver configuration of whoever installs it, and from Cabal-syntax etc. standpoint both versions are identical. Otherwise we have to generate a freeze file before the release and wire it into all the cabal files, just in case one of our transitive dependencies changes.