exe:cabal doesn't properly ignore .cabal files with unsupported `cabal-version:`

[hvr]

This is causing problems for e.g. cabal-install-1.24 (and older versions as well), which fails to parse bleeding edge .cabal files.

So e.g.

cabal-1.24 info base-noprelude-4.10.0.0

succeeds, even though base-noprelude-4.10.0.0 would require cabal-2.0 to be properly decodable.

But there's more harmful cases, like e.g. for

cabal-1.24 info haddock-library
cabal: internal error when reading package index: failed to parse .cabal file
The package index or index cache is probably corrupt. Running cabal update might fix it.

or even cabal-1.24 list which fails as soon as such a .cabal file is encountered.

Or even worse, cause breakages such as in jgm/pandoc#3814 which render cabal-1.24 unusable.

In the past this would have been mitigated by having cabal update remind users to update, e.g.

$ cabal-1.20 update
Downloading the latest package list from hackage.haskell.org
Skipping download: Local and remote files match.
Note: there is a new version of cabal-install available.
To upgrade, run: cabal install cabal-install

How to fix (imho)

Quickfix

• ignore .cabal files in the index we can't parse (i.e. which the parser currently fails with an "internal error")
• possibly, for those .cabal files we can parse but have a too-new cabal-version, ignore them as well

(see also #4624 (comment))

Proper fix

For packages that are newer than cabal-install supports, we can only know the package name & version, and possibly the specified cabal-version. Everything beyond that would require the parser to know how to parse and interpret the .cabal contents (note that semantics may change depending on the cabal-version, so we really cannot know what a .cabal files means if we don't actively support the declared cabal-version).

I've looked at the code in D.C.IndexUtils, and one way to go about this is to do something along the lines of

--- a/cabal-install/Distribution/Client/IndexUtils.hs
+++ b/cabal-install/Distribution/Client/IndexUtils.hs
@@ -56,7 +56,7 @@ import Distribution.Simple.Program
 import qualified Distribution.Simple.Configure as Configure
          ( getInstalledPackages, getInstalledPackagesMonitorFiles )
 import Distribution.ParseUtils
-         ( ParseResult(..) )
+         ( ParseResult(..), PError(FromString) )
 import Distribution.Version
          ( Version(Version), intersectVersionRanges )
 import Distribution.Text
@@ -254,7 +254,7 @@ whenCacheOutOfDate index action = do
 
 -- | An index entry is either a normal package, or a local build tree reference.
 data PackageEntry =
-  NormalPackage  PackageId GenericPackageDescription ByteString BlockNo
+  NormalPackage  PackageId (Maybe GenericPackageDescription) ByteString BlockNo
   | BuildTreeRef BuildTreeRefType
                  PackageId GenericPackageDescription FilePath   BlockNo
 
@@ -541,13 +542,16 @@ packageListFromCache mkPkg hnd Cache{..} mode = accum mempty [] cacheEntries
           -> return content
         _ -> interror "unexpected tar entry type"
 
-    readPackageDescription :: ByteString -> IO GenericPackageDescription
+    readPackageDescription :: ByteString -> IO (Maybe GenericPackageDescription)
     readPackageDescription content =
       case parsePackageDescription . ignoreBOM . fromUTF8 . BS.Char8.unpack $ content of
-        ParseOk _ d -> return d
-        _           -> interror "failed to parse .cabal file"
+        ParseOk _ d -> return (Just $! d)
+        ParseFailed (FromString e _)
+            | "This package requires at least Cabal version" `isPrefixOf` e
+               -> return Nothing
+        _      -> interror "failed to parse .cabal file"

I.e. when a GenericPackageDescription cannot be parsed, turn it into a Nothing (or something with more information, providing more details about the minimum required version).

Then we could have the solver handle "future" packages by placing them into a blacklist
(as if they had an unconditional fail: this package requires a newer cabal version, see #393) - or we could just fake a minimal GenericPackageDescription which consists merely of an empty lib and exe with such a fail directive. Actually, would it work already now if we just faked a dummy GenericPackageDescription with a too-new spec-cabal-version?

Anyway, then the solver would avoid selecting such packages, and if forced to (freeze file, --constraints, or merely cabal install foo-1.2.3), would present the user with an informative error message which points to a possible resolution.

Commands like cabal info or cabal list would have to gracefully skip, ignore, or if forced to (cabal info foo-1.2.3), emit useful messages.

/cc @grayjay

[jgm]

Note that this also seems to affect cabal-1.22 and cabal-1.18, as the pandoc travis builds show.

[hvr]

Indeed. We may require to release hotfixes to a couple of older major versions of cabal-install, so Linux distributions can pick it up.

/cc @23Skidoo

[23Skidoo]

:-(

I'll try to fix this (and the remaining exe:cabal-install-2.0 blockers) during this week.

[hvr]

@23Skidoo here's an alternative & complementary idea: older cabals (i.e. cabal-install-1.22 and older) only know about the 00-index.tar.gz; so we could remove cabal-version:>=2.0 files from the 00-index.tar.gz server-side; so only 01-index.tar.gz would contain .cabal files requiring cabal>=2. I've looked at hackage-server's code, and this requires a bit of work to get implemented...

Then we'd only need a quickfix for cabal-install-1.24, and a proper fix for cabal-install-2.0+ (so we won't run into this when we extend the .cabal format again in 2.2)

[sol]

@RyanGlScott I'm not sure if ekmett/semigroupoids@ef085d1 is a viable solution. As I understand it masks the problem on Travis, but users that use a released version of cabal-install are still affected.

@hvr for now I'm avoiding to run cabal update. Which packages on Hackage cause this issue? Wouldn't we want to blacklist (or revise) those packages for now?

[gallais]

Which packages on Hackage cause this issue?

Yesterday I ran a small script untarring and grepping for these cabal-version: >=2.0 constraints and the packages having them were: haddock-api, haddock-library, haddock, and base-noprelude

[ezyang]

I don't think this is a blocker for 2.0. We can always get the fix which solves cabal-version: >= 2.2 into cabal-install-2.0 into a bugfix release before Cabal-2.2 comes out. In the meantime, not having a cabal-install-2.0 means that it's not possible to work around this bug by installing the latest cabal-install.

[RyanGlScott]

@sol, to be clear, I don't intend for ekmett/semigroupoids@ef085d1 to be a permanent fix. I only committed that to repair Travis breakage after a previous commit I made, and I needed to bandage the wound to save face.

That being said, I've pretty much stopped making commits to most of my Haskell projects until this issue is fixed so as not to break things further.