Distinguish between deprecated and non-preferred versions in the package index

[grayjay]

Hackage shows deprecated and preferred versions on each package page, but it seems to combine the two types of version ranges in the package index. This means that cabal can't distinguish between versions that are deprecated and versions that are only non-preferred. It would be great to add more functionality to cabal related to identifying and avoiding deprecated versions. Here is an example of an issue that could be solved: haskell/cabal#10097

Adding preferred versions to the index:

hackage-server/src/Distribution/Server/Features/PreferredVersions.hs

Lines 344 to 350 in 5b0d3b0

	updateIndexPackagePreferredVersions :: MonadIO m => PackageName -> PreferredInfo -> m ()
	updateIndexPackagePreferredVersions pkgname prefinfo = do
	now <- liftIO getCurrentTime
	let prefEntryName = display pkgname </> "preferred-versions"
	prefContent = fromMaybe "" $
	formatSinglePreferredVersions pkgname prefinfo
	updateArchiveIndexEntry prefEntryName (BS.pack prefContent) now

Formatting deprecated/preferred versions for one package:

hackage-server/src/Distribution/Server/Features/PreferredVersions.hs

Lines 411 to 413 in 5b0d3b0

	formatSinglePreferredVersions :: PackageName -> PreferredInfo -> Maybe String
	formatSinglePreferredVersions pkgname pref =
	display . (\vr -> Dependency pkgname vr mainLibSet) <$> sumRange pref

PreferredInfo, with sumRange field for merged version ranges:

hackage-server/src/Distribution/Server/Features/PreferredVersions/State.hs

Lines 30 to 34 in 5b0d3b0

	data PreferredInfo = PreferredInfo {
	preferredRanges :: [VersionRange],
	deprecatedVersions :: [Version],
	sumRange :: Maybe VersionRange -- cached form of 'consolidateRanges' below
	} deriving (Typeable, Show, Eq)

Merging version ranges:

hackage-server/src/Distribution/Server/Features/PreferredVersions/State.hs

Lines 39 to 44 in 5b0d3b0

	consolidateRanges :: [VersionRange] -> [Version] -> Maybe VersionRange
	consolidateRanges ranges depr =
	let range = simplifyVersionRange $ foldr intersectVersionRanges anyVersion (map notThisVersion depr ++ ranges)
	in if isAnyVersion range || isNoVersion range
	then Nothing
	else Just range

[grayjay]

One idea is to add a new JSON file, such as preferred-versions.json, to eventually replace preferred-versions. The new file would be at the package level, and it would contain both the list of preferred version ranges and the list of deprecated versions for the package.

[phadej]

Contraproposal: remove deprecations (and FWI preferred versions) all-together. Hackage supports metadata revisions, so what problems version deprecations solve? AFAICT, some people are using it to simply mark old versions as old, but cabal-install already prefers newer by default.

[grayjay]

I'm not sure what the use case is for preferred versions, since cabal already prefers the latest, but I thought that the deprecation feature was still useful for marking specific versions as bad (e.g., after a bug is found). How could that be replaced by metadata revisions? Do you mean that we could add a deprecation field to the .cabal file and update the package directly?

[phadej]

@grayjay What semantics do we (you?) want for "bad" versions?

If it's really bad, we can always do base <0 revision, essentially removing the package version from the existence. "hard removal"

The current deprecation is only a "soft removal", solver can still pick these versions.

Why I particularly don't like the "soft removal" is that cabal-install does not notify users that a deprecated version is picked. Wouldn't users want to avoid these versions?

What are situations where we need to keep "quite bad but not really bad" versions around?

There are situations where "soft removal" might be a less bad (than e.g. doing nothing), but I can only think that from Hackage Trustee POV: there might be somewhat bad version, and to fix one the new release as needed; and particular example I have in mind is because revision mechanism is over-conservative: we cannot make revisions with new conditions like if os(windows) build-depends: dependency >= 1.2.3, and doing build-depends: dependency >=1.2.3 would be too restrictive.

I'd rather relax the the revisions checks (the current checks are syntactical, which disallow many reasonable edits)

EDIT: the one point for deprecations is that those are explicit; base <0 (or any other trivial-to-recognize unsatisfiable constraint) are indirect. It might make sense to add more direct way to specify those in .cabal spec; and e.g. include a message, so users would be able to see why the version is retracted from Hackage index. Hackage would disallow uploading packages with such field set.

[gbaz]

I can't remember why preferred were added -- before my time. It was motivated by a very small number of packages where there was some deliberate reason to prefer not-the-latest. It may have predated revisions? I think we could definitely depreciate preferred -- after discussion and a survey of how much it is currently used (probably very little)

I do think we need to keep deprecation, because it is a good "social signal" to end-users, and it lets people "undo" a bad upload in a more visible and straightforward way than a revision.

We could definitely consider switching deprecations to a harder semantics -- i.e. perhaps disable such packages entirely unless exactly that version is specified.

[phadej]

We could definitely consider switching

This is not an easy thing to do anymore. Stackage (and stack separately), cabal-install, nixpkgs (though mostly via stackage, but also directly) are all "interpreters" of the metadata, so any semantical change needs to be long enough and well coordinated process.

I don't see that happening.

---

I do think we need to keep deprecation, because it is a good "social signal" to end-users, and it lets people "undo" a bad upload in a more visible and straightforward way than a revision.

That's an UI problem: #1324

[ulysses4ever]

I see two problems with base<0: (a) it's not very widely known, and (b) it just feels like an ugly hack. For (a), we could add a Hackage warning: when a package author marks a package version as deprecated, Hackage could show an instructive popup saying that they should consider revising base<0 as a more effective tool to prevent this version being used. As for (b), I don't really know what to do about it. I wish "deprecation" somehow implied base<0 but I don't see a clear path there.

[phadej]

(a) it's not very widely known,

As I said above

the one point for deprecations is that those are explicit; base <0 (or any other trivial-to-recognize unsatisfiable constraint) are indirect. It might make sense to add more direct way to specify those in .cabal spec;

Having more direct way to express the intention would make it easier to advertise its usage. (And a base <0 as a variant usable in older cabal-version .cabal files).

EDIT: In particular people incorrectly assume that buildable: False is unsatisfiable constraint; it isn't. it has different semantics (solver doesn't backtrack on it; it's not a constraint) and that is not really explained well anywhere (or at all?)

[ulysses4ever]

@phadej

As I said above

the one point for deprecations is that those are explicit; base <0 (or any other trivial-to-recognize unsatisfiable constraint) are indirect. It might make sense to add more direct way to specify those in .cabal spec;

That's a good idea but a one demanding more work. Fortunately, our ideas are not mutually exclusive. We can start advertizing base<0 more and when the spec receives the new feature you're talking about, it'll be easy to change the suggestion. But I'd start small and implement the easier thing first.

[ulysses4ever]

I opened a feature request for Hackage UI: #1370

[grayjay]

@grayjay What semantics do we (you?) want for "bad" versions?

If it's really bad, we can always do base <0 revision, essentially removing the package version from the existence. "hard removal"

The current deprecation is only a "soft removal", solver can still pick these versions.

Why I particularly don't like the "soft removal" is that cabal-install does not notify users that a deprecated version is picked. Wouldn't users want to avoid these versions?

What are situations where we need to keep "quite bad but not really bad" versions around?

I think that cabal should try harder to avoid deprecated versions. For example, it could avoid deprecated versions unless the user specifies the exact version, as @gbaz suggested. It could also give a warning whenever a deprecated version is used. Here are some other improvements that have been suggested: https://github.com/haskell/cabal/labels/re%3A%20deprecated%20packages%20and%20versions

I opened this issue because I'd like to give cabal access to the lists of deprecated versions so that it is possible to treat them differently.

We could definitely consider switching

This is not an easy thing to do anymore. Stackage (and stack separately), cabal-install, nixpkgs (though mostly via stackage, but also directly) are all "interpreters" of the metadata, so any semantical change needs to be long enough and well coordinated process.

I don't see that happening.

Why would it be difficult to change the way that cabal handles deprecated versions? Trying harder to avoid them seems like moving in the safe direction. The change seems especially safe if there is still a way for a user to force the use of specific deprecated versions.

I prefer improving the existing Hackage deprecation feature over adding a new field to the .cabal spec because of the complexity of either maintaining two ways of marking deprecation or migrating from one to the other.

[grayjay]

I opened #1374 for removing preferred versions.

[yutotakano]

Chiming in as a package maintainer, I wouldn't mind if deprecations as a feature were removed and we have to use base<0 to imply deprecation -- but I'd like some sort of UI for doing this!

Marking versions as deprecated on Hackage is thankfully very easy, just checking boxes for the deprecated versions. But changing metadata revisions for all deprecated releases is time consuming. If you will remove deprecations, I would appreciate it if ideally the same UI (checkboxes) is kept, but under the hood it adds base<0 as revisions or something. It'd also be nice if the versions would still be coloured red and some banner or something shows end users about the deprecation, which Hackage can generate on the fly by checking if base<0 exists as a bound.

Or, of course, none of this hacky workaround would be necessary if cabal avoids and warns against deprecated versions properly, as this thread has previously mentioned.