Discussion: Ensure that projects are rebuild if upstream decides to retag

[leonklingele]

Retags, unfortunately, are a thing and happen from time to time.
For those unaware of the issue: Tags of projects tracked by Homebrew sometimes are pushed to their upstream repository, removed shortly after, and are then re-pushed in a slightly different form (e.g. to include supposedly important fixes). Most likely a lot of those "emergency" retags go without notice, yet it happens that Homebrew maintainers are quick enough to spot the initial tag, issuing a new "Homebrew release" of that project. As Homebrew formulae include the checksum of the project's release tarball and this checksum changes after a retag, installations from source via Homebrew will yield a checksum mismatch error. The checksum of those formula will eventually be fixed, making source-builds succeed again.

It so happens that Homebrew end users can be quick enough to build the initial release tag (by running brew upgrade before the retag was issued). Those users will continue to use a build made using the outdated release if no revision bump was made. Revisions of Homebrew formulae are only issued if the retag appears to be changed in a significant way (e.g. if changes to the source code are made). Yet source code changes are not the only reason why a rebuild might be required.

Take #25845 as an example where libjpeg retagged their most recent v9c release, apparently only removing a seemingly unimportant, empty file (jpeg-9c/.directory) which couldn't possibly result in a different build, therefore an increase of revision was not made. Well, this empty file could actually cause a completely different build if the build process somehow depends on it. How was it verified that this is not the case? Did the libjpeg authors bless the retag? A revision bump is available in #34767.

Here is a list of some other checksum fixes which did not receive a revision bump:

• Fix stable checksums #18048 – the checksums of 94 formulas were fixed. Most of the mismatches are likely caused by GitHub applying a bugfix to their internal Git version. It remains unknown whether some of those projects retagged a release before the issue was introduced by GitHub. There's no easy way to find out if the new tarballs ship with an important fix.
• expect: update checksum #23852, the initial release misses a part of the changelog. (How) was it verified that this changelog file doesn't affect the build process? A revision bump is available in expect: revision after checksum fix #34766.
• bower: update checksum #31767 – no information is available on what actually changed inside the release tarball. A revision bump is available in bower: revision after checksum fix #34768.
• exiv2: update checksum #15663 – project maintainer assures that no code changes have been made to the modified tar.gz files, yet this doesn't imply that the build itself will not be a different.
• dante: update checksum #31790 – although the project's author confirmed that no code but only a NEWS file was changed, he didn't assert that this file is not used inside the build process, possibly causing a different build to be made. A revision bump is available in dante: revision after checksum fix #34769.
• xqilla: update checksum #32042 – some Windows build scripts were altered. Does this affect builds on other platforms? If not, how was this verified? A revision bump is available in xqilla: revision after checksum fix #34770.
• nuttcp: fix checksum #32043 – changes to the README. While this file most likely is not used inside the build process, how was this verified? A revision bump is available in nuttcp: revision after checksum fix #34771.
• cracklib: update patch checksum #31948 – apparently a patch's checksum changed, although the patch itself wasn't. Does this possibly affect the build process in any way? A revision bump is available in cracklib: bump revision after checksum fix #34773.
• git-secret: adjust checksum #32060 – no information available on what changed. A revision bump is available in git-secret: revision after checksum fix #34772.

Humans, by their very nature, make mistakes. As it is easy to not rely on humans to decide whether forcing a rebuild is sensible or not and most formulae don't take hours to build, I ask to simply always enforce rebuilds (this includes bottles as well) by revision bumping formulas after any of their included checksums change for whatever reason.

[fxcoudert]

First, retags should not happen in projects following semantic versioning (version numbers should uniquely identify a given state of the source and its distribution). Moreover, when they happen it is because upstream deemed it unnecessary to issue a new release and force all users to update. In other words: a retag is, by its very definition, not a new version but the same version, i.e. the same state of the software.

Our current policy is the following:

1. The change of checksum with identical version number is treated as suspicious, since it could be malicious (someone took control of the repository or file server, and switched the sources)
2. Once we confirm that the change in sources is intentional (usually by contacting upstream and asking them), we fix the checksum in Homebrew
3. The maintainer merging the retag then makes an informed decision about the necessity for pulling new bottles and bumping revision. Pulling new bottles cannot really hurt, but bumping revision has a cost: it will make all users upgrade, i.e. download new bottles or rebuild from source. So:
   • Unless there was something seriously wrong with the previous build, we do not bump the revision number.
   • We do, however, bump the revision number if the previous software was broken in significant way: for example, if the retag is actually a hot fix, because the maintainer realised they shipped a buggy tarball.
4. We advise all upstream projects to stick to semantic versioning, and rather than retag, change the version number in a minor way to indicate the bug fix, when one is made.

---

My opinion is that the policy above is close to optimal, as it minimises the discomfort to our users while taking all elements into account before making a decision to bump revision or not.

If this decision was incorrectly made in some cases in the past, this should be fixed and specific pull requests addressing the issue are welcome (including details on what is wrong).

[MikeMcQuaid]

Agreed with @fxcoudert. Closing but feel free to continue discussion here.

[leonklingele]

This just feels wrong to me. A package manager clearly should never — under no circumstance — treat two different releases as being the same, giving a false sense of security. I can guarantee you that this will eventually hit someone sooner or later. Even if there is a negligible probability that there will be any problems due to this, why not choose a way which will not? Sure, rebuilding sucks, but end users shouldn't blame Homebrew for that. It's the software developer who made a mistake.

What do you think of adding support for a new env var, e.g. HOMEBREW_REBUILD_RETAGS, which will enforce software rebuilds on retags? Similar to revision [NUM] there could be a retag [NUM].

And what about the cases I've mentioned in the initial report? What about those 94 formulae?

[SMillerDev]

The git tarbal issue weren't retags as is stated in the linked issue. A retag is the same release though, 6.0.0 is the exact same as 6.0.0. If we can't trust that to be the case we're better off giving up on versioning altogether and just building the software from the latest upstream locally on install.

[leonklingele]

The git tarbal issue weren't retags as is stated in the linked issue.

What makes you so sure about that? Have all the tarballs been checked for diffs? Just because GitHub had some issues which caused different tarballs to be available doesn't mean those projects didn't retag just minutes before / after that. You see, I'm just speculating here. But given that 94 formulae were updated it's not so unlikely that a retag was amongst them.

[SMillerDev]

Issue #18044 which is referenced in the PR you linked mentions the checks people did to make sure there weren't any retags in there.

[MikeMcQuaid]

Sure, rebuilding sucks, but end users shouldn't blame Homebrew for that.

Regardless of whether they should or will: it makes the experience of using Homebrew worse for those users for no good reason so we're not going to do it.

What do you think of adding support for a new env var, e.g. HOMEBREW_REBUILD_RETAGS, which will enforce software rebuilds on retags? Similar to revision [NUM] there could be a retag [NUM].

I would want to see many more users (and maintainers for whom the work will fall to keep up with that) asking for something like this before it would be even considered.

What makes you so sure about that?

Because: we're running a package manager, do a lot of work behind that scenes that's not always make public, we have two GitHub employees as maintainers and many other reasons.

I appreciate your contributions to Homebrew but I don't like your tone in this issue. Please address it in future issues to sound less like you are trying to teach people who are successfully running a package manager how to do so. Issues like this based on speculation are not helpful. Contributions to Homebrew are useful and if you contribute enough you may be able to become a maintainer in future and have influence over how the project is run.