formulary: migrate fully-scoped formulae names. #1371

MikeMcQuaid · 2016-10-24T18:14:19Z

Have you followed the guidelines in our Contributing document?
Have you checked to ensure there aren't other open Pull Requests for the same change?
Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your changes? Here's an example.
Have you successfully run brew tests with your changes locally?

e.g. allow brew install some/tap/formula to look that formula up in
another tap if it has been migrated.

CC @ilovezfs @tdsmith this will mean we can migrate formulae from taps and have e.g. brew install homebrew/python/pillow not go 💥

ilovezfs · 2016-10-24T23:45:23Z

Is this going to mean keeping dead/empty (or mostly dead/empty) taps around "forever" as a registry, and then causing it to break if/when the repo is ever removed?

If so, let's just put the registry/registries in brew itself in a structured file/files, not rely on zombie repositories to serve as alias databases.

zmwangx · 2016-10-25T01:08:50Z

let's just put the registry/registries in brew itself in a structured file/files

Don't forget tap migrations is a feature that's available to third party taps as well.

I would suggest keeping the info in the destination tap as well, so that if the source tap is unavailable, we can still track down the destination if the destination tap is tapped. This at the very least solves the third-party-tap-incorporated-into-core problem.

MikeMcQuaid · 2016-10-25T02:34:21Z

Is this going to mean keeping dead/empty (or mostly dead/empty) taps around "forever" as a registry, and then causing it to break if/when the repo is ever removed?

In the short-term: yes. In the long-term: no.

Don't forget tap migrations is a feature that's available to third party taps as well.

Indeed.

ilovezfs · 2016-10-25T03:35:31Z

I would suggest keeping the info in the destination tap as well

I agree with this. Relying on the original taps, even in the short-run, seems like a mistake when the needed information is available and can be kept in brew or core.

MikeMcQuaid · 2016-10-25T11:40:34Z

Relying on the original taps, even in the short-run, seems like a mistake when the needed information is available and can be kept in brew or core.

We will need both approaches for the cases where a migration happens between a tap that's tapped and one that is not.

ilovezfs · 2016-10-25T11:47:54Z

If pillow moves to homebrew/core,

depends_on "homebrew/python/pillow"

should not require tapping homebrew/python to figure out that it should use the homebrew/core pillow.

MikeMcQuaid · 2016-10-25T12:09:17Z

If some random, non Homebrew-org tap wants to move a formula from one tap to another it shouldn't require us to make a change in homebrew-core or Homebrew/brew so this change will still be needed regardless. Even when moving things from taps to homebrew/core I don't know how I feel about maintaining migration lists in multiple places.

Something that's not here yet but will be added to this PR is a user-facing deprecation warning to fix up the dependency/Brewfile/install line accordingly.

ilovezfs · 2016-10-25T12:16:13Z

If some random, non Homebrew-org tap wants to move a formula from one tap to another it shouldn't require us to make a change in homebrew-core or Homebrew/brew so this change will still be needed regardless.

That's not the use case we currently face, and supporting insecure redirects from third parties to fourth parties is not exactly a high priority.

e.g. allow `brew install some/tap/formula` to look that formula up in another tap if it has been migrated. Also, add a deprecation message to point people towards the correct naming.

ilovezfs · 2016-11-06T14:50:48Z

Um, ok ...

MikeMcQuaid · 2016-11-06T14:57:13Z

@ilovezfs This is an improvement as-is and we're going to need this even if we additionally decide to have these migrations live in e.g. homebrew/core.

MikeMcQuaid · 2016-11-06T14:57:57Z

@ilovezfs If/when we do kill homebrew/versions I'll take the approach you've mentioned where this logic will live in homebrew/core.

ilovezfs · 2016-11-06T15:11:03Z

I just don't like the precedent of "silent" merges when the conversation was left as it was above.

MikeMcQuaid · 2016-11-06T15:14:18Z

Apologies, I thought the conversation had reached a conclusion in that we'd disagreed on the short-term approach but agreed on the long-term one. This is ready to go and working so I don't think it makes sense to block this (already pretty delayed) work on the long-term version.

ilovezfs · 2016-11-06T15:17:00Z

Yup, agreed. Just like things to end with 😸 not 😡

No apologies needed.

ilovezfs · 2016-12-30T02:06:43Z

It looks like this isn't able to handle the homebrew/versions/glfw3 -> homebrew/core/glfw migration: https://bot.brew.sh/job/Homebrew%20Testing/1219/version=el_capitan/console

zmwangx · 2016-12-30T02:10:06Z

Migration and rename shouldn't be mutually exclusive.

ilovezfs · 2016-12-30T02:17:12Z

The underlying entries look correct:

https://github.com/Homebrew/homebrew-versions/blob/2ca6b5e2eb4d6fdd0457bf8a9ecfa0917b4d0923/tap_migrations.json#L5

https://github.com/Homebrew/homebrew-core/blob/12401996b6d851a5cf573f770cb894c96e8d858b/formula_renames.json#L18

https://github.com/Homebrew/homebrew-science/blob/34eb0d2bd590b0eb50720161365d7ee874b58a46/arrayfire.rb#L34

zmwangx · 2016-12-30T02:19:22Z

Yeah, the migration & rename entries are correct, but migration and rename are handled as mutually exclusive cases in the code: https://github.com/Homebrew/brew/blob/c3a2bf3/Library/Homebrew/formulary.rb#L182-L197.

ilovezfs · 2016-12-30T02:26:38Z

@zmwangx I think the bug may be specific to test-bot though

iMac-TMP:~ joe$ brew install -s arrayfire
==> Installing arrayfire from homebrew/science
==> Tapping homebrew/versions
Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-versions'...
remote: Counting objects: 210, done.
remote: Compressing objects: 100% (207/207), done.
remote: Total 210 (delta 17), reused 33 (delta 2), pack-reused 0
Receiving objects: 100% (210/210), 246.21 KiB | 264.00 KiB/s, done.
Resolving deltas: 100% (17/17), done.
Tapped 201 formulae (231 files, 898.6K)
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
==> Installing dependencies for homebrew/science/arrayfire: boost-compute, clblas, clfft, homebrew/versions/glfw3
==> Installing homebrew/science/arrayfire dependency: boost-compute
==> Downloading https://homebrew.bintray.com/bottles-science/boost-compute-1.62.0.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring boost-compute-1.62.0.el_capitan.bottle.tar.gz
🍺  /usr/local/Cellar/boost-compute/1.62.0: 318 files, 1.3M
==> Installing homebrew/science/arrayfire dependency: clblas
==> Downloading https://homebrew.bintray.com/bottles-science/clblas-2.6.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring clblas-2.6.el_capitan.bottle.tar.gz
🍺  /usr/local/Cellar/clblas/2.6: 20 files, 4M
==> Installing homebrew/science/arrayfire dependency: clfft
==> Downloading https://homebrew.bintray.com/bottles-science/clfft-2.12.1.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring clfft-2.12.1.el_capitan.bottle.tar.gz
🍺  /usr/local/Cellar/clfft/2.12.1: 23 files, 1.1M
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
==> Installing homebrew/science/arrayfire dependency: homebrew/versions/glfw3
==> Downloading https://homebrew.bintray.com/bottles/glfw-3.2.1.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring glfw-3.2.1.el_capitan.bottle.tar.gz
🍺  /usr/local/Cellar/glfw/3.2.1: 14 files, 286.5K
==> Installing homebrew/science/arrayfire 
==> Using the sandbox
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
Warning: Use glfw instead of deprecated homebrew/versions/glfw3
==> Cloning https://github.com/arrayfire/arrayfire.git
Updating /Users/joe/Library/Caches/Homebrew/arrayfire--git
==> Checking out tag v3.3.2
==> Downloading https://downloads.sourceforge.net/project/glew/glew/1.13.0/glew-1.13.0.tgz
Already downloaded: /Users/joe/Library/Caches/Homebrew/arrayfire--glew1-1.13.0.tgz
==> make GLEW_PREFIX=/usr/local/Cellar/arrayfire/3.3.2/libexec/glew1 GLEW_DEST=/usr/local/Cellar/arrayfire/3.3.2/libexec/glew1 all

zmwangx · 2016-12-30T02:28:59Z

Hmm, okay, isn't immediately obvious from the code... Good thing it's not user facing 🤔

ilovezfs · 2016-12-30T02:35:12Z

@zmwangx I think it's the new aggressive untapping behavior which is causing test-bot to shoot itself in the foot:

==> brew uses --recursive homebrew/science/arrayfire
==> Tapping homebrew/versions
Cloning into '/usr/local/Homebrew/Library/Taps/homebrew/homebrew-versions'...
Tapped 201 formulae (230 files, 3M)
==> git checkout master -f
==> brew cleanup --prune=7
==> git clean -ffdx --exclude=Library/Taps
Untapping homebrew/versions... (230 files, 3M)
Untapped 201 formulae
Already on 'master'
HEAD is now at 4ca2eaf Merge pull request #1682 from MikeMcQuaid/tap_migrations_rename
Already on 'master'
Your branch is up-to-date with 'origin/master'.
HEAD is now at 9d524e4 thrift: update 0.9.3 bottle.
Already on 'master'
Your branch is up-to-date with 'origin/master'.
HEAD is now at 121855e Merge pull request #46 from MikeMcQuaid/cleanup-top-level-dirs
Error: No available formula with the name "homebrew/versions/glfw3" 
Please tap it and then try again: brew tap homebrew/versions
Build step 'Execute shell' marked build as failure

MikeMcQuaid · 2016-12-30T17:32:45Z

Yeah, the migration & rename entries are correct, but migration and rename are handled as mutually exclusive cases in the code:

That's not correct, the migration case calls back into formula_name_path specifically to handle migrations and renames.

Partial implementation of https://github.com/Homebrew/brew-evolution/pull/15, along with the ability to search for deleted formulae in git history (inspired by Homebrew#1996) which is not described in the proposal. See also: Homebrew#1371.

MikeMcQuaid added the in progress Maintainers are working on this label Oct 24, 2016

formulary: migrate fully-scoped formulae names.

133bad9

e.g. allow `brew install some/tap/formula` to look that formula up in another tap if it has been migrated. Also, add a deprecation message to point people towards the correct naming.

MikeMcQuaid mentioned this pull request Nov 2, 2016

elasticsearch17: migrate from homebrew/versions. Homebrew/homebrew-core#6533

Merged

4 tasks

MikeMcQuaid merged commit ce2b11f into Homebrew:master Nov 6, 2016

MikeMcQuaid deleted the tap-migrate-fully-scoped branch November 6, 2016 13:22

MikeMcQuaid removed the in progress Maintainers are working on this label Nov 6, 2016

MikeMcQuaid mentioned this pull request Nov 7, 2016

Deprecating old taps in favour of core repository Homebrew/homebrew-core#5909

Closed

DabeDotCom mentioned this pull request Nov 7, 2016

How Do I Fix/Silence "Use new_tap/xxx instead of deprecated old_tap/xxx" Warnings? #1459

Closed

MikeMcQuaid mentioned this pull request Nov 23, 2016

Migrate MySQL and Elasticsearch formulae from homebrew/versions Homebrew/homebrew-core#7181

Closed

4 tasks

zmwangx mentioned this pull request Dec 26, 2016

Hint at new location of migrated formulae #1732

Merged

5 tasks

Homebrew locked and limited conversation to collaborators May 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

formulary: migrate fully-scoped formulae names. #1371

formulary: migrate fully-scoped formulae names. #1371

MikeMcQuaid commented Oct 24, 2016 •

edited

Loading

ilovezfs commented Oct 24, 2016

zmwangx commented Oct 25, 2016

MikeMcQuaid commented Oct 25, 2016

ilovezfs commented Oct 25, 2016

MikeMcQuaid commented Oct 25, 2016

ilovezfs commented Oct 25, 2016

MikeMcQuaid commented Oct 25, 2016

ilovezfs commented Oct 25, 2016

ilovezfs commented Nov 6, 2016

MikeMcQuaid commented Nov 6, 2016

MikeMcQuaid commented Nov 6, 2016

ilovezfs commented Nov 6, 2016

MikeMcQuaid commented Nov 6, 2016

ilovezfs commented Nov 6, 2016

ilovezfs commented Dec 30, 2016

zmwangx commented Dec 30, 2016

ilovezfs commented Dec 30, 2016

zmwangx commented Dec 30, 2016

ilovezfs commented Dec 30, 2016

zmwangx commented Dec 30, 2016

ilovezfs commented Dec 30, 2016

MikeMcQuaid commented Dec 30, 2016

formulary: migrate fully-scoped formulae names. #1371

formulary: migrate fully-scoped formulae names. #1371

Conversation

MikeMcQuaid commented Oct 24, 2016 • edited Loading

ilovezfs commented Oct 24, 2016

zmwangx commented Oct 25, 2016

MikeMcQuaid commented Oct 25, 2016

ilovezfs commented Oct 25, 2016

MikeMcQuaid commented Oct 25, 2016

ilovezfs commented Oct 25, 2016

MikeMcQuaid commented Oct 25, 2016

ilovezfs commented Oct 25, 2016

ilovezfs commented Nov 6, 2016

MikeMcQuaid commented Nov 6, 2016

MikeMcQuaid commented Nov 6, 2016

ilovezfs commented Nov 6, 2016

MikeMcQuaid commented Nov 6, 2016

ilovezfs commented Nov 6, 2016

ilovezfs commented Dec 30, 2016

zmwangx commented Dec 30, 2016

ilovezfs commented Dec 30, 2016

zmwangx commented Dec 30, 2016

ilovezfs commented Dec 30, 2016

zmwangx commented Dec 30, 2016

ilovezfs commented Dec 30, 2016

MikeMcQuaid commented Dec 30, 2016

MikeMcQuaid commented Oct 24, 2016 •

edited

Loading