Skip to content

Conversation

@presidentbeef
Copy link
Contributor

Okay, I think this is probably the third attempt for this patch :)

Previously: Details on RubyForge, another poke on RubyGems, comments on Gavin's patch, Luis' thread on the mailing list.

What this patch does:

$ gem install red_cloth
ERROR:  Could not find a valid gem 'red_cloth' (>= 0) in any repository
        Possible alternatives: RedCloth

$ gem install nokojiri
ERROR:  Could not find a valid gem 'nokojiri' (>= 0) in any repository
        Possible alternatives: nokogiri, nokogirl

$ gem install sinatar-capcha
ERROR:  Could not find a valid gem 'sinatar-capcha' (>= 0) in any repository
        Possible alternatives: sinatra-captcha, sinatra-cache, sinatra-cas, sinatra-auth, sinatra-css, sinatra-path, sinatras-hat

This does not provide suggestions when a version is specified (because I don't know how to tell if a failure is because of a name or version mismatch). There are some arbitrary limitations as described on the RubyForge page.

On the mailing list, James Tucker had the following questions, either about this patch or Gavin's:

What about casing? Example: ZenTest and zentest - the ideal is to prefer the prior.

Secondary sort by release date? - prefer native gems (of same versions / release dates)?

I am not sure I understand the first question, because I am not aware of any reason why camel-casing would be considered "more" canonical and I don't see a zentest gem:

$ gem install zentest
ERROR:  Could not find a valid gem 'zentest' (>= 0) in any repository
        Possible alternatives: ZenTest

The secondary sorting might be worthwhile, although I am not sure how big of a difference it will make when the suggestions are already sorted by similarity.

Thanks for your consideration.

@qrush
Copy link
Contributor

qrush commented Dec 10, 2010

+1 to this, I'd like to do a second iteration where we provide this logic on Gemcutter's side and sort the list not alphabetically but by how many downloads a gem has.

@zenspider
Copy link
Member

I think gstark wins both on clarity, simplicity, and testing. I'd like to reject this one in favor of gstark's patch.

Also, I totally love qrush's idea of doing it by download count on the server... but that can wait, obviously.

gstark: I don't see a pull request from you at all. Please submit one.

@presidentbeef
Copy link
Contributor Author

Sorry for commenting when it appears a decision has already been made. I would encourage people to actually try out both patches if you have not already.

For example, this patch:

$ gem install Nokogiri
ERROR:  Could not find a valid gem 'Nokogiri' (>= 0) in any repository
        Possible alternatives: nokogiri

gstark's:

$ gem install Nokogiri
ERROR:  Could not find a valid gem 'Nokogiri' (>= 0) in any repository (did you mean Nephila)

This patch:

$ gem install redcloath
ERROR:  Could not find a valid gem 'redcloath' (>= 0) in any repository
        Possible alternatives: RedCloth, BBRedCloth

gstark's:

$ gem install redcloath
ERROR:  Could not find a valid gem 'redcloath' (>= 0) in any repository (did you mean recloner)

This patch:

$ gem install ActiveRecord
ERROR:  Could not find a valid gem 'ActiveRecord' (>= 0) in any repository
        Possible alternatives: activerecord

gstark's:

$ gem install ActiveRecord
ERROR:  Could not find a valid gem 'ActiveRecord' (>= 0) in any repository (did you mean ActiveExcel)

This patch:

$ gem install tails
ERROR:  Could not find a valid gem 'tails' (>= 0) in any repository
        Possible alternatives: rails, trails

gstark's:

$ gem install tails
ERROR:  Could not find a valid gem 'tails' (>= 0) in any repository (did you mean trails)

This patch:

$ gem install zentest
ERROR:  Could not find a valid gem 'zentest' (>= 0) in any repository
        Possible alternatives: ZenTest

gstark's:

$ gem install zentest
ERROR:  Could not find a valid gem 'zentest' (>= 0) in any repository (did you mean zenweb

@zenspider
Copy link
Member

I agree that you currently have better output than gstark's. He still wins for clarity, simplicity and tests. I think some compromise can be met. I'm working on both patches atm to see what can be done.

P.S. tabs... really?

@zenspider
Copy link
Member

I've got both sets of code in my repo with ENV guards around both so I can switch back and forth. I've massaged both sets of code a bit (like, I like your output better, but it was going to stdout, not stderr; so now they're both the same and going to stderr). In almost all cases, gstark's code is faster (esp after adopting some of your shortcutting). It is also tested and easier to read in almost all cases.

I can send you a combined patch if you'd like to play with it. I'll save one off now just in case. I'm going to reduce this now to a single changeset and push it up soon.

% for WORD in ActiveRecord Nokogiri redcloath tails zentest ; do GSTARK=1 time ruby -Ilib bin/gem install $WORD; BEEF=1 time ruby -Ilib bin/gem install $WORD; done &> compare.txt

ERROR:  Could not find a valid gem 'ActiveRecord' (>= 0) in any repository
ERROR:  Possible alternatives: activerecord
        1.41 real         0.67 user         0.06 sys
ERROR:  Could not find a valid gem 'ActiveRecord' (>= 0) in any repository
ERROR:  Possible alternatives: activerecord
        1.11 real         0.69 user         0.06 sys
ERROR:  Could not find a valid gem 'Nokogiri' (>= 0) in any repository
ERROR:  Possible alternatives: nokogiri
        2.73 real         2.31 user         0.06 sys
ERROR:  Could not find a valid gem 'Nokogiri' (>= 0) in any repository
ERROR:  Possible alternatives: nokogiri
        3.83 real         3.38 user         0.07 sys
ERROR:  Could not find a valid gem 'redcloath' (>= 0) in any repository
ERROR:  Possible alternatives: RedCloth, BBRedCloth
        4.95 real         4.49 user         0.07 sys
ERROR:  Could not find a valid gem 'redcloath' (>= 0) in any repository
ERROR:  Possible alternatives: RedCloth, BBRedCloth
        7.77 real         7.29 user         0.11 sys
ERROR:  Could not find a valid gem 'tails' (>= 0) in any repository
ERROR:  Possible alternatives: rails, trails
        1.81 real         1.37 user         0.07 sys
ERROR:  Could not find a valid gem 'tails' (>= 0) in any repository
ERROR:  Possible alternatives: rails, trails
        2.61 real         1.79 user         0.07 sys
ERROR:  Could not find a valid gem 'zentest' (>= 0) in any repository
ERROR:  Possible alternatives: ZenTest
        3.06 real         2.62 user         0.07 sys
ERROR:  Could not find a valid gem 'zentest' (>= 0) in any repository
ERROR:  Possible alternatives: ZenTest
        4.67 real         4.20 user         0.08 sys

@zenspider
Copy link
Member

pushed. please let me know what you think.

@presidentbeef
Copy link
Contributor Author

Thanks for reconsidering (and sorry about the formatting).

It looks like this produces equivalent results, but faster, so that's good.

I should have included a test case with a dash or underscore in the name, which is really common (looks like 4,626 gems with an underscore, 5,745 with a dash). Clearly, removing dashes and underscores adds overhead. In cases where it allows the matching to short circuit (e.g., "red_cloth" or "mimetypes"), it will be faster. Otherwise, slower. On the other hand, in my opinion, it (slightly) increases the user experience by returning more precise results.

I've added my changes here and can submit as a pull request if desired: https://github.com/presidentbeef/rubygems/tree/remove_underscores_on_suggest

Examples:

Remove underscores/dashes:

$ time -p gem install zentest
ERROR:  Could not find a valid gem 'zentest' (>= 0) in any repository
ERROR:  Possible alternatives: ZenTest
real 5.57  user 4.32  sys 0.06

$ time -p gem install zen_test
ERROR:  Could not find a valid gem 'zen_test' (>= 0) in any repository
ERROR:  Possible alternatives: ZenTest
real 4.85  user 4.28  sys 0.06

$ time -p gem install red_cloth
ERROR:  Could not find a valid gem 'red_cloth' (>= 0) in any repository
ERROR:  Possible alternatives: RedCloth
real 6.19  user 4.32  sys 0.07

$ time -p gem install mimetypes
ERROR:  Could not find a valid gem 'mimetypes' (>= 0) in any repository
ERROR:  Possible alternatives: mime-types
real 4.31  user 3.83  sys 0.06

Don't remove:

$ time -p gem install zentest
ERROR:  Could not find a valid gem 'zentest' (>= 0) in any repository
ERROR:  Possible alternatives: ZenTest
real 4.53  user 3.89  sys 0.09

$ time -p gem install zen_test
ERROR:  Could not find a valid gem 'zen_test' (>= 0) in any repository
ERROR:  Possible alternatives: ZenTest, etest, deep_test, detest, contest
real 6.08  user 5.60  sys 0.07

$ time -p gem install red_cloth
ERROR:  Could not find a valid gem 'red_cloth' (>= 0) in any repository
ERROR:  Possible alternatives: RedCloth, mediacloth, BBRedCloth
real 7.15  user 6.63  sys 0.07

$ time -p gem install mimetypes
ERROR:  Could not find a valid gem 'mimetypes' (>= 0) in any repository
ERROR:  Possible alternatives: mime-types, mimetype-fu, dm-types
real 7.08  user 6.60  sys 0.06

@presidentbeef
Copy link
Contributor Author

Sorry, one more example for comparison. This is the time for when a gem name does not match any existing gems, which is probably the worst case scenario.

Remove underscores/dashes:

$ time -p gem install arghfophasdoj
ERROR:  Could not find a valid gem 'arghfophasdoj' (>= 0) in any repository
real 13.08  user 12.48  sys 0.13

Don't remove:

$ time -p gem install arghfophasdoj
ERROR:  Could not find a valid gem 'arghfophasdoj' (>= 0) in any repository
real 13.10  user 12.55  sys 0.08

@luislavena
Copy link
Member

Hello,

Based on the other thread of discussion and recent 1.4.0 release with such functionality, I'm going ahead and close this pull request.

Happy new year!

nobu referenced this pull request in nobu/rubygems Jan 12, 2014
Time was wrongly accept non-numeric objects.
zzak pushed a commit to zzak/rubygems that referenced this pull request May 9, 2014
MSP-Greg referenced this pull request in MSP-Greg/rubygems Dec 27, 2017
MSP-Greg referenced this pull request in MSP-Greg/rubygems Dec 27, 2017
MSP-Greg referenced this pull request in MSP-Greg/rubygems Dec 27, 2017
MSP-Greg referenced this pull request in MSP-Greg/rubygems Dec 27, 2017
MSP-Greg referenced this pull request in MSP-Greg/rubygems Dec 28, 2017
deivid-rodriguez added a commit that referenced this pull request Feb 27, 2021
Sometimes, as a result of parallel installation of gems reading &
writing gemspecs concurrently during `bundle update`, some thread can
end up printing warnings like the following:

```
[/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead).
```

This commit fixes the issue by completely overriding rubygems installer
and removing all the unnecessary stuff. In particular, the
`Gem::Specification.reset` call that was constantly invalidating gem
specification caches and thus mutating shared state.

As a results, `bundle install` time is noticiably faster. A quick test
on my system (with ~500 gems installed) on an app with about ~100 locked
gems (to avoid resolution overhead) without extensions (to avoid
extension compilation overhead), with a fully loaded gem cache (to avoid
network overhead), it results in ~15% speed up in `bundle install` time:

```
$ hyperfine -p 'rm -rf vendor/bundle' 'bundle install' 'bundle _2.2.11_ install'
Benchmark #1: bundle install
  Time (mean ± σ):      5.713 s ±  0.387 s    [User: 5.215 s, System: 2.163 s]
  Range (min … max):    5.443 s …  6.610 s    10 runs

Benchmark #2: bundle _2.2.11_ install
  Time (mean ± σ):      6.549 s ±  0.106 s    [User: 6.087 s, System: 2.353 s]
  Range (min … max):    6.412 s …  6.773 s    10 runs

Summary
  'bundle install' ran
    1.15 ± 0.08 times faster than 'bundle _2.2.11_ install'
```
deivid-rodriguez added a commit that referenced this pull request Feb 27, 2021
Sometimes, as a result of parallel installation of gems reading &
writing gemspecs concurrently during `bundle update`, some thread can
end up printing warnings like the following:

```
[/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead).
```

This commit fixes the issue by completely overriding rubygems installer
and removing all the unnecessary stuff. In particular, the
`Gem::Specification.reset` call that was constantly invalidating gem
specification caches and thus mutating shared state.

As a results, `bundle install` time is noticiably faster. A quick test
on my system (with ~500 gems installed) on an app with about ~100 locked
gems (to avoid resolution overhead) without extensions (to avoid
extension compilation overhead), with a fully loaded gem cache (to avoid
network overhead), it results in ~15% speed up in `bundle install` time:

```
$ hyperfine -p 'rm -rf vendor/bundle' 'bundle install' 'bundle _2.2.11_ install'
Benchmark #1: bundle install
  Time (mean ± σ):      5.713 s ±  0.387 s    [User: 5.215 s, System: 2.163 s]
  Range (min … max):    5.443 s …  6.610 s    10 runs

Benchmark #2: bundle _2.2.11_ install
  Time (mean ± σ):      6.549 s ±  0.106 s    [User: 6.087 s, System: 2.353 s]
  Range (min … max):    6.412 s …  6.773 s    10 runs

Summary
  'bundle install' ran
    1.15 ± 0.08 times faster than 'bundle _2.2.11_ install'
```
deivid-rodriguez added a commit that referenced this pull request Feb 27, 2021
Sometimes, as a result of parallel installation of gems reading &
writing gemspecs concurrently during `bundle update`, some thread can
end up printing warnings like the following:

```
[/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead).
```

This commit fixes the issue by completely overriding rubygems installer
and removing all the unnecessary stuff. In particular, the
`Gem::Specification.reset` call that was constantly invalidating gem
specification caches and thus mutating shared state has been moved to
the main thread, after after installation of all gems is finished.

As a results, `bundle install` time is noticiably faster. A quick test
on my system (with ~500 gems installed) on an app with about ~100 locked
gems (to avoid resolution overhead) without extensions (to avoid
extension compilation overhead), with a fully loaded gem cache (to avoid
network overhead), it results in ~15% speed up in `bundle install` time:

```
$ hyperfine -p 'rm -rf vendor/bundle' 'bundle install' 'bundle _2.2.11_ install'
Benchmark #1: bundle install
  Time (mean ± σ):      5.713 s ±  0.387 s    [User: 5.215 s, System: 2.163 s]
  Range (min … max):    5.443 s …  6.610 s    10 runs

Benchmark #2: bundle _2.2.11_ install
  Time (mean ± σ):      6.549 s ±  0.106 s    [User: 6.087 s, System: 2.353 s]
  Range (min … max):    6.412 s …  6.773 s    10 runs

Summary
  'bundle install' ran
    1.15 ± 0.08 times faster than 'bundle _2.2.11_ install'
```
deivid-rodriguez added a commit that referenced this pull request Feb 27, 2021
Sometimes, as a result of parallel installation of gems reading &
writing gemspecs concurrently during `bundle update`, some thread can
end up printing warnings like the following:

```
[/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead).
```

The fix is to completely overriding rubygems installer to get rid of the
places where it accessed and modified shared gem specification cache. In
particular:

* Stop calling `Gem::Specification.reset` after each gem installation and
  instead move it the main thread, after after installation of all gems
  has finished.

* Stop using rubygems implementation of
  `Gem::Specification.latest_spec_for(name)` in favor of
  `Gem::Specification.stubs_for(name).first`, which does the same thing but
  without traversing and potentially loading all specifications.

* Remove all the other code that's not actually necessary for bundler.

Technically only the first point would be necessary to fix the race
condition but I figured since I needed to overwrite the method I would
only include what's needed, and make it faster.

As a result, `bundle install` time is noticiably faster. A quick test on
my system (with ~500 gems installed) on an app with about ~100 locked
gems (to avoid resolution overhead) without extensions (to avoid
extension compilation overhead), with a fully loaded gem cache (to avoid
network overhead), it results in ~15% speed up in `bundle install` time:

```
$ hyperfine -p 'rm -rf vendor/bundle' 'bundle install' 'bundle _2.2.11_ install'
Benchmark #1: bundle install
  Time (mean ± σ):      5.713 s ±  0.387 s    [User: 5.215 s, System: 2.163 s]
  Range (min … max):    5.443 s …  6.610 s    10 runs

Benchmark #2: bundle _2.2.11_ install
  Time (mean ± σ):      6.549 s ±  0.106 s    [User: 6.087 s, System: 2.353 s]
  Range (min … max):    6.412 s …  6.773 s    10 runs

Summary
  'bundle install' ran
    1.15 ± 0.08 times faster than 'bundle _2.2.11_ install'
```
composerinteralia added a commit to composerinteralia/rubygems that referenced this pull request Feb 7, 2023
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.

Something similar came up in ruby#6210,
and this commit would have helped with that case as well
(although we've already handled it a different way with ruby#6219).

Details:
---

Back on Bundler 2.2.23, a corrupt lockfile would cause a helpful error:

```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```

Bundler 2.3.26 gave a helpful warning:

```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```

But then continued on and crashed while trying to report the unmet
dependency:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'

...
```

Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
   rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0

* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0

* #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0

<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'

...
```

Changes
---

This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue a
spec and treating that as a failure.

That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
composerinteralia added a commit to composerinteralia/rubygems that referenced this pull request Feb 7, 2023
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.

Something similar came up in ruby#6210,
and this commit would have helped with that case as well
(although we've already handled it a different way with ruby#6219).

Details:
---

Back on Bundler 2.2.23, a corrupt lockfile would cause a helpful error:

```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```

Bundler 2.3.26 gave a helpful warning:

```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```

But then continued on and crashed while trying to report the unmet
dependency:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'

...
```

Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
   rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0

* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0

* #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0

<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'

...
```

Changes
---

This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue a
spec and treating that as a failure.

That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
composerinteralia added a commit to composerinteralia/rubygems that referenced this pull request Feb 7, 2023
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.

Something similar came up in ruby#6210,
and this commit would have helped with that case as well
(although we've already handled this a different way with ruby#6219).

Details:
---

Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error:

```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```

Bundler 2.3.26 gave a helpful warning:

```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```

But then continued on and crashed while trying to report the unmet
dependency:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'

...
```

Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
   rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0

* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0

* #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0

<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'

...
```

Changes
---

This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue a
spec and treating that as a failure.

That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
composerinteralia added a commit to composerinteralia/rubygems that referenced this pull request Feb 7, 2023
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.

Something similar came up in ruby#6210,
and this commit would have helped with that case as well
(although we've already handled this a different way with ruby#6219).

Details:
---

Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error:

```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```

Bundler 2.3.26 gave a helpful warning:

```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```

But then continued on and crashed while trying to report the unmet
dependency:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'

...
```

Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
   rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0

* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0

* #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0

<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'

...
```

Changes
---

This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue. When
that happens we treat it as a failure so the install can finish.

That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
composerinteralia added a commit to composerinteralia/rubygems that referenced this pull request Feb 7, 2023
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.

Something similar came up in ruby#6210,
and this commit would have helped with that case as well
(although we've already handled this a different way with ruby#6219).

Details:
---

Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error:

```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```

Bundler 2.3.26 gave a helpful warning:

```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```

But then continued on and crashed while trying to report the unmet
dependency:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'

...
```

Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
   rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0

* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0

* #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0

<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'

...
```

Changes
---

This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue. When
that happens we treat it as a failure so the install can finish.

That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
composerinteralia added a commit to composerinteralia/rubygems that referenced this pull request Feb 7, 2023
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.

Something similar came up in ruby#6210,
and this commit would have helped with that case as well
(although we've already handled this a different way with ruby#6219).

Details:
---

Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error:

```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```

Bundler 2.3.26 gave a helpful warning:

```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```

But then continued on and crashed while trying to report the unmet
dependency:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'

...
```

Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
   rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0

* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0

* #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0

<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'

...
```

Changes
---

This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue. When
that happens we treat it as a failure so the install can finish.

That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
composerinteralia added a commit to composerinteralia/rubygems that referenced this pull request Feb 8, 2023
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.

Something similar came up in ruby#6210,
and this commit would have helped with that case as well
(although we've already handled this a different way with ruby#6219).

Details:
---

Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error:

```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```

Bundler 2.3.26 gave a helpful warning:

```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```

But then continued on and crashed while trying to report the unmet
dependency:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'

...
```

Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
   rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0

* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0

* #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0

<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'

...
```

Changes
---

This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue. When
that happens we treat it as a failure so the install can finish.

That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
composerinteralia added a commit to composerinteralia/rubygems that referenced this pull request Feb 8, 2023
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.

Something similar came up in ruby#6210,
and this commit would have helped with that case as well
(although we've already handled this a different way with ruby#6219).

Details:
---

Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error:

```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```

Bundler 2.3.26 gave a helpful warning:

```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```

But then continued on and crashed while trying to report the unmet
dependency:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'

...
```

Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
   rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0

* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0

* #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0

<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'

...
```

Changes
---

This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue. When
that happens we treat it as a failure so the install can finish.

That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
composerinteralia added a commit to composerinteralia/rubygems that referenced this pull request Feb 8, 2023
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.

Something similar came up in ruby#6210,
and this commit would have helped with that case as well
(although we've already handled this a different way with ruby#6219).

Details:
---

Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error:

```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```

Bundler 2.3.26 gave a helpful warning:

```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```

But then continued on and crashed while trying to report the unmet
dependency:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'

...
```

Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
   rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0

* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0

* #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0

<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'

...
```

Changes
---

This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue. When
that happens we treat it as a failure so the install can finish.

That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
composerinteralia added a commit to composerinteralia/rubygems that referenced this pull request Feb 8, 2023
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.

Something similar came up in ruby#6210,
and this commit would have helped with that case as well
(although we've already handled this a different way with ruby#6219).

Details:
---

Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error:

```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```

Bundler 2.3.26 gave a helpful warning:

```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```

But then continued on and crashed while trying to report the unmet
dependency:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'

...
```

Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
   rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0

* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0

* #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0

<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'

...
```

Changes
---

This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue. When
that happens we treat it as a failure so the install can finish.

That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants