-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Suggest similar gem names when install fails to find any matches #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Suggest similar gem names when install fails to find any matches #1
Conversation
original patch was submitted here: http://rubyforge.org/tracker/index.php?func=detail&aid=28149&group_id=126&atid=577
|
+1 to this, I'd like to do a second iteration where we provide this logic on Gemcutter's side and sort the list not alphabetically but by how many downloads a gem has. |
|
I think gstark wins both on clarity, simplicity, and testing. I'd like to reject this one in favor of gstark's patch. Also, I totally love qrush's idea of doing it by download count on the server... but that can wait, obviously. gstark: I don't see a pull request from you at all. Please submit one. |
|
Sorry for commenting when it appears a decision has already been made. I would encourage people to actually try out both patches if you have not already. For example, this patch: gstark's: This patch: gstark's: This patch: gstark's: This patch: gstark's: This patch: gstark's: |
|
I agree that you currently have better output than gstark's. He still wins for clarity, simplicity and tests. I think some compromise can be met. I'm working on both patches atm to see what can be done. P.S. tabs... really? |
|
I've got both sets of code in my repo with ENV guards around both so I can switch back and forth. I've massaged both sets of code a bit (like, I like your output better, but it was going to stdout, not stderr; so now they're both the same and going to stderr). In almost all cases, gstark's code is faster (esp after adopting some of your shortcutting). It is also tested and easier to read in almost all cases. I can send you a combined patch if you'd like to play with it. I'll save one off now just in case. I'm going to reduce this now to a single changeset and push it up soon. |
|
pushed. please let me know what you think. |
|
Thanks for reconsidering (and sorry about the formatting). It looks like this produces equivalent results, but faster, so that's good. I should have included a test case with a dash or underscore in the name, which is really common (looks like 4,626 gems with an underscore, 5,745 with a dash). Clearly, removing dashes and underscores adds overhead. In cases where it allows the matching to short circuit (e.g., "red_cloth" or "mimetypes"), it will be faster. Otherwise, slower. On the other hand, in my opinion, it (slightly) increases the user experience by returning more precise results. I've added my changes here and can submit as a pull request if desired: https://github.com/presidentbeef/rubygems/tree/remove_underscores_on_suggest Examples: Remove underscores/dashes: Don't remove: |
|
Sorry, one more example for comparison. This is the time for when a gem name does not match any existing gems, which is probably the worst case scenario. Remove underscores/dashes: Don't remove: |
|
Hello, Based on the other thread of discussion and recent 1.4.0 release with such functionality, I'm going ahead and close this pull request. Happy new year! |
Time was wrongly accept non-numeric objects.
fake ALT_SEPARATOR
Sometimes, as a result of parallel installation of gems reading & writing gemspecs concurrently during `bundle update`, some thread can end up printing warnings like the following: ``` [/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead). ``` This commit fixes the issue by completely overriding rubygems installer and removing all the unnecessary stuff. In particular, the `Gem::Specification.reset` call that was constantly invalidating gem specification caches and thus mutating shared state. As a results, `bundle install` time is noticiably faster. A quick test on my system (with ~500 gems installed) on an app with about ~100 locked gems (to avoid resolution overhead) without extensions (to avoid extension compilation overhead), with a fully loaded gem cache (to avoid network overhead), it results in ~15% speed up in `bundle install` time: ``` $ hyperfine -p 'rm -rf vendor/bundle' 'bundle install' 'bundle _2.2.11_ install' Benchmark #1: bundle install Time (mean ± σ): 5.713 s ± 0.387 s [User: 5.215 s, System: 2.163 s] Range (min … max): 5.443 s … 6.610 s 10 runs Benchmark #2: bundle _2.2.11_ install Time (mean ± σ): 6.549 s ± 0.106 s [User: 6.087 s, System: 2.353 s] Range (min … max): 6.412 s … 6.773 s 10 runs Summary 'bundle install' ran 1.15 ± 0.08 times faster than 'bundle _2.2.11_ install' ```
Sometimes, as a result of parallel installation of gems reading & writing gemspecs concurrently during `bundle update`, some thread can end up printing warnings like the following: ``` [/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead). ``` This commit fixes the issue by completely overriding rubygems installer and removing all the unnecessary stuff. In particular, the `Gem::Specification.reset` call that was constantly invalidating gem specification caches and thus mutating shared state. As a results, `bundle install` time is noticiably faster. A quick test on my system (with ~500 gems installed) on an app with about ~100 locked gems (to avoid resolution overhead) without extensions (to avoid extension compilation overhead), with a fully loaded gem cache (to avoid network overhead), it results in ~15% speed up in `bundle install` time: ``` $ hyperfine -p 'rm -rf vendor/bundle' 'bundle install' 'bundle _2.2.11_ install' Benchmark #1: bundle install Time (mean ± σ): 5.713 s ± 0.387 s [User: 5.215 s, System: 2.163 s] Range (min … max): 5.443 s … 6.610 s 10 runs Benchmark #2: bundle _2.2.11_ install Time (mean ± σ): 6.549 s ± 0.106 s [User: 6.087 s, System: 2.353 s] Range (min … max): 6.412 s … 6.773 s 10 runs Summary 'bundle install' ran 1.15 ± 0.08 times faster than 'bundle _2.2.11_ install' ```
Sometimes, as a result of parallel installation of gems reading & writing gemspecs concurrently during `bundle update`, some thread can end up printing warnings like the following: ``` [/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead). ``` This commit fixes the issue by completely overriding rubygems installer and removing all the unnecessary stuff. In particular, the `Gem::Specification.reset` call that was constantly invalidating gem specification caches and thus mutating shared state has been moved to the main thread, after after installation of all gems is finished. As a results, `bundle install` time is noticiably faster. A quick test on my system (with ~500 gems installed) on an app with about ~100 locked gems (to avoid resolution overhead) without extensions (to avoid extension compilation overhead), with a fully loaded gem cache (to avoid network overhead), it results in ~15% speed up in `bundle install` time: ``` $ hyperfine -p 'rm -rf vendor/bundle' 'bundle install' 'bundle _2.2.11_ install' Benchmark #1: bundle install Time (mean ± σ): 5.713 s ± 0.387 s [User: 5.215 s, System: 2.163 s] Range (min … max): 5.443 s … 6.610 s 10 runs Benchmark #2: bundle _2.2.11_ install Time (mean ± σ): 6.549 s ± 0.106 s [User: 6.087 s, System: 2.353 s] Range (min … max): 6.412 s … 6.773 s 10 runs Summary 'bundle install' ran 1.15 ± 0.08 times faster than 'bundle _2.2.11_ install' ```
Sometimes, as a result of parallel installation of gems reading & writing gemspecs concurrently during `bundle update`, some thread can end up printing warnings like the following: ``` [/home/runner/work/rubygems/rubygems/bundler/tmp/2/gems/system/specifications/zeitwerk-2.4.2.gemspec] isn't a Gem::Specification (NilClass instead). ``` The fix is to completely overriding rubygems installer to get rid of the places where it accessed and modified shared gem specification cache. In particular: * Stop calling `Gem::Specification.reset` after each gem installation and instead move it the main thread, after after installation of all gems has finished. * Stop using rubygems implementation of `Gem::Specification.latest_spec_for(name)` in favor of `Gem::Specification.stubs_for(name).first`, which does the same thing but without traversing and potentially loading all specifications. * Remove all the other code that's not actually necessary for bundler. Technically only the first point would be necessary to fix the race condition but I figured since I needed to overwrite the method I would only include what's needed, and make it faster. As a result, `bundle install` time is noticiably faster. A quick test on my system (with ~500 gems installed) on an app with about ~100 locked gems (to avoid resolution overhead) without extensions (to avoid extension compilation overhead), with a fully loaded gem cache (to avoid network overhead), it results in ~15% speed up in `bundle install` time: ``` $ hyperfine -p 'rm -rf vendor/bundle' 'bundle install' 'bundle _2.2.11_ install' Benchmark #1: bundle install Time (mean ± σ): 5.713 s ± 0.387 s [User: 5.215 s, System: 2.163 s] Range (min … max): 5.443 s … 6.610 s 10 runs Benchmark #2: bundle _2.2.11_ install Time (mean ± σ): 6.549 s ± 0.106 s [User: 6.087 s, System: 2.353 s] Range (min … max): 6.412 s … 6.773 s 10 runs Summary 'bundle install' ran 1.15 ± 0.08 times faster than 'bundle _2.2.11_ install' ```
I did a bad thing (script that edits the Gemfile.lock directly) and ended up with a Gemfile.lock that was completely missing some indirect dependencies. While this is my fault and an error is reasonable, I noticed that the error got progressively less friendly in recent versions of bundler. Something similar came up in ruby#6210, and this commit would have helped with that case as well (although we've already handled it a different way with ruby#6219). Details: --- Back on Bundler 2.2.23, a corrupt lockfile would cause a helpful error: ``` Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted? ``` Bundler 2.3.26 gave a helpful warning: ``` Warning: Your lockfile was created by an old Bundler that left some things out. Because of the missing DEPENDENCIES, we can only install gems one at a time, instead of installing 16 at a time. You can fix this by adding the missing gems to your Gemfile, running bundle install, and then removing the gems from your Gemfile. The missing gems are: * minitest depended upon by activesupport ``` But then continued on and crashed while trying to report the unmet dependency: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- NoMethodError: undefined method `full_name' for nil:NilClass lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies' ... ``` Bundler 2.4.0 and up crash as above when jobs=1, but crash even harder when run in parallel: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- fatal: No live threads left. Deadlock? 3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660 * #<Thread:0x000000010833b130 sleep_forever> rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0 * #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0 * #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0 <internal:thread_sync>:18:in `pop' tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq' ... ``` Changes --- This commit fixes the confusing thread deadlock crash by detecting if dependencies are missing such that we'll never be able to enqueue a spec and treating that as a failure. That gets us back to the `NoMethodError`, which this commit fixes by using a different warning in the case where no spec is found.
I did a bad thing (script that edits the Gemfile.lock directly) and ended up with a Gemfile.lock that was completely missing some indirect dependencies. While this is my fault and an error is reasonable, I noticed that the error got progressively less friendly in recent versions of bundler. Something similar came up in ruby#6210, and this commit would have helped with that case as well (although we've already handled it a different way with ruby#6219). Details: --- Back on Bundler 2.2.23, a corrupt lockfile would cause a helpful error: ``` Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted? ``` Bundler 2.3.26 gave a helpful warning: ``` Warning: Your lockfile was created by an old Bundler that left some things out. Because of the missing DEPENDENCIES, we can only install gems one at a time, instead of installing 16 at a time. You can fix this by adding the missing gems to your Gemfile, running bundle install, and then removing the gems from your Gemfile. The missing gems are: * minitest depended upon by activesupport ``` But then continued on and crashed while trying to report the unmet dependency: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- NoMethodError: undefined method `full_name' for nil:NilClass lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies' ... ``` Bundler 2.4.0 and up crash as above when jobs=1, but crash even harder when run in parallel: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- fatal: No live threads left. Deadlock? 3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660 * #<Thread:0x000000010833b130 sleep_forever> rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0 * #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0 * #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0 <internal:thread_sync>:18:in `pop' tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq' ... ``` Changes --- This commit fixes the confusing thread deadlock crash by detecting if dependencies are missing such that we'll never be able to enqueue a spec and treating that as a failure. That gets us back to the `NoMethodError`, which this commit fixes by using a different warning in the case where no spec is found.
I did a bad thing (script that edits the Gemfile.lock directly) and ended up with a Gemfile.lock that was completely missing some indirect dependencies. While this is my fault and an error is reasonable, I noticed that the error got progressively less friendly in recent versions of bundler. Something similar came up in ruby#6210, and this commit would have helped with that case as well (although we've already handled this a different way with ruby#6219). Details: --- Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error: ``` Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted? ``` Bundler 2.3.26 gave a helpful warning: ``` Warning: Your lockfile was created by an old Bundler that left some things out. Because of the missing DEPENDENCIES, we can only install gems one at a time, instead of installing 16 at a time. You can fix this by adding the missing gems to your Gemfile, running bundle install, and then removing the gems from your Gemfile. The missing gems are: * minitest depended upon by activesupport ``` But then continued on and crashed while trying to report the unmet dependency: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- NoMethodError: undefined method `full_name' for nil:NilClass lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies' ... ``` Bundler 2.4.0 and up crash as above when jobs=1, but crash even harder when run in parallel: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- fatal: No live threads left. Deadlock? 3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660 * #<Thread:0x000000010833b130 sleep_forever> rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0 * #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0 * #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0 <internal:thread_sync>:18:in `pop' tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq' ... ``` Changes --- This commit fixes the confusing thread deadlock crash by detecting if dependencies are missing such that we'll never be able to enqueue a spec and treating that as a failure. That gets us back to the `NoMethodError`, which this commit fixes by using a different warning in the case where no spec is found.
I did a bad thing (script that edits the Gemfile.lock directly) and ended up with a Gemfile.lock that was completely missing some indirect dependencies. While this is my fault and an error is reasonable, I noticed that the error got progressively less friendly in recent versions of bundler. Something similar came up in ruby#6210, and this commit would have helped with that case as well (although we've already handled this a different way with ruby#6219). Details: --- Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error: ``` Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted? ``` Bundler 2.3.26 gave a helpful warning: ``` Warning: Your lockfile was created by an old Bundler that left some things out. Because of the missing DEPENDENCIES, we can only install gems one at a time, instead of installing 16 at a time. You can fix this by adding the missing gems to your Gemfile, running bundle install, and then removing the gems from your Gemfile. The missing gems are: * minitest depended upon by activesupport ``` But then continued on and crashed while trying to report the unmet dependency: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- NoMethodError: undefined method `full_name' for nil:NilClass lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies' ... ``` Bundler 2.4.0 and up crash as above when jobs=1, but crash even harder when run in parallel: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- fatal: No live threads left. Deadlock? 3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660 * #<Thread:0x000000010833b130 sleep_forever> rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0 * #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0 * #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0 <internal:thread_sync>:18:in `pop' tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq' ... ``` Changes --- This commit fixes the confusing thread deadlock crash by detecting if dependencies are missing such that we'll never be able to enqueue. When that happens we treat it as a failure so the install can finish. That gets us back to the `NoMethodError`, which this commit fixes by using a different warning in the case where no spec is found.
I did a bad thing (script that edits the Gemfile.lock directly) and ended up with a Gemfile.lock that was completely missing some indirect dependencies. While this is my fault and an error is reasonable, I noticed that the error got progressively less friendly in recent versions of bundler. Something similar came up in ruby#6210, and this commit would have helped with that case as well (although we've already handled this a different way with ruby#6219). Details: --- Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error: ``` Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted? ``` Bundler 2.3.26 gave a helpful warning: ``` Warning: Your lockfile was created by an old Bundler that left some things out. Because of the missing DEPENDENCIES, we can only install gems one at a time, instead of installing 16 at a time. You can fix this by adding the missing gems to your Gemfile, running bundle install, and then removing the gems from your Gemfile. The missing gems are: * minitest depended upon by activesupport ``` But then continued on and crashed while trying to report the unmet dependency: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- NoMethodError: undefined method `full_name' for nil:NilClass lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies' ... ``` Bundler 2.4.0 and up crash as above when jobs=1, but crash even harder when run in parallel: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- fatal: No live threads left. Deadlock? 3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660 * #<Thread:0x000000010833b130 sleep_forever> rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0 * #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0 * #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0 <internal:thread_sync>:18:in `pop' tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq' ... ``` Changes --- This commit fixes the confusing thread deadlock crash by detecting if dependencies are missing such that we'll never be able to enqueue. When that happens we treat it as a failure so the install can finish. That gets us back to the `NoMethodError`, which this commit fixes by using a different warning in the case where no spec is found.
I did a bad thing (script that edits the Gemfile.lock directly) and ended up with a Gemfile.lock that was completely missing some indirect dependencies. While this is my fault and an error is reasonable, I noticed that the error got progressively less friendly in recent versions of bundler. Something similar came up in ruby#6210, and this commit would have helped with that case as well (although we've already handled this a different way with ruby#6219). Details: --- Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error: ``` Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted? ``` Bundler 2.3.26 gave a helpful warning: ``` Warning: Your lockfile was created by an old Bundler that left some things out. Because of the missing DEPENDENCIES, we can only install gems one at a time, instead of installing 16 at a time. You can fix this by adding the missing gems to your Gemfile, running bundle install, and then removing the gems from your Gemfile. The missing gems are: * minitest depended upon by activesupport ``` But then continued on and crashed while trying to report the unmet dependency: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- NoMethodError: undefined method `full_name' for nil:NilClass lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies' ... ``` Bundler 2.4.0 and up crash as above when jobs=1, but crash even harder when run in parallel: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- fatal: No live threads left. Deadlock? 3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660 * #<Thread:0x000000010833b130 sleep_forever> rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0 * #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0 * #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0 <internal:thread_sync>:18:in `pop' tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq' ... ``` Changes --- This commit fixes the confusing thread deadlock crash by detecting if dependencies are missing such that we'll never be able to enqueue. When that happens we treat it as a failure so the install can finish. That gets us back to the `NoMethodError`, which this commit fixes by using a different warning in the case where no spec is found.
I did a bad thing (script that edits the Gemfile.lock directly) and ended up with a Gemfile.lock that was completely missing some indirect dependencies. While this is my fault and an error is reasonable, I noticed that the error got progressively less friendly in recent versions of bundler. Something similar came up in ruby#6210, and this commit would have helped with that case as well (although we've already handled this a different way with ruby#6219). Details: --- Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error: ``` Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted? ``` Bundler 2.3.26 gave a helpful warning: ``` Warning: Your lockfile was created by an old Bundler that left some things out. Because of the missing DEPENDENCIES, we can only install gems one at a time, instead of installing 16 at a time. You can fix this by adding the missing gems to your Gemfile, running bundle install, and then removing the gems from your Gemfile. The missing gems are: * minitest depended upon by activesupport ``` But then continued on and crashed while trying to report the unmet dependency: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- NoMethodError: undefined method `full_name' for nil:NilClass lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies' ... ``` Bundler 2.4.0 and up crash as above when jobs=1, but crash even harder when run in parallel: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- fatal: No live threads left. Deadlock? 3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660 * #<Thread:0x000000010833b130 sleep_forever> rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0 * #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0 * #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0 <internal:thread_sync>:18:in `pop' tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq' ... ``` Changes --- This commit fixes the confusing thread deadlock crash by detecting if dependencies are missing such that we'll never be able to enqueue. When that happens we treat it as a failure so the install can finish. That gets us back to the `NoMethodError`, which this commit fixes by using a different warning in the case where no spec is found.
I did a bad thing (script that edits the Gemfile.lock directly) and ended up with a Gemfile.lock that was completely missing some indirect dependencies. While this is my fault and an error is reasonable, I noticed that the error got progressively less friendly in recent versions of bundler. Something similar came up in ruby#6210, and this commit would have helped with that case as well (although we've already handled this a different way with ruby#6219). Details: --- Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error: ``` Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted? ``` Bundler 2.3.26 gave a helpful warning: ``` Warning: Your lockfile was created by an old Bundler that left some things out. Because of the missing DEPENDENCIES, we can only install gems one at a time, instead of installing 16 at a time. You can fix this by adding the missing gems to your Gemfile, running bundle install, and then removing the gems from your Gemfile. The missing gems are: * minitest depended upon by activesupport ``` But then continued on and crashed while trying to report the unmet dependency: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- NoMethodError: undefined method `full_name' for nil:NilClass lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies' ... ``` Bundler 2.4.0 and up crash as above when jobs=1, but crash even harder when run in parallel: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- fatal: No live threads left. Deadlock? 3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660 * #<Thread:0x000000010833b130 sleep_forever> rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0 * #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0 * #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0 <internal:thread_sync>:18:in `pop' tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq' ... ``` Changes --- This commit fixes the confusing thread deadlock crash by detecting if dependencies are missing such that we'll never be able to enqueue. When that happens we treat it as a failure so the install can finish. That gets us back to the `NoMethodError`, which this commit fixes by using a different warning in the case where no spec is found.
I did a bad thing (script that edits the Gemfile.lock directly) and ended up with a Gemfile.lock that was completely missing some indirect dependencies. While this is my fault and an error is reasonable, I noticed that the error got progressively less friendly in recent versions of bundler. Something similar came up in ruby#6210, and this commit would have helped with that case as well (although we've already handled this a different way with ruby#6219). Details: --- Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error: ``` Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted? ``` Bundler 2.3.26 gave a helpful warning: ``` Warning: Your lockfile was created by an old Bundler that left some things out. Because of the missing DEPENDENCIES, we can only install gems one at a time, instead of installing 16 at a time. You can fix this by adding the missing gems to your Gemfile, running bundle install, and then removing the gems from your Gemfile. The missing gems are: * minitest depended upon by activesupport ``` But then continued on and crashed while trying to report the unmet dependency: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- NoMethodError: undefined method `full_name' for nil:NilClass lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies' ... ``` Bundler 2.4.0 and up crash as above when jobs=1, but crash even harder when run in parallel: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- fatal: No live threads left. Deadlock? 3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660 * #<Thread:0x000000010833b130 sleep_forever> rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0 * #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0 * #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0 <internal:thread_sync>:18:in `pop' tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq' ... ``` Changes --- This commit fixes the confusing thread deadlock crash by detecting if dependencies are missing such that we'll never be able to enqueue. When that happens we treat it as a failure so the install can finish. That gets us back to the `NoMethodError`, which this commit fixes by using a different warning in the case where no spec is found.
I did a bad thing (script that edits the Gemfile.lock directly) and ended up with a Gemfile.lock that was completely missing some indirect dependencies. While this is my fault and an error is reasonable, I noticed that the error got progressively less friendly in recent versions of bundler. Something similar came up in ruby#6210, and this commit would have helped with that case as well (although we've already handled this a different way with ruby#6219). Details: --- Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error: ``` Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted? ``` Bundler 2.3.26 gave a helpful warning: ``` Warning: Your lockfile was created by an old Bundler that left some things out. Because of the missing DEPENDENCIES, we can only install gems one at a time, instead of installing 16 at a time. You can fix this by adding the missing gems to your Gemfile, running bundle install, and then removing the gems from your Gemfile. The missing gems are: * minitest depended upon by activesupport ``` But then continued on and crashed while trying to report the unmet dependency: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- NoMethodError: undefined method `full_name' for nil:NilClass lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies' ... ``` Bundler 2.4.0 and up crash as above when jobs=1, but crash even harder when run in parallel: ``` --- ERROR REPORT TEMPLATE ------------------------------------------------------- fatal: No live threads left. Deadlock? 3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660 * #<Thread:0x000000010833b130 sleep_forever> rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0 * #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0 * #<Thread:0x0000000108dea4a0@Parallel Installer Worker ruby#1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever> rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0 <internal:thread_sync>:18:in `pop' tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq' ... ``` Changes --- This commit fixes the confusing thread deadlock crash by detecting if dependencies are missing such that we'll never be able to enqueue. When that happens we treat it as a failure so the install can finish. That gets us back to the `NoMethodError`, which this commit fixes by using a different warning in the case where no spec is found.
Okay, I think this is probably the third attempt for this patch :)
Previously: Details on RubyForge, another poke on RubyGems, comments on Gavin's patch, Luis' thread on the mailing list.
What this patch does:
This does not provide suggestions when a version is specified (because I don't know how to tell if a failure is because of a name or version mismatch). There are some arbitrary limitations as described on the RubyForge page.
On the mailing list, James Tucker had the following questions, either about this patch or Gavin's:
I am not sure I understand the first question, because I am not aware of any reason why camel-casing would be considered "more" canonical and I don't see a zentest gem:
The secondary sorting might be worthwhile, although I am not sure how big of a difference it will make when the suggestions are already sorted by similarity.
Thanks for your consideration.