Default gems can't be upgraded without booting RubyGems

[headius]

TruffleRuby currently lazy-loads RubyGems to improve startup. Unfortunately this means that unless RubyGems is forced to boot, default gems will not honor gem installed upgrades.

Default gems are installed in the standard library (so they are trivially loadable by require) but also have a specially-treated set of gemspecs. Those gemspecs are parsed by RubyGems at boot time so that upgraded default gems can be activated when the related standard libraries get required.

The full set of default gems in TruffleRuby is visible here: https://github.com/oracle/truffleruby/tree/master/lib/gems/specifications/default

RubyGems can be forced to boot by running bundle exec or via a gem-based executable like rails, but a simple Ruby script that requires a default gem-based standard library will not see upgraded versions of that library.

This is important for a couple reasons:

• Running bundler-based apps by requiring bundler/setup. By the time you require anything from the bundler/ subdir, it's too late to activate an upgraded bundler.
• Security and functionality updates to standard libraries like json or psych won't be honored even if the gems are installed.
• Different default gems may be activated when running a script directly or running it with RubyGems booted, which will be unexpected for users coming from CRuby.

I do not have a fix to suggest, because it may not be possible (currently) to lazy load RubyGems and support always-upgradeable default gems.

Full disclosure: I implemented the default gem feature many years ago, and it is necessarily limited by the way RubyGems and require work.

For a related example see jruby/jruby#6109 which adapts the same lazy logic to JRuby, and has the same impact on default gems.

[eregon]

Thanks for the bug report.
I wasn't aware of that mechanism, I thought one needs to explicitly activate a gem like gem 'foo', 'version' to take precedence over the standard library, but indeed that's not the case with default gems.

That's quite unfortunate because it means installing any of those gems will take precedence over the stdlib, and it means any script ran without Bundler/require "rubygems" will behave differently based on the gems installed, even if the script only requires standard libraries. Standard libraries currently have better testing than the latest gem and make it easier to reproduce issues. Anyway, that's how the semantics in RubyGems are, so we should try to fix this.

I would be interested to hear if anyone gets a problem in practice due to this.
I agree your 3 points above are all relevant compatibility issues.

[headius]

FWIW we'd love to find a way around this too. This issue is why we had never explored the possibility of lazy RubyGems in JRuby, but the gains (as seen in the attached JRuby PR) are substantial. Folks shouldn't have to --disable-gems to get good startup when not using gems.

[eregon]

Similar to what you mentioned, probably the best way is for RubyGems to maintain an easy-to-parse file in a well-known location (so it can be found without loading the full RubyGems) containing the list of installed gems.
Could even be more specialized and contain only the list of gems upgrading default gems, if we assume the set of default gems is a given for a Ruby installation (gem install --default seems broken since a long time, even on MRI and it doesn't seem user-oriented).
And then when requiring one of the default gems, make sure there is no upgraded gem, otherwise load the full RubyGems and re-require.

[headius]

gem install --default was never intended as a user feature. It was a best attempt at meeting ruby-core's requirements that gems still exist in stdlib once they are moved out for gemification.

[eregon]

Fixed in b9669c4