Fix gem installation location bug with Bundler env vars

When the move from bundler flags to bundler env vars was first merged in an issue was reported #1046 , this lead to an investigation and bug report to bundler rubygems/rubygems#3890 which lead to some older issues/prs:

- rubygems/bundler#3552
- rubygems/bundler#6628

The issue is that global configuration and local configuration results in different behavior when installing and using some features in bundler, notedly the ability to specify install path. Due to this change, when the switch to bundler environment variables happened, it caused the size of some applications' slugs to increase dramatically, this was because the gems were essentially being installed twice.

The issue appears to not be in Bundler 2.1.4 so we're bumping the version for 2.x series. In the 1.x series an environment variable `BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE=1` can be used to force the desired behavior.
# This is the commit message #2:

Co-authored-by: Ed Morley <501702+edmorley@users.noreply.github.com>

CHANGELOG.md

@@ -1,5 +1,6 @@
 # Main (unreleased)
 
+* Bundler 2.x is now 2.1.4 (https://github.com/heroku/heroku-buildpack-ruby/pull/1052)
 * Persistent bundler config is now being set using the `BUNDLE_*` env vars (https://github.com/heroku/heroku-buildpack-ruby/pull/1039)
 * Rake task "assets:clean" will not get called if it does not exist (https://github.com/heroku/heroku-buildpack-ruby/pull/1018)
 * CNB: Fix the `gems` layer not being made accessible by subsequent buildpacks (https://github.com/heroku/heroku-buildpack-ruby/pull/1033)

changelogs/unreleased/bundler-214.md

@@ -0,0 +1,3 @@
+##  Ruby apps with Bundler 2.x now receive version 2.1.4
+
+The [Ruby Buildpack](https://devcenter.heroku.com/articles/ruby-support#libraries) now includes Bundler 2.1.4. Applications specifying Bundler 2.x in their `Gemfile.lock` will now receive bundler: 2.1.4.

changelogs/unreleased/bundler-env-vars.md

@@ -0,0 +1,20 @@
+## Ruby applications now configure bundler with environment variables instead of flags
+
+Previously the Ruby buildpack ran bundler installation with flags:
+
+```
+Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
+[DEPRECATED] The `--deployment` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set deployment 'true'`, and stop using this flag
+[DEPRECATED] The `--path` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set path 'vendor/bundle'`, and stop using this flag
+[DEPRECATED] The `--without` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set without 'development:test'`, and stop using this flag
+[DEPRECATED] The --binstubs option will be removed in favor of `bundle binstubs`
+```
+
+In order to remove deprecations from Bundler 2.x, Ruby applications now run bundler installation with environment variables instead:
+
+```
+Running: BUNDLE_WITHOUT=development:test BUNDLE_PATH=vendor/bundle BUNDLE_BIN=vendor/bundle/bin BUNDLE_DEPLOYMENT=1 bundle install -j4
+```
+
+This behavior is documented in the [Heroku Ruby Support page](https://devcenter.heroku.com/articles/ruby-support).
+

lib/language_pack/helpers/bundler_wrapper.rb

@@ -37,7 +37,7 @@ class LanguagePack::Helpers::BundlerWrapper
 
   BLESSED_BUNDLER_VERSIONS = {}
   BLESSED_BUNDLER_VERSIONS["1"] = "1.17.3"
-  BLESSED_BUNDLER_VERSIONS["2"] = "2.0.2"
+  BLESSED_BUNDLER_VERSIONS["2"] = "2.1.4"
   BUNDLED_WITH_REGEX = /^BUNDLED WITH$(\r?\n)   (?<major>\d+)\.\d+\.\d+/m
 
   class GemfileParseError < BuildpackError
@@ -161,6 +161,13 @@ def lockfile_parser
     @lockfile_parser ||= parse_gemfile_lock
   end
 
+  # Some bundler versions have different behavior
+  # if config is global versus local. These versions need
+  # the environment variable BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE=1
+  def needs_ruby_global_append_path?
+    Gem::Version.new(@version) < Gem::Version.new("2.1.4")
+  end
+
   private
   def fetch_bundler
     instrument 'fetch_bundler' do

lib/language_pack/ruby.rb

@@ -408,6 +408,7 @@ def setup_language_pack_environment(ruby_layer_path:, gem_layer_path:, bundle_pa
       ENV["BUNDLE_PATH"] = bundle_path
       ENV["BUNDLE_BIN"] = bundler_binstubs_path
       ENV["BUNDLE_DEPLOYMENT"] = "1"
+      ENV["BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE"] = "1" if bundler.needs_ruby_global_append_path?
     end
   end
 
@@ -445,6 +446,7 @@ def setup_export(layer = nil)
       set_export_default "BUNDLE_PATH", ENV["BUNDLE_PATH"], layer
       set_export_default "BUNDLE_WITHOUT", ENV["BUNDLE_WITHOUT"], layer
       set_export_default "BUNDLE_BIN", ENV["BUNDLE_BIN"], layer
+      set_export_default "BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE", ENV["BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE"], layer if bundler.needs_ruby_global_append_path?
       set_export_default "BUNDLE_DEPLOYMENT", ENV["BUNDLE_DEPLOYMENT"], layer if ENV["BUNDLE_DEPLOYMENT"] # Unset on windows since we delete the Gemfile.lock
     end
   end
@@ -485,6 +487,7 @@ def setup_profiled(ruby_layer_path: , gem_layer_path: )
       set_env_default "BUNDLE_PATH", ENV["BUNDLE_PATH"]
       set_env_default "BUNDLE_WITHOUT", ENV["BUNDLE_WITHOUT"]
       set_env_default "BUNDLE_BIN", ENV["BUNDLE_BIN"]
+      set_env_default "BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE", ENV["BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE"] if bundler.needs_ruby_global_append_path?
       set_env_default "BUNDLE_DEPLOYMENT", ENV["BUNDLE_DEPLOYMENT"] if ENV["BUNDLE_DEPLOYMENT"] # Unset on windows since we delete the Gemfile.lock
     end
   end
@@ -900,6 +903,7 @@ def build_bundler
         bundle_command << "BUNDLE_PATH=#{ENV["BUNDLE_PATH"]} "
         bundle_command << "BUNDLE_BIN=#{ENV["BUNDLE_BIN"]} "
         bundle_command << "BUNDLE_DEPLOYMENT=#{ENV["BUNDLE_DEPLOYMENT"]} " if ENV["BUNDLE_DEPLOYMENT"] # Unset on windows since we delete the Gemfile.lock
+        bundle_command << "BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE=#{ENV["BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE"]} " if bundler.needs_ruby_global_append_path?
         bundle_command << "bundle install -j4"
 
         topic("Installing dependencies using bundler #{bundler.version}")

spec/hatchet/bundler_spec.rb

@@ -1,18 +1,59 @@
 require 'spec_helper'
 
 describe "Bundler" do
-  it "deploys with version 2.x" do
-    before_deploy = Proc.new do
-      run!(%Q{echo "ruby '2.5.7'" >> Gemfile})
-      run!(%Q{printf "\nBUNDLED WITH\n   2.0.1\n" >> Gemfile.lock})
+  it "deploys with version 2.x with Ruby 2.5" do
+    ruby_version = "2.5.7"
+    abi_version = ruby_version.dup
+    abi_version[-1] = "0" # turn 2.5.7 into 2.5.0
+    pending("Must enable HATCHET_EXPENSIVE_MODE") unless ENV["HATCHET_EXPENSIVE_MODE"]
+
+    Hatchet::Runner.new("default_ruby", run_multi: true).tap do |app|
+      app.before_deploy do
+        run!(%Q{echo "ruby '#{ruby_version}'" >> Gemfile})
+        run!(%Q{printf "\nBUNDLED WITH\n   2.0.1\n" >> Gemfile.lock})
+      end
+      app.deploy do
+        expect(app.output).to match("Installing dependencies using bundler 2.")
+        expect(app.output).to_not match("BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE=1")
+
+        # Double deploy problem with Ruby 2.5.5
+        app.commit!
+        app.push!
+
+        app.run_multi("ls vendor/bundle/ruby/#{abi_version}/gems") do |ls_output|
+          expect(ls_output).to match("rake-")
+        end
+
+        app.run_multi("which -a rake") do |which_rake|
+          expect(which_rake).to include("/app/vendor/bundle/bin/rake")
+          expect(which_rake).to include("/app/vendor/bundle/ruby/#{abi_version}/bin/rake")
+        end
+      end
     end
+  end
+
+  it "deploys with version 1.x" do
+    abi_version = LanguagePack::RubyVersion::DEFAULT_VERSION_NUMBER
+    abi_version[-1] = "0" # turn 2.6.6 into 2.6.0
+    pending("Must enable HATCHET_EXPENSIVE_MODE") unless ENV["HATCHET_EXPENSIVE_MODE"]
+
+    Hatchet::Runner.new("default_ruby", run_multi: true).tap do |app|
+      app.before_deploy do
+        run!(%Q{printf "\nBUNDLED WITH\n   1.0.1\n" >> Gemfile.lock})
+      end
+      app.deploy do
+        expect(app.output).to match("Installing dependencies using bundler 1.")
+        expect(app.output).to match("BUNDLE_GLOBAL_PATH_APPENDS_RUBY_SCOPE=1")
 
-    Hatchet::Runner.new("default_ruby", before_deploy: before_deploy).deploy do |app|
-      expect(app.output).to match("Installing dependencies using bundler 2.")
+        app.run_multi("ls vendor/bundle/ruby/#{abi_version}/gems") do |ls_output|
+          expect(ls_output).to match("rake-")
+        end
 
-      # Double deploy problem with Ruby 2.5.5
-      run!(%Q{git commit --allow-empty -m 'Deploying again'})
-      app.push!
+        app.run_multi("which -a rake") do |which_rake|
+          expect(which_rake).to include("/app/vendor/bundle/bin/rake")
+          expect(which_rake).to include("/app/vendor/bundle/ruby/#{abi_version}/bin/rake")
+        end
+      end
     end
   end
 end