v216: Bundler not loaded?

[sato11]

Deploying my Hanami app started to fail today, and I have found switching to v215 fixes it.

The error took place during rake assets:precompile, saying uninitialized constant Bundler,
which this line is invoking: https://github.com/hanami/hanami/blob/v1.3.3/lib/hanami/environment.rb#L409

My environment is:

• Ruby 2.7.1
• Bundler 2.0.2
• Hanami 1.3.3

Thanks for the maintenance, and I hope this report helps somehow.

[destroytoday]

Relatedly (I think), my Heroku CI test setup failed out of nowhere last night saying that the newest version of the json gem conflicted with another gem, but my bundle had json 1.8.6 installed—not the newest version. After some digging, I realized that the buildpack was running rake db:migrate, not bundle exec rake db:migrate. I couldn’t find a way to not have it run rake db:migrate, so I made that task a no-op and have my test-setup script run prefixed with bundle exec, which fixes it.

tldr; Seems like something changed where rake tasks are no longer called from bundler?

[schneems]

I think this is related to #1005. But it might not be as it's a different error. Can you open a support ticket help.heroku.com search "bundler not loaded". Once you've done that give me the ticket number so I can access your application.

[destroytoday]

I opened a Heroku support ticket. They’re rolling back the release and fixing it. 👍

[sato11]

Thanks for the quick action! 👍 As for a support ticket, it seems I'm not eligible to create one. Let me know however, if there is anything that I can help.

[schneems]

I don't see any "bundler not loaded" tickets right now. I might have responded by accidentally grouped it in with another failure mode. If anyone has a support ticket number please let me know on this thread, please.

@sato11 I can make one for you can you send me your email address that you use for Heroku? You can find my email on my github profile. I'll create a ticket for you that you can attach an app and grant access from there.

[sato11]

@schneems Thank you very much. I've emailed to you.

[schneems]

It took longer than I would like but I've created the ticket, you should get an email from our system. Also i've rolled out a fix for #1005 i've included instructions at the bottom to determine if an issue still exists. If you all can run the commands and let me know if it fixes your issue then it will let me know if there's still a problem that needs to be worked on or if i've fixed your issue.

[sato11]

@schneems
Thanks for the follow-up. I've written info in the ticket.
Also, I've tried the latest master but it seems it does not fix this bundler issue.
I regret that I cannot really pay attention to the details that you are working on. I really appreciate it though. Thank you.

[schneems]

I'm not sure if everyone on the ticket is seeing the same problem or not, but this is what I think is happening right now. Here's two different rake binstubs on the platform of a 2.4.x app:

~ $ cat ./vendor/bundle/bin/rake
#!/usr/bin/env ruby
# frozen_string_literal: true

#
# This file was generated by Bundler.
#
# The application 'rake' is installed as part of a gem, and
# this file is here to facilitate running it.
#

require "pathname"
ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../../../Gemfile",
  Pathname.new(__FILE__).realpath)

bundle_binstub = File.expand_path("../bundle", __FILE__)

if File.file?(bundle_binstub)
  if File.read(bundle_binstub, 300) =~ /This file was generated by Bundler/
    load(bundle_binstub)
  else
    abort("Your `bin/bundle` was not generated by Bundler, so this binstub cannot run.
Replace `bin/bundle` by running `bundle binstubs bundler --force`, then run this command again.")
  end
end

require "rubygems"
require "bundler/setup"

load Gem.bin_path("rake", "rake")

Versus

~ $ cat bin/rake
#!/bin/sh
# -*- ruby -*-
# This file was generated by RubyGems.
#
# The application 'rake' is installed as part of a gem, and
# this file is here to facilitate running it.
#
_=_\
=begin
bindir="${0%/*}"
exec "$bindir/ruby" "-x" "$0" "$@"
=end
#!/usr/bin/env ruby

require 'rubygems'

version = ">= 0.a"

if ARGV.first
  str = ARGV.first
  str = str.dup.force_encoding("BINARY") if str.respond_to? :force_encoding
  if str =~ /\A_(.*)_\z/ and Gem::Version.correct?($1) then
    version = $1
    ARGV.shift
  end
end

if Gem.respond_to?(:activate_bin_path)
load Gem.activate_bin_path('rake', 'rake', version)
else
gem "rake", version
load Gem.bin_path("rake", "rake", version)
end

You can see that in bin/rake which is generated when the binary is compiled is not aware of Bundler at all, while the vendored rubygem in vendor/bundle/bin/rake is generated from bundle install and it fully knows about bundler (since bundler installed it).

The problem is this: Rubygems has no concept of a "git" gem. The concept of using "git" as a source for a gem comes from Bundler. So in this case i'm guessing that there is one or more gem in the project that is using a "git" gem that cannot be found when rake -P is called.

Again, there might be multiple issues here, but that's the behavior/problem that some are seeing.

Ways that it can be mitigated: If you're using Rails then you can generate and check in binstubs:

rake app:update:bin
git add .
git commit -m "adding rails generated binstubs"

In terms of moving forwards with this specific bug/regression, i'm not sure. It's interesting that even with v215, running it might work at build time but will fail at runtime as runtime puts bin/rake first on the path while it is not at build time.

[schneems]

It looks like this is the same failure mode but with slightly different behavior:

When using bin/rake it fails:

~ $ rake assets:precompile

uninitialized constant Bundler
/app/vendor/bundle/ruby/2.7.0/gems/hanami-1.3.3/lib/hanami/environment.rb:409:in `require_application_environment'
    /app/vendor/bundle/ruby/2.7.0/gems/hanami-1.3.3/lib/hanami/cli/commands/command.rb:77:in `call'
    # ...
    /app/bin/rake:29:in `<main>'

When running bundle exec it works:

~ $ bundle exec rake assets:precompile
# Worked

It fails since bin/rake does not load bundler.

Existing behavior

Existing Build behavior: ./vendor/bundle/bin/rake is first on the path. This file loads bundler so both examples here are able to run rake just fine.
Existing Runtime behavior: /app/bin/rake is first on the path. This means that both existing examples would fail if they tried to add a rake task such as rake db:migrate in their release phase (which runs in the runtime environment).

This behavior is problematic. It's confusing that the behavior in runtime does not match build-time behavior. Ideally, we would consolidate so that the same rake binary is run in both.

Staged behavior

Here's the new behavior that's staged on the main branch:

Staged Build behavior: bin/rake is first on the path. This file does not load bundler which causes these failures seen in this issue. (ticket reference: https://heroku.support/890577 and https://heroku.support/893446)
Staged Runtime behavior: /app/bin/rake is first on the path. This means that both existing examples would still fail if they tried to add a rake task such as rake db:migrate in their release phase (which runs in the runtime environment).

This behavior is consistent, but it is a change from prior behavior. Here's the two cases we're trying to accommodate:

Use cases

• Checked in binstubs: Frameworks like rails will generate and check-in a bin/rake binstub. If I did this, then I would expect that my binstub gets used.
• No Checked in binstubs: If you've not got a bin/rake binstub checked in, then previously you're relying on the binstub generated by bundler in vendor/bundler/bin/ruby in deploy-time and bin/ruby in runtime. We've seen how the discrepancy here can cause failures in one but not the other.

Next steps

Here's what I think I want to do: I can detect if you've added a bin/rake file to your code manually. If you've done that then I'll give your app bin/rake in both build and runtime. If I don't see a checked-in binstub, then I'll assume you don't care which version of rake binstub is called. In that case, I'll either make sure vendor/bundle/bin/rake is first on the path somehow, or symlink it, or delete bin/rake etc. I'm not totally sure.

I think I can accommodate existing expected behavior while fixing problems (different binstub used in runtime and build time) and not break anyone's existing application. I'll work on some experiments now and get back to yall when I have more to share.

[sato11]

Thanks for the investigation. It seems it's becoming more complicated than I've initially expected.
And since it sounds like you want me to decide which path to take, as for the next step I prefer not to add bin/rake, just because it's the way it's used to be.
On the other hand if you really need me to do it I'll be doing it. My app is just for fun after all and I do not really mind about the subtleties. Thanks again for the consistent help!

[schneems]

@sato11 I don't need action on your part. I'm mostly talking through the context of the problem and what the opportunities to fix it are before I discuss a specific implementation. I believe I have a fix in #1031 and I'm about to merge it in.

[schneems]

The fix has been merged and deployed let me know if you're still seeing the same failure