ChefDK 2.5.3 reports GemLoadError for activesupport

[haidangwa]

Description

I upgraded from ChefDK 2.4.17 to 2.5.3; subsequently, running chef exec bundle exec kitchen list or other kitchen commands would result in a GemLoadError. Reverting back to 2.4.17 does not have this problem.

ChefDK Version

2.5.3

Platform Version

Mac OSX 10.12.6

Replication Case

Upgrade from ChefDK 2.4.17 to 2.5.3; after getting the error, I've tried rm -rf ~/.chefdk and re-installing all the gems in my cookbook's Gemfile (i.e., chef exec bundle install). Then run chef exec bundle kitchen list. The error persists.

Revert to 2.4.17, rm -rf ~/.chefdk, chef exec bundle install, chef exec bundle kitchen list is successful.

Stacktrace

$ chef exec bundle kitchen list
/opt/chefdk/embedded/lib/ruby/site_ruby/2.4.0/bundler/rubygems_integration.rb:404:in `block (2 levels) in replace_gem': activesupport is not part of the bundle. Add it to your Gemfile. (Gem::LoadError)
	from /opt/chefdk/bin/kitchen:7:in `<main>'
$ chef -v
Chef Development Kit Version: 2.5.3
chef-client version: 13.8.5
delivery version: master (73ebb72a6c42b3d2ff5370c476be800fee7e5427)
berks version: 6.3.1
kitchen version: 1.20.0
inspec version: 1.51.21

NOTE: CHEFDK BUGS ONLY

This issue tracker is for the code contained within this repo -- chefdk.

• Server issues
• Chef-client issues
• Cookbook Issues (see the https://github.com/chef-cookbooks repos or search Supermarket or GitHub/Google)

[coderanger]

Nothing in Test Kitchen should be loading activesupport. Perhaps this is a mistake in your kitchen config or a third-party plugin of some kind?

[haidangwa]

we don't have anything loading or depending on activesupport. When I reinstall chefdk 2.4.17, I do see that it is installed with the activesupport gem, but with chefdk 2.5.3, that gem is not installed.

[haidangwa]

I also see this in chefdk's Gemfile.lock: https://github.com/chef/chef-dk/blob/3a33ab1a688c7dc24cd55aba5ccf9ee951069b5d/Gemfile.lock#L20

[thommay]

Are you using digital ocean for your kitchen driver?

[haidangwa]

we use kitchen-openstack and kitchen-ec2

…

On Mon, Apr 16, 2018 at 1:17 AM, Thom May ***@***.***> wrote: Are you using digital ocean for your kitchen driver? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1547 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB2A7ROdYQQrzVm69awNJmzHcNtYV9hKks5tpFOXgaJpZM4TVUsS> .

[coderanger]

Okay, so you are running bundle kitchen list, remove the bundle. We do not use bundler in ChefDK (on the user side).

You cannot use a Gemfile in your cookbook along with ChefDK, the approaches are incompatible.

[haidangwa]

/opt/chefdk/embedded/bin/bundle exists

…

On Mon, Apr 16, 2018 at 12:45 PM, Noah Kantrowitz ***@***.***> wrote: Okay, so you are running bundle kitchen list, remove the bundle. We do not use bundler in ChefDK (on the user side). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1547 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB2A7UpzuMlk81Rpcy9b4rzIFllvx0rcks5tpPS-gaJpZM4TVUsS> .

[coderanger]

Indeed it does, it's an implementation detail of how we build the packages. In general, stuff inside embedded/bin isn't for general consumption (though I think for Complicated Reasons the rspec executable stub is still only in there).

[benclark]

This is happening for me as well: I upgraded from ChefDK 2.1.11 to ChefDK 2.5.3 on macOS, and bundle exec knife no longer works. (Note that in my case, I'm using bundler to pull in my custom knife plugin.)

Same error as @haidangwa:

/opt/chefdk/embedded/lib/ruby/site_ruby/2.4.0/bundler/rubygems_integration.rb:404:in `block (2 levels) in replace_gem': activesupport is not part of the bundle. Add it to your Gemfile. (Gem::LoadError)
	from /opt/chefdk/bin/knife:7:in `<main>'

Also, Gemfiles are fairly ubiquitous in my cookbooks, so @coderanger's comment confused me. I don't know of any other way to pull in chef-sugar gem, for example, so that chefspec can find it, for cookbooks that depend on chef-sugar (I have some in my cookbooks' dependency trees). I also use bundler to ensure my custom Rake library is present in the cookbook workspace for my CI pipeline.

tl;dr if bundler is wrong, what's the correct way to be doing this with ChefDK?

--
Edit: I copy-pasta'd the wrong stacktrace. Corrected.

[haidangwa]

our workflow is similar to @benclark. In addition, we leverage bundler (yes, we commit the Gemfile.lock) to ensure that our CI environment is exactly the same as what we use to test locally on our workstations.

[lamont-granquist]

the chef-cookbooks pattern uses ChefDK to provide most of the gems and bundler to provide gems which ChefDK does not provide.

so Gemfiles in cookbooks look like:

https://github.com/chef-cookbooks/activemq/blob/master/Gemfile

travis installs chefdk:

https://github.com/chef-cookbooks/activemq/blob/master/.travis.yml#L8-L9

it then puts /opt/chefdk/bin first in the path and the execution of foodcritic, et al is equivalent to /opt/chefdk/bin/foodcritic:

https://github.com/chef-cookbooks/activemq/blob/master/.travis.yml#L36-L40

where necessary it uses chef exec and not bundler (this will apply to chef exec rspec to run chefspec in particular):

https://github.com/chef-cookbooks/activemq/blob/master/.travis.yml#L47

...

the other alternative way of doing things is to only use bundler, in which case you must not use the appbundled binstubs.

$ bundle exec /opt/chefdk/embedded/bin/kitchen list

in this case though you must have test-kitchen, et al in your Gemfile and you're not really using ChefDK.

there's an issue against appbundler to make it not setup the binstub pins if it detects it is running through bundler which would make this more transparent:

chef/appbundler#33

[haidangwa]

The alternative way that @lamont-granquist mentioned is exactly how we're using it. I manage a jenkins server that happens to use chefdk to setup the jenkins environment, but we leave it to the different segments and their teams to define everything else in Gemfile.

Our typical Gemfile (I've removed the internal rubygems that we host on a private server):

source 'https://rubygems.org/'

# Public Gems
gem 'berkshelf', '~> 6.3.1'
gem 'chef', '~> 13.8'
gem 'chefspec', '~> 7.1.0'
gem 'foodcritic', '~> 12.2.2'
gem 'inspec' # constrained in kitchen-inspec
gem 'kitchen-inspec', '~> 0.23.1'
gem 'kitchen-openstack', '~> 3.6.0'
gem 'kitchen-sync', '~> 2.2.1'
gem 'rubocop', '~> 0.54.0'

[lamont-granquist]

This is addressed by chef/appbundler#43 which is the best we can do. Binstubs will now not activate if you're running under bundler. That means bundle exec + Gemfile behaves normally. That means that you're just using chef-dk as a normal ruby though with a bunch of preinstalled gems -- you buy yourself all the normal problems of constructing a sane Gemfile lock yourself.

At the point where you're putting that many gems in your Gemfile, you're actually not using chefdk for very much you're just using ruby -- which is entirely fine, just pointing out that you're basically on your own doing your own thing.

This is actually the recommended mode of operation for dealing with old chef version on new chefdk (i.e. supporting chef-client 12.x while using ChefDK 3.x tooling). The only thing you need there though is a Gemfile with your chef 12.x version + chefspec. Then bundle exec rspec to run chefspec against the bundled chef-12 install, and do not use bundle exec for kitchen, berks, knife and other tooling (and set your bootstrap_version in your knife config and the appropriate chef version in all your .kitchen.ymls).

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.