Relax requirements for local gem versions changes

[dee-kryvenko]

Description

I am in some kind of a lock down.

Because test-kitchen/kitchen-ec2#358 I needed something higher than 2.4.22, but all the newer from stable channel still broken with test-kitchen/kitchen-ec2#394. So I had to stick to non-stable 2.4.22 for now. Now, for the sake of having inspec 2.0 (because I am doing some tests using kitchen-terraform and I need aws resources that being introduced in 2.0) I've been fighting with gems versions locks, I've found appbundle-updater but then stuck with this inspec/inspec#2445.

I am angry, I am tired, I want to cry and sleep and sh!t again because living as infant was so much easier. I understand the intentions, to prevent users to shoot their legs, I get why it's a bad idea to messing up chefdk gems, but these experienced like me packing all the stuff in a container anyway, testing it with all kind of tests and promoting to real prod usage only after making 100% sure it works - why in the world am I spending time fighting these stupid problems. Now it sounds like to build completely custom chefdk package is easier than just to override some single gem in the original distribution. It would be so much easier if I would be able just do exactly it using well-known standard open source toolset such as gem/bundler. For the sake of newbies who will shoot their leg one way or another, now I am spending hours and hours and my brain cells and hours again fighting this lockdown instead of creativity and fun.

I know one would say it's an inspec Gemfile issue, but I want to move the focus on that I've been hitting this problems for the last couple of months here and there, and currently my believes are - the current packaging approach is not up to reality (where we're getting bugs and regressions here and there all the time and can't afford waiting for all the fixing/release activities to finish), it makes chef-dk mush less agile, and it has to be changed in a way to relax and simplify managing gem versions in my local setup. I know in 99% cases what I'm doing is a bad idea - but what else can I do to continue my work while waiting for fix+new release+getting it in chef-dk which can easily take months?

All I want is a chef-dk 2.4.22 with any 2.0 inspec, and realizing there is no two-liner to get that done is making me so much disappointed in all the humanity.

ChefDK Version

Any

Platform Version

Any

Replication Case

Many

[lamont-granquist]

The basic point of ChefDK is to provide a consistent Gemfile.lock around all of the tooling in the ecosystem.

So we fight with this massive Gemfile to produce the Gemfile.lock that gets shipped.

https://github.com/chef/chef-dk/blob/master/Gemfile

If you don't want that, I'll reiterate my advice here which is to just use bundler:

#1547 (comment)

The only trick there is you need to bypass the appbundled binstubs in /opt/chefdk/bin which don't currently play nicely with bundler.

That means that you're on your own though when it comes to yak shaving dependency issues. If there's a new point release of ffi or net-ssh or something like that which breaks your whole world then you become entirely responsible for tracking that down, that's the trade off.

[lamont-granquist]

And I don't know what to do with this ticket since there's nothing actionable here. The entire point of ChefDK is to provide the mega-Gemfile.lock.

Without it being strict we're back to people yelling at us about issues like #1187

[coderanger]

Yeah, there is nothing we can do here. Even if we relaxed things in the Gemfile, appbundler would still set exact pins on every bin stub. And we can't really do much about that without a major structural reworking of appbundler (and I don't even know how to make it better without removing most of the benefit). The core issue is probably dependency sprawl in TK with it pulling in both Berkshelf and InSpec as libraries rather than command-line tools which leaves TK very tied to the internals of both. The long-term fix is probably to use them as command-line tools instead, which means they can be in isolated solutions with less cross-talk.

[dee-kryvenko]

@lamont-granquist well you answer yourself on why I don't want to deal with the bundler myself. Been there for years before chefdk. I like Gemfile.lock, not the appbundled though.

You see, with chef-dk I at least have some common and well-known baseline, that I can diverge from on my own responsibility and manage that technical debt somehow. For instance, consider I have my chefdk CI machine as a docker container, so in my Dockerfile I'd get chefdk and then slightly modify it by getting some minor version changes, and then leaving the comment there on why and the link to the ticket which it waits a fix for and that's it.

With the bundler approach people are completely on their own, they have to figure out their own way to keep as close to chefdk baseline as possible to avoid issues (IF they are smart enough to understand that need), and that leaves a huge room for mistakes.

I don't suggest any solutions as of now, I just throwing out my problem statement together with (which I think is valid) use case (having CI server that runs kitchen tests and etc that sometimes needs something else than what is in chefdk but not entirely different thing). Let's have an open discussion.

[coderanger]

@llibicpep We welcome a discussion, but that's not what we use GitHub Issues for. If you have something to discuss, check out our mailing list or Slack.

[dee-kryvenko]

Just from the top of my head - maybe the alternative is to have a community-managed Dockerfile for CI tied to/produced from/aligned with chefdk baseline and version cycles. I get that ChefDK first of all must be a user-friendly, but the way it currently implemented makes it hostile to well managed CI environments. In my scenario I like ChefDK as a baseline bundle but getting 0 benefit for such a hardcore hardcoded nails (there's no such thing as messing up with /opt/chefdk, when I need a change it'll be completely new image at the end). Even if I'd switch to bundler for now to unblock myself, I'd aim to get back to ChefDK once other issues relevant to me is fixed, because this is much safer way. That makes too much work just for the sake of having a few gems updated in the ChefDK. If not make it CI-friendly, maybe to create CI alternative, where such a hardcoded nails is not necessary.

[coderanger]

@llibicpep Again, this isn't really an actionable bug report and we don't use Issues for broad architecture discussion. You raise important points (even if I think there is no actual solution) but this is not the place. Incidentally we do have a Docker Hub image for ChefDK but that doesn't help since it installs the same package as everything else. The issue deep down is the choice we made that appbundler locks the universe in a way that you can add to it, but not subtract or modify. This works well in most cases, but due to the aforementioned library-usage-sprawl in TK, it has become a nexus of frustrating dependency locks.

[dee-kryvenko]

Yeah sorry I missed your message about that (was replying in the same time). Will post something to the mail list tomorrow. I am sure there is solution if only there is a will to find it...

[dee-kryvenko]

Just to leave a link here... https://discourse.chef.io/t/ci-friendly-chefdk/12998

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

[lamont-granquist]

Here's the process to follow to actually do this:

git clone https://github.com/chef/chef-dk.git
cd chef-dk
git checkout v3.8.14
vim Gemfile # or whatever editor you like
bundle update
sudo bash # because i'm about to set the PATH and sudo likes to strip it
export PATH=/opt/chef/embedded/bin:$PATH
bundle install
appbundler . /opt/chefdk/bin chef foodcritic test-kitchen inspec chef-dk chef-apply chef-vault ohai opscode-pushy-client cookstyle dco berkshelf

I just did this to roll back chef-client in my installed chef-dk 3.8.14 from 14.10.9 to 14.9.13, which went flawlessly because that was a tiny change.

# /opt/chefdk/bin/chef --version
Chef Development Kit Version: 3.8.14
chef-client version: 14.9.13
delivery version: master (9d07501a3b347cc687c902319d23dc32dd5fa621)
berks version: 7.0.7
kitchen version: 1.24.0
inspec version: 3.6.6

So this is where all the major difficulty is going to lie:

vim Gemfile # or whatever editor you like
bundle update

That requires the user to actually know the care-and-feeding of Gemfile's and how to use bundler. And given the side of the Gemfile.lock in chef-dk it requires expert knowledge. Anyone how knows that much about Gemfile's and bundler will be better served by the alternative approach I am going to give. In some cases what the user wishes will be explicitly impossible to solve (since the chef-dk gemspec itself requires a recent chef any older version of chef (like 13.x version of chef in this example) will not work and will fail hard and impossibly. Constructing a ChefDK 3 with a Chef Client 12 gem embedded in it is not a coherent thing (not without rolling back all of the gems to a state which resembles ChefDK 1.x).

The other point is that the user is now to free to cheat on this line:

appbundler . /opt/chefdk/bin chef foodcritic test-kitchen inspec chef-dk chef-apply chef-vault ohai opscode-pushy-client cookstyle dco berkshelf

Cheating on that line in any way will produce poor results. Not including all of those apps on that line will not update all the binstubs which will produce and incoherent set of binstubs. That sort of voids your warranty and will work up until it doesn't. There are numerous issues opened against ChefDK 0.x and 1.x that are the kinds of gem activation errors which will crop up:

• gem load error for chef-dk 1.2.22 on mac #1187
• berks-api fails due to json gem version conflict #281
• berks stops working after bundle install (ChefDK) berkshelf/berkshelf#1357
• https://stackoverflow.com/questions/31906153/chef-gemconflicterror-when-running-knife-bootstrap-windows
• berks upload fails with gem error on 0.10.0 #603

The aggressive gem pinning in ChefDK >= 2.0 is what has made those kinds of issues entirely go away. That kind of "can't activate net-ssh-4.0.1, already activated net-ssh-3.2.0 (Gem::LoadError)" is indicative of the problem. Nobody else in the world is going to be likely to be able to help you.

So we've established that the process outlined about requires being an expert in ruby and bundler usage and wrangling depsolving, and that it needs to be followed largely to the letter. The better alternative is to simply use bundler and create Gemfiles that can be orders of magnitude more simple. For example to manage chef-client 12.x with ChefDK 3.x you do not need to patch chef-client 12 into the ChefDK 3.x. Most of the tooling can be configured to install and test against chef 12 just fine (knife boostrap and test-kitchen). The only outlier is chefspec. To solve the chefspec problem, then tactically just for that use case construct a Gemfile:

source "https://rubygems.org"

gem "chef", "~> 12.0"
gem "berkshelf"
gem "chefspec"

Then just bundle install that and bundle exec rspec against that bundle. You will get old berskhelf 6.x because new berkshelf 7.x has a hard requirement on bugfixes in the chef-13 codebase.

So I can't tell anyone not to hack up their appbundled binstubs, but if you do that without understanding the consequences and following the correct procedure you are likely to hit issues that nobody will be able to help you debug. And if you have the capacity and expertise to correctly follow these instructions it is vastly simpler to just use bundler and roll your own specific Gemfiles. The other alternatives are of course to either upgrade your infrastructure to get off of old software, or to just be patient until new ChefDK releases drop.

[lamont-granquist]

The other obvious thing here is that the best way to reproduce a build is to simply check out the ChefDK source repo, update the Gemfile and create a Gemfile.lock with all the correct constraints and then use the omnibus project to build a new artifact. That is by-far the sanest way to go. But at any given time master of ChefDK may be horribly broken and not build on its own, so it is necessary to find the appropriate starting point to fork off of in the git history. And then the gems may not depsolve sanely depending on what the user is trying to do (this is the same problem as above -- gotta produce a Gemfile.lock which is the whole hard bit) and then the user needs to be careful that the gems all work with the ruby version which is going to be built by the omnibus project. There's a lot of flexibility there but also a million ways to beat your head against dependency issues. For simple relatively small tweaks around a stable base this can work. But, for example, trying to get Chef-12.old client installed into a modern ChefDK4 with ruby 2.6.x cannot work due to the patches for the newer ruby version not existing on a chef client that old.