Git Gems Don't Load DevKit Under Windows And Fail

[warhammerkid]

Steps:

1. Install RubyInstaller on Windows
2. Install DevKit
3. Create Gemfile with following contents: gem 'rdiscount', :git => 'https://github.com/rtomayko/rdiscount.git'
4. Run bundle install
5. It fails to compile extensions

Notes:
DevKit appears to be properly loaded if you just do gem 'rdiscount' in your Gemfile, but for gems that are loaded from git, DevKit doesn't load and it fails.

[indirect]

I have no idea what DevKit does to make gems work, but I'm sure Bundler doesn't do it for git gems. They're not gems. If you need DevKit to load your gems, I think you need to build a gem and put it in vendor/cache.

[warhammerkid]

So after several hours of research I've determined what is going on and the potential solution. DevKit uses Gem.pre_install to check and see if the gem spec has native extensions and then update the PATH and environment variables so the C compiler and all the other build necessities are available.

# rubygems/defaults/operating_system.rb
# :DK-BEG: override 'gem install' to enable RubyInstaller DevKit usage
Gem.pre_install do |gem_installer|
  unless gem_installer.spec.extensions.empty?
    unless ENV['PATH'].include?('C:\\DevKit\\mingw\\bin') then
      Gem.ui.say 'Temporarily enhancing PATH to include DevKit...' if Gem.configuration.verbose
      ENV['PATH'] = 'C:\\DevKit\\bin;C:\\DevKit\\mingw\\bin;' + ENV['PATH']
    end
    ENV['RI_DEVKIT'] = 'C:\\DevKit'
    ENV['CC'] = 'gcc'
    ENV['CPP'] = 'cpp'
    ENV['CXX'] = 'g++'
  end
end
# :DK-END:

In the RubyGems installer process it iterates through its list of pre-install hooks and runs them before building extensions and generating bins (rubygems/installer.rb). If the pre-install hooks were run, then native extensions for git sourced gems would build properly under windows. I modified source.rb to include those lines, and was able to successfully build warhammerkid/rocket-amf:

# source.rb
    installer = Installer.new(spec, :env_shebang => false)
+   Gem.pre_install_hooks.each do |hook|
+     hook.call installer
+   end
    installer.build_extensions
    installer.generate_bin

Since there doesn't seem to be an easy way to determine that the user needs the pre-install hooks to be run because DevKit needs to load, I'm not entirely comfortable submitting a patch that runs all the pre-install hooks. I didn't see anything that appeared to suggest that anyone besides the DevKit teams was using the pre-install hooks, so it's probably reasonable to assume that starting to run them won't cause problems, but I couldn't be sure. If you'd like, I'll submit a patch that runs the pre and post install hooks, but I didn't want to do the work if it would be rejected because it might break things.

[indirect]

A patch sounds great. Thanks for figuring it out!

On Apr 13, 2011, at 7:42 PM, warhammerkid reply@reply.github.com wrote:

So after several hours of research I've determined what is going on and the potential solution. DevKit uses Gem.pre_install to check and see if the gem spec has native extensions and then update the PATH and environment variables so the C compiler and all the other build necessities are available.

rubygems/defaults/operating_system.rb

:DK-BEG: override 'gem install' to enable RubyInstaller DevKit usage

Gem.pre_install do |gem_installer|
unless gem_installer.spec.extensions.empty?
unless ENV['PATH'].include?('C:\DevKit\mingw\bin') then
Gem.ui.say 'Temporarily enhancing PATH to include DevKit...' if Gem.configuration.verbose
ENV['PATH'] = 'C:\DevKit\bin;C:\DevKit\mingw\bin;' + ENV['PATH']
end
ENV['RI_DEVKIT'] = 'C:\DevKit'
ENV['CC'] = 'gcc'
ENV['CPP'] = 'cpp'
ENV['CXX'] = 'g++'
end
end

:DK-END:

In the RubyGems installer process it iterates through its list of pre-install hooks and runs them before building extensions and generating bins (rubygems/installer.rb). If the pre-install hooks were run, then native extensions for git sourced gems would build properly under windows. I modified source.rb to include those lines, and was able to successfully build warhammerkid/rocket-amf:

source.rb

   installer = Installer.new(spec, :env_shebang => false)

• Gem.pre_install_hooks.each do |hook|

• hook.call installer
  

• end
  installer.build_extensions
  installer.generate_bin

Since there doesn't seem to be an easy way to determine that the user needs the pre-install hooks to be run because DevKit needs to load, I'm not entirely comfortable submitting a patch that runs all the pre-install hooks. I didn't see anything that appeared to suggest that anyone besides the DevKit teams was using the pre-install hooks, so it's probably reasonable to assume that starting to run them won't cause problems, but I couldn't be sure. If you'd like, I'll submit a patch that runs the pre and post install hooks, but I didn't want to do the work if it would be rejected because it might break things.

Reply to this email directly or view it on GitHub:
#1120 (comment)

[soulnafein]

When is this fixing going in?

[warhammerkid]

To write the tests, I need to evaluate some ruby code that establishes the pre-install or post-install hooks before bundle install. The bundle spec helper does not allow me to require my own files, the ruby helper doesn't seem to be the correct way to run bundle install after additional ruby code is run, and it doesn't seem "appropriate" to establish the hooks in the Gemfile. Does anyone have any suggestions for how I should run code in the same Ruby context as bundle install, or should I just use sys_exec to run bundle install with my own custom requires?

it "runs pre-install hooks" do
  build_git "foo" do |s|
    s.write "install_hooks.rb", <<-H
      require 'rubygems'
      Gem.pre_install_hooks << lambda do |inst|
        STDERR.puts "Ran pre-install hook: \#{inst.spec.full_name}"
      end
    H
  end
  gemfile <<-G
    gem "foo", :git => "#{lib_path('foo-1.0')}"
  G
  bundle_bin = File.expand_path('../../../bin/bundle', __FILE__)
  cmd = "#{Gem.ruby} -I#{lib} -r#{lib_path('foo-1.0/install_hooks.rb')} #{bundle_bin} install"
  sys_exec(cmd)
  err.should == "Ran pre-install hook: foo-1.0"
end

[indirect]

As soon as I have time to review it.

On Jun 29, 2011, at 5:45 AM, soulnafein reply@reply.github.com wrote:

When is this fixing going in?

Reply to this email directly or view it on GitHub:
#1120 (comment)