Allow pre-compiled binaries for ruby 3.1.0 #9566

Maaarcocr · 2022-03-02T10:49:18Z

Why?

What are the changes?

I added 3.1.1 as a RUBY_CC_VERSION. I had to upgrade rake-compiler-dock to 1.2.0 as this is the only version of this gem that supports compilation for ruby 3.1+

Furthermore, I had to add a bit of logic (and a new platform this compiles to) as since RubyInstaller 3.1 x64 Windows is not x64-mingw32 but rather x64-mingw-ucrt as explained here

Also x64-mingw-ucrt only supports 3.1+ whereas x64-mingw32 only supports anything up to 3.0 so a little if statement is needed.

Maaarcocr · 2022-03-02T13:05:10Z

@elharo (or anyone else who can) I saw you added the ruby label, could you please also add release notes: yes as a label? (I think it makes sense to add the PR description to the next release notes, as it would let people now that this package now ships with 3.1 compatible compiled shims).

I cannot add any labels as I do not have the privileges to do so :/

rochlefebvre · 2022-03-02T15:54:48Z

ruby/Rakefile

+    ['x86-mingw32', 'x64-mingw32', 'x86_64-linux', 'x86-linux', 'x64-mingw-ucrt'].each do |plat|
+      # x64-mingw-ucrt only supports 3.1+ whereas 'x64-mingw32' only supports up to 3.0
+      versions = if plat == 'x64-mingw-ucrt'
+        '3.1.1'


I notice that the other versions are minor versions with a .0 as the patch, e.g. 3.0.0, 2.7.0... Your Ruby 3.1 version string is written as 3.1.1 instead of 3.1.0.

Is that intentional?

I went for the latest version of ruby. I'm not sure if it would make any difference, I could use 3.1.0 if it also would support 3.1.1 (not sure about this)

I think it's worth a shot, if only for consistency.

rochlefebvre · 2022-03-02T16:02:33Z

ruby/google-protobuf.gemspec

  end
  s.test_files  = ["tests/basic.rb",
                  "tests/stress.rb",
                  "tests/generated_code_test.rb"]
  s.required_ruby_version = '>= 2.3'
-  s.add_development_dependency "rake-compiler", "~> 1.1.0"
+  s.add_development_dependency "rake-compiler", "~> 1.2.0"


rake-compiler does not have a 1.2.0 release yet: https://rubygems.org/gems/rake-compiler. Did you mean to move up to the latest release, 1.1.9?

I think I may have just mistakenly changed this. Fixed it back to what it was, as it also should support the latest (not that it's needed I think)

Maaarcocr · 2022-03-02T16:20:03Z

I did not noticed that more work would be needed to setup CI for both testing on ruby 31 and releasing. I've added the changes to fix this in my latest PR.

rochlefebvre · 2022-03-02T17:06:12Z

Looking at the logs for MacOSRuby Release I see this bit, toward the end:

installing bundled gems:            /Users/kbuilder/.rake-compiler/ruby/x86_64-darwin/ruby-3.1.0/lib/ruby/gems/3.1.0
                                    minitest 5.15.0
                                    power_assert 2.0.1
                                    rake 13.0.6
                                    test-unit 3.5.3
                                    rexml 3.2.5
                                    rss 0.2.9
                                    net-ftp 0.1.3
                                    net-imap 0.2.2
                                    net-pop 0.1.1
                                    net-smtp 0.3.1
                                    matrix 0.4.2
                                    prime 0.1.2
                                    rbs 2.0.0
Building native extensions. This could take a while...
/Users/kbuilder/.rake-compiler/sources/ruby-3.1.0/lib/rubygems/ext/builder.rb:95:in `run': ERROR: Failed to build gem native extension. (Gem::Ext::BuildError)

    current directory: /Users/kbuilder/.rake-compiler/ruby/x86_64-darwin/ruby-3.1.0/lib/ruby/gems/3.1.0/gems/rbs-2.0.0/ext/rbs_extension
/Users/kbuilder/.rvm/rubies/ruby-2.7.0/bin/ruby --disable\\=gems -I/Users/kbuilder/.rake-compiler/builds/x86_64-darwin/ruby-3.1.0 -I /Users/kbuilder/.rake-compiler/sources/ruby-3.1.0/lib -r ./siteconf20220302-86972-osxhy4.rb extconf.rb
/Users/kbuilder/.rake-compiler/builds/x86_64-darwin/ruby-3.1.0/rbconfig.rb:13:in `<module:RbConfig>': ruby lib version (3.1.0) doesn't match executable version (2.7.0) (RuntimeError)
	from /Users/kbuilder/.rake-compiler/builds/x86_64-darwin/ruby-3.1.0/rbconfig.rb:11:in `<top (required)>'
	from /Users/kbuilder/.rake-compiler/ruby/x86_64-darwin/ruby-3.1.0/lib/ruby/gems/3.1.0/gems/rbs-2.0.0/ext/rbs_extension/siteconf20220302-86972-osxhy4.rb:1:in `require'
	from /Users/kbuilder/.rake-compiler/ruby/x86_64-darwin/ruby-3.1.0/lib/ruby/gems/3.1.0/gems/rbs-2.0.0/ext/rbs_extension/siteconf20220302-86972-osxhy4.rb:1:in `<top (required)>'
	from extconf.rb:in `require'

extconf failed, exit code 1

While building native extensions for gems bundled in Ruby 3.1, it complains that we're running a 2.7 interpreter with a 3.1 library.

I see that ruby_build_environment.sh defaults to Ruby 2.7 in a few spots. I'm unsure if that is related somehow?

Maaarcocr · 2022-03-02T17:16:52Z

While building native extensions for gems bundled in Ruby 3.1, it complains that we're running a 2.7 interpreter with a 3.1 library.

I see that ruby_build_environment.sh defaults to Ruby 2.7 in a few spots. I'm unsure if that is related somehow?

yeah I'm trying to understand how we build 3.0.0 and what's missing to get 3.1.0 working

Maaarcocr · 2022-03-02T18:06:57Z

I think that maybe I was overthinking the ucrt changes. I simplified the code to get rid of any logic specific to that and also made sure to run the 3.1 build for macos under ruby 3.1

acozzette · 2022-03-04T16:48:39Z

Sorry for the confusion on this, there is a Google-internal config we need to add before the 3.1 test runs are fully enabled. I can take care of this after the PR is merged.

acozzette · 2022-03-05T00:48:44Z

@Maaarcocr I rebuilt the Docker image, so that should fix most or all of the failing test runs. The change looks good to me overall but I want to look more closely at the changes to ruby_build_environment.sh on Monday. Thank you for all your work on this!

acozzette

Looks good to me but I guess we will still need to wait for your rake-compiler-dock change to be released.

rochlefebvre · 2022-03-07T14:01:43Z

In the meantime, I was trying to figure out how to fix the linux release build, as maven was failing to run. I made a pr to rake-compiler-dock, as their current jruby docker image contains a version of maven that does not work with the version of jdk they use. rake-compiler/rake-compiler-dock#64

Hopefully I can get this merged soon and then I'll probably have to update this PR to use the new version.

I'm very happy to see that your rake-compiler-dock fix was accepted and merged 🎉 . While you wait for confirmation on a new release, may I suggest you temporarily depend on rake-compiler-dock's head? Something like:

# ruby/Gemfile
group :development do
  gem 'rake-compiler-dock', git: 'https://github.com/rake-compiler/rake-compiler-dock.git'
end

We'll see if this update fixes the Linux Ruby checks or if other changes are needed.

Maaarcocr · 2022-03-07T15:06:52Z

I'm very happy to see that your rake-compiler-dock fix was accepted and merged 🎉 . While you wait for confirmation on a new release, may I suggest you temporarily depend on rake-compiler-dock's head? Something like:
# ruby/Gemfile
group :development do
  gem 'rake-compiler-dock', git: 'https://github.com/rake-compiler/rake-compiler-dock.git'
end
We'll see if this update fixes the Linux Ruby checks or if other changes are needed.

I tried this, but the issue is that the docker image that is pulled still does not show the changes I made (I guess someone has to push a new image version). So we have to wait on a new release (of at least the docker image)

haberman · 2022-03-08T01:11:23Z

I tried this, but the issue is that the docker image that is pulled still does not show the changes I made (I guess someone has to push a new image version).

Is this kokoro/linux/dockerfile/test/ruby/Dockerfile?

I pulled your branch and tried running kokoro/linux/dockerfile/push_testing_images.sh, which usually updates our Docker images, but it didn't affect this one.

haberman · 2022-03-08T01:13:23Z

When I look at the latest test runs, everything seems to have passed except for "Linux Ruby Release ", which failed in a JRuby step (attn: @JasonLunn ).

[ERROR] Error executing Maven.
[ERROR] java.lang.IllegalStateException: Unable to load cache item
[ERROR] Caused by: Unable to load cache item
[ERROR] Caused by: Could not initialize class com.google.inject.internal.cglib.core.$MethodWrapper
rake aborted!
Command failed with status (1): [docker run -v /tmpfs/src/github/protobuf:/tmpfs/src/github/protobuf -e UID\=1000 -e GID\=1000 -e USER\=kbuilder -e GROUP\=kbuilder -e GEM_PRIVATE_KEY_PASSPHRASE -e ftp_proxy -e http_proxy -e https_proxy -e RCD_HOST_RUBY_PLATFORM\=x86_64-linux -e RCD_HOST_RUBY_VERSION\=2.4.1 -e RCD_IMAGE\=larskanis/rake-compiler-dock-jruby:1.2.0 -w /tmpfs/src/github/protobuf --rm -i larskanis/rake-compiler-dock-jruby:1.2.0 runas sigfw bash -c \ \ \ \ \ \ sudo\ apt-get\ install\ maven\ -y\ \&\&\ \ \ \ \ \ \ cd\ java\ \&\&\ mvn\ install\ -Dmaven.test.skip\=true\ \&\&\ cd\ ../ruby\ \&\&\ \ \ \ \ \ \ bundle\ \&\&\ \ \ \ \ \ \ IN_DOCKER\=true\ rake\ compile\ gem'
']
/usr/local/rvm/gems/ruby-2.4.1/bundler/gems/rake-compiler-dock-cd2afa24deca/lib/rake_compiler_dock/starter.rb:97:in `block in exec'
/usr/local/rvm/gems/ruby-2.4.1/bundler/gems/rake-compiler-dock-cd2afa24deca/lib/rake_compiler_dock/starter.rb:47:in `each'
/usr/local/rvm/gems/ruby-2.4.1/bundler/gems/rake-compiler-dock-cd2afa24deca/lib/rake_compiler_dock/starter.rb:47:in `exec'
/usr/local/rvm/gems/ruby-2.4.1/bundler/gems/rake-compiler-dock-cd2afa24deca/lib/rake_compiler_dock/starter.rb:14:in `sh'
/usr/local/rvm/gems/ruby-2.4.1/bundler/gems/rake-compiler-dock-cd2afa24deca/lib/rake_compiler_dock.rb:46:in `sh'
/tmpfs/src/github/protobuf/ruby/Rakefile:116:in `block in <top (required)>'
/usr/local/rvm/gems/ruby-2.4.1/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/rvm/gems/ruby-2.4.1/bin/ruby_executable_hooks:15:in `eval'
/usr/local/rvm/gems/ruby-2.4.1/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => gem:native => gem:java
(See full trace by running task with --trace)

This seems unrelated to this PR? I'll kick it one more time in case the failure was spurious.

acozzette · 2022-03-08T01:48:42Z

@haberman I believe that last failure is supposed to be fixed by @Maaarcocr's change to rake-compiler-dock, but it sounds like we need to wait for the fix to be released.

Maaarcocr · 2022-03-17T10:08:13Z

@haberman I believe that last failure is supposed to be fixed by @Maaarcocr's change to rake-compiler-dock, but it sounds like we need to wait for the fix to be released.

the changes just got released. hopefully this will fix all CIs.

acozzette · 2022-03-17T16:13:41Z

The codespell error is unrelated and I went ahead and fixed it separately.

haberman · 2022-03-17T16:39:07Z

We should cherry-pick this into the 3.20.x branch so it makes it into the 3.20.0 release.

* Allow pre-compiled binaries for ruby 3.1.1 * add comment * fix build and use ruby 3.1.0 * add ruby31 to build CI for tests and release * trying to fix ci * install ruby 3.1.0 in ruby_build_environment.sh * use head for rvm to install 3.1.0 * just install master version of rvm in prepare_build_macos_rc * force install of master rvm in ruby_build_environment.sh * Use coroutine=universal when compiling ruby31 * use ucontext * fix filename * fix coroutine name * use git head for rake-compiler-dock * use newest rake-compiler-dock version

* Allow pre-compiled binaries for ruby 3.1.0 (#9566) * Allow pre-compiled binaries for ruby 3.1.1 * add comment * fix build and use ruby 3.1.0 * add ruby31 to build CI for tests and release * trying to fix ci * install ruby 3.1.0 in ruby_build_environment.sh * use head for rvm to install 3.1.0 * just install master version of rvm in prepare_build_macos_rc * force install of master rvm in ruby_build_environment.sh * Use coroutine=universal when compiling ruby31 * use ucontext * fix filename * fix coroutine name * use git head for rake-compiler-dock * use newest rake-compiler-dock version * Updated CHANGES.txt for Ruby changes. * Fixed Ruby 3.1 tests by marking intersect? as unimplemented. (#9645) * Fixed Ruby 3.1 tests by marking intersect? as unimplemented. * Updated compatibility tests. Co-authored-by: Marco Concetto Rudilosso <marcoconcettorudilosso@gmail.com>

perezd · 2022-04-20T19:08:02Z

This change is creating fake arm64 ruby gems that are not actually arm64 (reporting as x86_64 Mach-O headers). What should we do about this given our pending release for 21.0?

haberman · 2022-04-20T19:14:12Z

Reference: #9804

Have arm64 binaries on macOS previously worked, and this PR broke them?

johnnyshields · 2022-05-13T19:07:22Z

I have the correct way (I think) of doing the mingw-ucrt here: #9963. Please kindly merge this so we can have Ruby 3.1 on Windows, thanks!

Allow pre-compiled binaries for ruby 3.1.1

91e7fe1

Maaarcocr mentioned this pull request Mar 2, 2022

Add Ruby 3.1 compability #9364

Closed

add comment

828b9f8

elharo requested a review from haberman March 2, 2022 12:34

elharo added the ruby label Mar 2, 2022

elharo added the release notes: yes label Mar 2, 2022

acozzette added the kokoro:run label Mar 2, 2022