[GR-48942] Cleanups for running C extensions natively

PullRequest: truffleruby/4060

Author: eregon

CHANGELOG.md

@@ -2,7 +2,7 @@
 
 New features:
 
-* C/C++ extensions are now compiled using the system toolchain and executed natively instead of using GraalVM LLVM (Sulong). This leads to faster startup, no warmup, better compatibility and faster installation for C/C++ extensions (#3118, @eregon).
+* C/C++ extensions are now compiled using the system toolchain and executed natively instead of using GraalVM LLVM (Sulong). This leads to faster startup, no warmup, better compatibility, smaller distribution and faster installation for C/C++ extensions (#3118, @eregon).
 
 Bug fixes:
 

ci.jsonnet

@@ -232,16 +232,7 @@ local part_definitions = {
   },
 
   platform: {
-    local devtoolset = { # Until there is a proper object in common.jsonnet for it
-      packages+: if self.os == "linux" then
-        (if self.arch == "aarch64" then {
-          "00:devtoolset": "==10",
-        } else {
-          "00:devtoolset": "==11",
-        })
-      else {},
-    },
-    local common_deps = common.deps.truffleruby + common.deps.sulong + devtoolset,
+    local common_deps = common.deps.truffleruby + common.deps.sulong,
 
     linux: common.linux_amd64 + common_deps + {
       platform_name:: "LinuxAMD64",
@@ -377,6 +368,7 @@ local part_definitions = {
     },
 
     test_cexts_sulong: {
+      mx_env:: "toolchain",
       environment+: {
         TRUFFLERUBYOPT: "--experimental-options --cexts-sulong",
       },

doc/contributor/docker.md

@@ -50,8 +50,6 @@ jt docker test --filter debian --standalone $PATH_TO_STANDALONE_TAR_GZ --test re
 For example, to run a full set of tests on a set of new release candidate tarballs:
 
 ```bash
-jt docker test --graalvm graalvm-ce.tar.gz llvm-toolchain-installable.jar:ruby-installable-ce.jar --test release_branch
-jt docker test --graalvm graalvm-ee.tar.gz llvm-toolchain-installable.jar:ruby-installable-ee.jar --test release_branch
 jt docker test --standalone truffleruby-linux-amd64.tar.gz --test release_branch
 ```
 
@@ -66,10 +64,7 @@ Pick one of the distributions in [docker-configs.yaml](../../tool/docker-configs
 
 ## Methods of installing
 
-Pick one of:
-
-* From a GraalVM binary tarball and Ruby component you have locally, `--graalvm graalvm.tar.gz llvm-toolchain-installable.jar:ruby-installable.jar`
-* From a TruffleRuby standalone distribution you have locally, `--standalone truffleruby-1.0.0-linux-amd64.tar.gz`
+From a TruffleRuby standalone distribution you have locally, `--standalone truffleruby-1.0.0-linux-amd64.tar.gz`
 
 ## What to do
 

doc/contributor/workflow.md

@@ -127,20 +127,6 @@ This warning is important.
 - If you want to set the `suite.py` import to that checked out in `graal` (unlikely), you should run 
   jt mx scheckimports` beforehand.
 
-### Building C Extensions more quickly
-
-Speeding up compilation of C extensions can be achieved by using *native* toolchain launchers,
-instead of the Bash toolchain launchers used with the `jvm*` build configurations.
-
-For bundled C extensions (that is, extensions under `src/main/c`),
-you can `export JT_CACHE_TOOLCHAIN=true` and that will then use native toolchain launchers for the
-[bootstrap toolchain](https://github.com/oracle/graal/blob/master/sulong/docs/contributor/TOOLCHAIN.md#using-a-prebuilt-graalvm-as-a-bootstrapping-toolchain).
-This has no effect after `jt build`.
-
-For C extensions installed after `jt build` by e.g. `gem install` or `bundle install`,
-one can use the `jvm-ce-ntl` build configuration which includes *native* toolchain launchers,
-or use one of the `native*` build configuration (which also builds a native truffleruby launcher).
-
 ## Editors and IDEs
 
 * [Using the IntelliJ IDE for Development](using-intellij.md) (recommended)

doc/user/installing-llvm.md

@@ -6,11 +6,8 @@ permalink: /reference-manual/ruby/InstallingLLVM/
 ---
 # Installing Make and GCC
 
-Since TruffleRuby 19.3.0, TruffleRuby ships with its own LLVM toolchain.
-Therefore, it is no longer necessary to install LLVM.
-If you are using an older version, see [the documentation for that version](https://github.com/oracle/truffleruby/blob/vm-19.2.0/doc/user/installing-llvm.md).
-
-The `make` utility as well as the dependencies of the toolchain still need to be available to build C and C++ extensions.
+Since TruffleRuby 24.0.0, TruffleRuby no longer needs a LLVM toolchain and instead uses the system toolchain.
+The packages below are required to build C and C++ extensions.
 
 ### Fedora-based: RHEL, Oracle Linux, etc
 
@@ -24,28 +21,10 @@ sudo dnf install make gcc
 sudo apt-get install make gcc
 ```
 
-### Mandriva-based and other Linux distributions
-
-Note: Such distributions are not tested and not [supported](../../README.md#system-compatibility).
-
-First, install the `make` and `gcc` dependencies.
-
-Mandriva uses a not-yet-upstreamed patch to let `clang` find the GCC installation (see [this comment](https://github.com/oracle/truffleruby/issues/2009#issuecomment-630019082)).
-Therefore the internal LLVM toolchain cannot find the necessary `libgcc_s` by default.
-The proper fix is for those distributions to upstream their changes to LLVM.
-
-A workaround is to create a symlink explicitly so that the LLVM toolchain can find `libgcc_s`:
-```bash
-cd /usr/lib/gcc
-sudo ln -s x86_64-mandriva-linux-gnu x86_64-linux-gnu
-```
-
 ### macOS
 
 On macOS, make sure you have installed the command line developer tools from Xcode:
 
 ```bash
 xcode-select --install
 ```
-
-You might need to add `export SDKROOT=$(xcrun --show-sdk-path)` in your shell profile.

doc/user/polyglot.md

@@ -28,7 +28,7 @@ The JVM configuration automatically has access to other languages.
 To use other GraalVM languages, you need the [JVM Standalone](../../README.md#getting-started).
 The Native Standalone does not support installing extra languages.
 
-Note that `ruby`, `llvm`, the LLVM toolchain and host Java interop are available without installing anything extra.
+Note that `ruby`, `llvm` and host Java interop are available without installing anything extra.
 
 Then you can install other languages with `truffleruby-polyglot-get $LANGUAGE`, for instance:
 ```bash

lib/cext/include/truffleruby/truffleruby-abi-version.h

@@ -10,6 +10,6 @@
 // $RUBY_VERSION must be the same as TruffleRuby.LANGUAGE_VERSION.
 // $ABI_NUMBER starts at 1 and is incremented for every ABI-incompatible change.
 
-#define TRUFFLERUBY_ABI_VERSION "3.2.2.5"
+#define TRUFFLERUBY_ABI_VERSION "3.2.2.6"
 
 #endif

lib/mri/mkmf.rb

@@ -48,10 +48,6 @@
   end
 end
 
-if defined?(::TruffleRuby) and Truffle::Boot.get_option('cexts-prepend-toolchain-to-path') and Truffle::Boot.get_option('cexts-sulong')
-  ENV['PATH'] = "#{RbConfig::CONFIG['toolchain_path']}:#{ENV['PATH']}"
-end
-
 class String # :nodoc:
   # Wraps a string in escaped quotes if it contains whitespace.
   def quote
@@ -2891,19 +2887,11 @@ def MAIN_DOES_NOTHING(*refs)
     "$(CFLAGS) $(src) $(LIBPATH) $(LDFLAGS) $(ARCH_FLAG) $(LOCAL_LIBS) $(LIBS)"
 
   if defined?(::TruffleRuby)
-    # We need to link to libtruffleruby for MakeMakefile#try_link to succeed.
-    # The created executable will link against both libgraalvm-llvm.so and libtruffleruby and
-    # might be executed for #try_constant and #try_run so we also need -rpath for both.
-    # TODO: could likely always use the second branch here
-    if Truffle::Boot.get_option('cexts-sulong')
-      libtruffleruby_dir = File.dirname(RbConfig::CONFIG['libtruffleruby'])
-      TRY_LINK << " -L#{libtruffleruby_dir} -rpath #{libtruffleruby_dir} -ltruffleruby"
-      libgraalvm_llvm_dir = ::Truffle::Boot.toolchain_paths(:LD_LIBRARY_PATH)
-      TRY_LINK << " -rpath #{libgraalvm_llvm_dir}"
-    else
-      libtrufflerubytrampoline_dir = File.dirname(RbConfig::CONFIG['libtrufflerubytrampoline'])
-      TRY_LINK << " -L#{libtrufflerubytrampoline_dir} -Wl,-rpath,#{libtrufflerubytrampoline_dir} -ltrufflerubytrampoline"
-    end
+    # We need to link to libtrufflerubytrampoline for MakeMakefile#try_link to succeed.
+    # The created executable will link against libtrufflerubytrampoline and
+    # might be executed for #try_constant and #try_run so we also need -rpath.
+    libtrufflerubytrampoline_dir = File.dirname(RbConfig::CONFIG['libtrufflerubytrampoline'])
+    TRY_LINK << " -L#{libtrufflerubytrampoline_dir} -Wl,-rpath,#{libtrufflerubytrampoline_dir} -ltrufflerubytrampoline"
   end
 
   ##

lib/truffle/rbconfig.rb

@@ -72,15 +72,13 @@ module RbConfig
     strip = Truffle::Boot.toolchain_executable(:STRIP)
 
     strip = "#{strip} --keep-section=.llvmbc" unless Truffle::Platform.darwin?
-    gcc, clang = false, true
   else
     if Truffle::Platform.linux?
       ar = 'gcc-ar'
       cc = 'gcc' # -std=gnu99 ?
       cxx = 'g++'
       ranlib = 'gcc-ranlib'
       strip = 'strip -S -x'
-      gcc, clang = true, false
     elsif Truffle::Platform.darwin?
       ar = 'ar'
       cc = 'clang'
@@ -90,7 +88,6 @@ module RbConfig
       # This is notably necessary for grpc where the current logic does not append -x for TruffleRuby:
       # https://github.com/grpc/grpc/blob/54f65e0dbd2151a3ba2ad364327c0c31b200a5ae/src/ruby/ext/grpc/extconf.rb#L125-L126
       strip = 'strip -A -n -x'
-      gcc, clang = false, true
       cflags_pre = '-fdeclspec '
     else
       raise 'Unknown platform'
@@ -107,9 +104,6 @@ module RbConfig
     '-Wno-incompatible-pointer-types', # Fix byebug 8.2.1 compile (st_data_t error)
     '-Wno-format-extra-args',          # Our PRIsVALUE generates this because compilers ignore printf extensions
   ]
-  warnflags << '-Wno-format-invalid-specifier' if clang # Our PRIsVALUE generates this because compilers ignore printf extensions
-  # TODO fix it, happens in openssl
-  warnflags << '-Wno-discarded-qualifiers' if gcc
 
   defs = ''
   cppflags = ''
@@ -119,8 +113,6 @@ module RbConfig
   cext_dir = "#{prefix}/lib/cext"
   soext = Truffle::Platform::SOEXT
 
-  toolchain_path = Truffle::Boot.toolchain_paths(:PATH)
-
   # Make C extensions use the same libssl as the one used for the openssl C extension
   if Truffle::Platform.darwin?
     require 'truffle/openssl-prefix'
@@ -225,7 +217,6 @@ module RbConfig
     'sysconfdir'        => "#{prefix}/etc", # doesn't exist, as in MRI
     'target_cpu'        => host_cpu,
     'target_os'         => host_os,
-    'toolchain_path'    => toolchain_path,
     'UNICODE_VERSION'   => Primitive.encoding_unicode_version,
     'UNICODE_EMOJI_VERSION' => Primitive.encoding_unicode_emoji_version,
     'warnflags'         => warnflags,

lib/truffle/truffle/cext.rb

@@ -2112,18 +2112,26 @@ def rb_imemo_tmpbuf_set_ptr(obj, ptr)
     obj.pointer = ptr
   end
 
-  def rb_debug_inspector_open_contexts
-    Truffle::Debug.get_frame_bindings.map do |binding|
+  RB_THREAD_CURRENT_BACKTRACE_LOCATIONS_OMIT = NFI ? 4 : 5
+
+  def rb_debug_inspector_open_contexts_and_backtrace
+    contexts = Truffle::Debug.get_frame_bindings.map do |binding|
       if binding
         [binding.receiver, Primitive.class(binding.receiver), binding]
       else
         [nil, nil, nil]
       end
     end.freeze
-  end
 
-  def rb_thread_current_backtrace_locations
-    Thread.current.backtrace_locations(4)
+    backtrace = Thread.current.backtrace_locations(RB_THREAD_CURRENT_BACKTRACE_LOCATIONS_OMIT)
+
+    if contexts.size != backtrace.size
+      STDERR.puts '', 'contexts:', contexts.map { |_,_,binding| binding.source_location.join(':') }
+      STDERR.puts '', 'backtrace:', backtrace, ''
+      raise 'debug_inspector contexts and backtrace lengths are not equal'
+    end
+
+    [contexts, backtrace]
   end
 
   def rb_syserr_new(errno, mesg)

lib/truffle/truffle/cext_constants.rb

@@ -8,7 +8,7 @@
 # GNU General Public License version 2, or
 # GNU Lesser General Public License version 2.1.
 
-# From tool/generate-cext-constants.rb
+# GENERATED BY tool/generate-cext-constants.rb
 
 module Truffle::CExt
   def rb_cArray

lib/truffle/truffle/cext_ruby.rb

@@ -20,49 +20,26 @@ def rb_define_method(mod, name, function, argc)
       raise ArgumentError, "arity out of range: #{argc} for -2..15"
     end
 
-    if NFI
-      wrapper = RB_DEFINE_METHOD_WRAPPERS[argc]
-      method_body = Truffle::Graal.copy_captured_locals -> *args, &block do
-        if argc == -1 # (int argc, VALUE *argv, VALUE obj)
-          args = [function, args.size, Truffle::CExt.RARRAY_PTR(args), Primitive.cext_wrap(self)]
-        elsif argc == -2 # (VALUE obj, VALUE rubyArrayArgs)
-          args = [function, Primitive.cext_wrap(self), Primitive.cext_wrap(args)]
-        elsif argc >= 0 # (VALUE obj); (VALUE obj, VALUE arg1); (VALUE obj, VALUE arg1, VALUE arg2); ...
-          if args.size != argc
-            raise ArgumentError, "wrong number of arguments (given #{args.size}, expected #{argc})"
-          end
-          args = [function, Primitive.cext_wrap(self), *args.map! { |arg| Primitive.cext_wrap(arg) }]
+    wrapper = RB_DEFINE_METHOD_WRAPPERS[argc]
+    method_body = Truffle::Graal.copy_captured_locals -> *args, &block do
+      if argc == -1 # (int argc, VALUE *argv, VALUE obj)
+        args = [function, args.size, Truffle::CExt.RARRAY_PTR(args), Primitive.cext_wrap(self)]
+      elsif argc == -2 # (VALUE obj, VALUE rubyArrayArgs)
+        args = [function, Primitive.cext_wrap(self), Primitive.cext_wrap(args)]
+      elsif argc >= 0 # (VALUE obj); (VALUE obj, VALUE arg1); (VALUE obj, VALUE arg1, VALUE arg2); ...
+        if args.size != argc
+          raise ArgumentError, "wrong number of arguments (given #{args.size}, expected #{argc})"
         end
-
-        exc = $!
-        Primitive.thread_set_exception(nil)
-        # We must set block argument if given here so that the
-        # `rb_block_*` functions will be able to find it by walking the stack.
-        res = Primitive.call_with_c_mutex_and_frame_and_unwrap(wrapper, args, Primitive.caller_special_variables_if_available, block)
-        Primitive.thread_set_exception(exc)
-        res
+        args = [function, Primitive.cext_wrap(self), *args.map! { |arg| Primitive.cext_wrap(arg) }]
       end
-    else
-      method_body = Truffle::Graal.copy_captured_locals -> *args, &block do
-        if argc == -1 # (int argc, VALUE *argv, VALUE obj)
-          args = [args.size, Truffle::CExt.RARRAY_PTR(args), Primitive.cext_wrap(self)]
-        elsif argc == -2 # (VALUE obj, VALUE rubyArrayArgs)
-          args = [Primitive.cext_wrap(self), Primitive.cext_wrap(args)]
-        elsif argc >= 0 # (VALUE obj); (VALUE obj, VALUE arg1); (VALUE obj, VALUE arg1, VALUE arg2); ...
-          if args.size != argc
-            raise ArgumentError, "wrong number of arguments (given #{args.size}, expected #{argc})"
-          end
-          args = [Primitive.cext_wrap(self), *args.map! { |arg| Primitive.cext_wrap(arg) }]
-        end
 
-        exc = $!
-        Primitive.thread_set_exception(nil)
-        # We must set block argument if given here so that the
-        # `rb_block_*` functions will be able to find it by walking the stack.
-        res = Primitive.call_with_c_mutex_and_frame_and_unwrap(function, args, Primitive.caller_special_variables_if_available, block)
-        Primitive.thread_set_exception(exc)
-        res
-      end
+      exc = $!
+      Primitive.thread_set_exception(nil)
+      # We must set block argument if given here so that the
+      # `rb_block_*` functions will be able to find it by walking the stack.
+      res = Primitive.call_with_c_mutex_and_frame_and_unwrap(wrapper, args, Primitive.caller_special_variables_if_available, block)
+      Primitive.thread_set_exception(exc)
+      res
     end
 
     # Even if the argc is -2, the arity number

mx.truffleruby/env_files.md

@@ -1,16 +1,14 @@
 Here is how the various env files relate to each other:
 * `jvm`
   * `jvm-ce`: + GraalVM Community Compiler
-    * `jvm-ce-ntl`: + native toolchain launchers
-      * `jvm-ce-libgraal`: + libgraal
-      * `native`: + librubyvm + `Truffle Macro`
-        * `native-host-inlining`: + `TruffleHostInliningPrintExplored`, - native toolchain launchers
-        * `native-profiling`: + `-H:-DeleteLocalSymbols`
+    * `jvm-ce-libgraal`: + libgraal
+    * `native`: + librubyvm + `Truffle Macro`
+      * `native-host-inlining`: + `TruffleHostInliningPrintExplored`
+      * `native-profiling`: + `-H:-DeleteLocalSymbols`
   * `jvm-ee`: + Oracle GraalVM Compiler + `Truffle enterprise` + license + `LLVM Runtime Native Enterprise`
-    * `jvm-ee-ntl`: + native toolchain launchers
-      * `jvm-ee-libgraal`: + libgraal
-      * `native-ee`: + librubyvm + `Truffle Macro Enterprise` + Native Image G1
-        * `native-ee-host-inlining`: + `TruffleHostInliningPrintExplored`, - native toolchain launchers
-        * `native-ee-aux`: + `AuxiliaryEngineCache`, - Native Image G1 (currently incompatible)
+    * `jvm-ee-libgraal`: + libgraal
+    * `native-ee`: + librubyvm + `Truffle Macro Enterprise` + Native Image G1
+      * `native-ee-host-inlining`: + `TruffleHostInliningPrintExplored`
+      * `native-ee-aux`: + `AuxiliaryEngineCache`, - Native Image G1 (currently incompatible)
   * `jvm-js`: + Graal.js
   * `jvm-py`: + GraalPython

mx.truffleruby/jvm-ce-libgraal

@@ -1,7 +1,7 @@
 GRAALVM_SKIP_ARCHIVE=true
 DYNAMIC_IMPORTS=/tools,/compiler,/substratevm
 COMPONENTS=TruffleRuby,suite:tools,GraalVM compiler,SubstrateVM,LibGraal
-NATIVE_IMAGES=suite:sulong,lib:jvmcicompiler
+NATIVE_IMAGES=lib:jvmcicompiler
 # To also create the standalone
 INSTALLABLES=TruffleRuby
 BUILD_TARGETS=GRAALVM,GRAALVM_STANDALONES

mx.truffleruby/jvm-ce-ntl

@@ -1,7 +1,7 @@
 GRAALVM_SKIP_ARCHIVE=true
 DYNAMIC_IMPORTS=/tools,/truffleruby-enterprise,/graal-enterprise,/vm-enterprise,/sulong-managed,/substratevm-enterprise
 COMPONENTS=TruffleRuby,suite:tools,GraalVM enterprise compiler,Truffle enterprise,GraalVM enterprise license files,LLVM Runtime Native Enterprise,SubstrateVM Enterprise,LibGraal Enterprise
-NATIVE_IMAGES=suite:sulong,lib:jvmcicompiler
+NATIVE_IMAGES=lib:jvmcicompiler
 # To also create the standalone
 INSTALLABLES=TruffleRuby
 BUILD_TARGETS=GRAALVM,GRAALVM_STANDALONES