Change side-by-side DLL loading to store dependencies in each extensi…

…on.so file

So far ruby.exe and rubyw.exe contained a process-global manifest for side-by-side loading.
This way the DLLs bundled to RubyInstaller could be moved to a dedicated directory, so that they aren't loaded accidently by other apps because they are in the PATH.

It was introduced in b2bd630
The downside of a global manifest is that the dependent DLLs are always preferred over other DLL versions with the same name.
This caused for instance openssl.gem to fail, when the libssl DLL linked at build time was newer than the libssl bundled with RubyInstaller.

This patch introduces manifests per extension.so file with dedicated private dependencies.
This way the bundled openssl.so is directly linked to it's own libssl.dll by an embedded manifest.
It's similar to static linking libssl into openssl.so, but it allows to use unchanged libssl.dll from MINGW packages.

If a new openssl.gem version is installed per "gem install openssl", it links to MSYS2/MINGW packages at build time.
Since the resulting openssl.so doesn't contain the manifest, it also links to the MINGW packages at run time.

Patching extension.so files requires to add all single files recursively as file tasks.

Fixes #60

lib/ruby_installer/build.rb

@@ -5,6 +5,7 @@ module Build
   autoload :ComponentsInstaller, 'ruby_installer/build/components_installer'
   autoload :DllDirectory, 'ruby_installer/build/dll_directory'
   autoload :ErbCompiler, 'ruby_installer/build/erb_compiler'
+  autoload :ManifestUpdater, 'ruby_installer/build/manifest_updater'
   autoload :Msys2Installation, 'ruby_installer/build/msys2_installation'
   autoload :GEM_VERSION, 'ruby_installer/build/gem_version'
   autoload :Task, 'ruby_installer/build/task'

lib/ruby_installer/build/manifest_updater.rb

@@ -0,0 +1,35 @@
+module RubyInstaller
+  module Build
+    class ManifestUpdater
+      def self.update_file(from_fname, manifest_xml_string, to_fname)
+        image = File.binread(from_fname)
+        update_blob(image, manifest_xml_string, filename: from_fname)
+        File.binwrite(to_fname, image)
+      end
+
+      def self.update_blob(dll_or_exe_data, manifest_xml_string, filename: nil)
+        # There are two regular options to add a custom manifest:
+        # 1. Change a given exe file per Microsofts "mt.exe" after the build
+        # 2. Specify a the manifest while linking with the MINGW toolchain
+        #
+        # Since we don't want to depend on particular Microsoft tools and want to avoid additional patching of the ruby build, we do a nifty trick here.
+        # We patch the exe file manually.
+        # Removing unnecessary spaces and comments from the embedded XML manifest gives us enough space to add the above XML elements.
+        # Then the default MINGW manifest gets replaced by our custom XML content.
+        # The rest of the available bytes is simply padded with spaces, so that we don't change positions within the EXE image.
+        success = false
+        dll_or_exe_data.gsub!(/<\?xml.*?<assembly.*?<\/assembly>\n/m) do |m|
+          success = true
+          newm = m.gsub(/^\s*<\/assembly>\s*$/, manifest_xml_string + "</assembly>")
+            .gsub(/<!--.*?-->/m, "")
+            .gsub(/^ +/, "")
+            .gsub(/\n+/m, "\n")
+
+          raise "replacement manifest too big #{m.bytesize} < #{newm.bytesize}" if m.bytesize < newm.bytesize
+          newm + " " * (m.bytesize - newm.bytesize)
+        end
+        raise "no manifest found#{ "in #{filename}" if filename}" unless success
+      end
+    end
+  end
+end

recipes/installer-inno/50-generate-filelist.rake

@@ -13,16 +13,12 @@ file filelist_iss => [__FILE__, ovl_expand_file(sandbox_task.sandboxfile_listfil
     else
       components = "ruby"
     end
-    args = if File.directory?(path)
-      flags = "recursesubdirs createallsubdirs #{flags}"
-      source = "../../#{path}/*"
-      dest = "{app}/#{reltosandbox_path}"
-    else
+    unless File.directory?(path)
       source = "../../#{path}"
       dest = "{app}/#{File.dirname(reltosandbox_path)}"
+      "Source: #{source}; DestDir: #{dest}; Flags: #{flags}; Components: #{components}"
     end
 
-    "Source: #{source}; DestDir: #{dest}; Flags: #{flags}; Components: #{components}"
   end.join("\n")
   File.write(filelist_iss, out)
 end

recipes/sandbox/50-gather-sandbox-files.rake

@@ -4,3 +4,14 @@ sandboxfiles_rel = File.readlines(ovl_expand_file(sandboxfile_listfile)) + File.
 sandboxfiles_rel = sandboxfiles_rel.map{|path| path.chomp }
 sandboxfiles_rel += import_files.values
 self.sandboxfiles += sandboxfiles_rel.map{|path| File.join(sandboxdir, path)}
+# go through directories and gather all files recursively
+self.sandboxfiles = sandboxfiles.flat_map do |path|
+  unpack_path = path.sub(sandboxdir, unpackdirmgw)
+  if File.directory?(unpack_path)
+    Dir.glob(File.join(unpack_path, "**/*")).map do |pa|
+      pa.sub(unpackdirmgw, sandboxdir)
+    end
+  else
+    path
+  end
+end

recipes/sandbox/60-side-by-side-assembly.rake

@@ -12,7 +12,95 @@ dlls = self.sandboxfiles.select do |destpath|
   destpath.start_with?(bin_dir+"/") && destpath =~ /\.dll$/i && destpath !~ libruby_regex
 end
 
-dlls.each do |destpath|
+ext_dll_defs = {
+  "lib/ruby/#{package.rubylibver}/#{package.ruby_arch}/fiddle.so" => /^libffi-\d.dll$/,
+  "lib/ruby/#{package.rubylibver}/#{package.ruby_arch}/openssl.so" => /^libssl-[\d_]+(-x64)?.dll$|^libcrypto-[\d_]+(-x64)?.dll$/,
+  "lib/ruby/#{package.rubylibver}/#{package.ruby_arch}/psych.so" => /^libyaml-[-\d]+.dll$/,
+  "lib/ruby/#{package.rubylibver}/#{package.ruby_arch}/zlib.so" => /^zlib\d.dll$/,
+}
+
+core_dll_defs = [
+  /^libgmp-\d+.dll$/,
+  /^libwinpthread-\d+.dll$/,
+  /^libgcc_s_.*.dll$/,
+]
+
+core_dlls, dlls = dlls.partition do |destpath|
+  core_dll_defs.any? { |re| re =~ File.basename(destpath) }
+end
+ext_dlls, dlls = dlls.partition do |destpath|
+  ext_dll_defs.values.any? { |re| re =~ File.basename(destpath) }
+end
+raise "DLL belonging neither to core nor to exts: #{dlls}" unless dlls.empty?
+
+
+###########################################################################
+# Add manifest to extension.so files pointing to linked MINGW library DLLs
+# next to it
+###########################################################################
+
+# Add tasks to move the DLLs into the extension directory
+ext_dlls.each do |destpath|
+  so_fname, _ = ext_dll_defs.find { |_, re| re =~ File.basename(destpath) }
+
+  new_destpath = File.join(sandboxdir, File.dirname(so_fname), File.basename(destpath))
+  file new_destpath => [destpath.sub(sandboxdir, unpackdirmgw), File.dirname(new_destpath)] do |t|
+    cp(t.prerequisites.first, t.name)
+  end
+
+  # Move the DLLs in the dependent files list to the subdirectory
+  self.sandboxfiles.delete(destpath)
+  self.sandboxfiles << new_destpath
+end
+
+# Add a custom manifest to each extension.so, so that they find the DLLs to be moved
+ext_dlls.each do |destpath|
+  so_fname, _ = ext_dll_defs.find { |_, re| re =~ File.basename(destpath) }
+  sandbox_so_fname = File.join(sandboxdir, so_fname)
+
+  file sandbox_so_fname => [sandbox_so_fname.sub(sandboxdir, unpackdirmgw), File.dirname(sandbox_so_fname)] do |t|
+    puts "patching manifest of #{t.name}"
+
+    # The XML elements we want to add to the default MINGW manifest:
+    new = <<~EOT
+    <dependency>
+      <dependentAssembly>
+        <assemblyIdentity version="1.0.0.0" type="win32" name="#{File.basename(so_fname)}-assembly" />
+      </dependentAssembly>
+    </dependency>
+    EOT
+
+    ManifestUpdater.update_file(t.prerequisites.first, new, t.name)
+  end
+end
+
+# Add a detached manifest file within the ext.so directory that lists all linked DLLs
+ext_dll_defs.each do |so_fname, re|
+  mani_path = File.join(sandboxdir, so_fname + "-assembly.manifest")
+  e_dlls = ext_dlls.select { |dll| re =~ File.basename(dll) }
+
+  file mani_path => [File.dirname(mani_path)] do |t|
+    puts "generate #{t.name}"
+
+    File.binwrite t.name, <<~EOT
+    <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+    <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
+      <assemblyIdentity type="win32" name="#{File.basename(so_fname)}-assembly" version="1.0.0.0"></assemblyIdentity>
+
+      #{ e_dlls.map{|dll| %Q{<file name="#{File.basename(dll)}"/>} }.join }
+    </assembly>
+    EOT
+  end
+  self.sandboxfiles << mani_path
+end
+
+
+#################################################################################
+# Add manifest to ruby.exe, rubyw.exe files pointing to DLLs in ruby_builtin_dlls
+#################################################################################
+
+core_dlls.each do |destpath|
+
   # Add tasks to write the DLLs into the sub directory
   new_destpath = File.join(File.dirname(destpath), "ruby_builtin_dlls", File.basename(destpath))
   file new_destpath => [destpath.sub(sandboxdir, unpackdirmgw), dlls_dir] do |t|
@@ -28,14 +116,30 @@ end
 self.sandboxfiles.select do |destpath|
   destpath =~ /\/rubyw?\.exe$/i
 end.each do |destpath|
+
   file destpath => [destpath.sub(sandboxdir, unpackdirmgw), bin_dir] do |t|
     puts "patching manifest of #{t.name}"
     libruby = File.basename(self.sandboxfiles.find{|a| a=~libruby_regex })
 
-    image = File.binread(t.prerequisites.first)
     # The XML elements we want to add to the default MINGW manifest:
-    new = <<-EOT
-      <application xmlns="urn:schemas-microsoft-com:asm.v3">
+    new = <<~EOT
+    <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
+      <security>
+        <requestedPrivileges>
+          <requestedExecutionLevel level="asInvoker"/>
+        </requestedPrivileges>
+      </security>
+    </trustInfo>
+    <compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1">
+      <application>
+        <supportedOS Id="{e2011457-1546-43c5-a5fe-008deee3d3f0}"/>
+        <supportedOS Id="{35138b9a-5d96-4fbd-8e2d-a2440225f93a}"/>
+        <supportedOS Id="{4a2f28e3-53b9-4441-ba9c-d69d4a4a6e38}"/>
+        <supportedOS Id="{1f676c76-80e1-4239-95bb-83d0f6d0da78}"/>
+        <supportedOS Id="{8e0f7a12-bfb3-4fe8-b9a5-48fd50a15a9a}"/>
+      </application>
+    </compatibility>
+    <application xmlns="urn:schemas-microsoft-com:asm.v3">
         <windowsSettings xmlns:ws2="http://schemas.microsoft.com/SMI/2016/WindowsSettings">
           <ws2:longPathAware>true</ws2:longPathAware>
         </windowsSettings>
@@ -48,38 +152,20 @@ end.each do |destpath|
       <file name="#{ libruby }"/>
     EOT
 
-    # There are two regular options to add a custom manifest:
-    # 1. Change a given exe file per Microsofts "mt.exe" after the build
-    # 2. Specify a the manifest while linking with the MINGW toolchain
-    #
-    # Since we don't want to depend on particular Microsoft tools and want to avoid additional patching of the ruby build, we do a nifty trick here.
-    # We patch the exe file manually.
-    # Removing unnecessary spaces and comments from the embedded XML manifest gives us enough space to add the above XML elements.
-    # Then the default MINGW manifest gets replaced by our custom XML content.
-    # The rest of the available bytes is simply padded with spaces, so that we don't change positions within the EXE image.
-    image.gsub!(/<\?xml.*?<assembly.*?<\/assembly>\n/m) do |m|
-      newm = m.gsub(/^\s*<\/assembly>\s*$/, new + "</assembly>")
-        .gsub(/<!--.*?-->/m, "")
-        .gsub(/^ +/, "")
-        .gsub(/\n+/m, "\n")
-
-      raise "replacement manifest to big #{m.bytesize} < #{newm.bytesize}" if m.bytesize < newm.bytesize
-      newm + " " * (m.bytesize - newm.bytesize)
-    end
-    File.binwrite(t.name, image)
+    ManifestUpdater.update_file(t.prerequisites.first, new, t.name)
   end
 end
 
 # Add a detached manifest file within the sub directory that lists all DLLs in question
 manifest2 = File.join(sandboxdir, "bin/ruby_builtin_dlls/ruby_builtin_dlls.manifest")
 file manifest2 => [dlls_dir] do |t|
   puts "generate #{t.name}"
-  File.binwrite t.name, <<-EOT
+  File.binwrite t.name, <<~EOT
     <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
     <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
       <assemblyIdentity type="win32" name="ruby_builtin_dlls" version="1.0.0.0"></assemblyIdentity>
 
-      #{ dlls.map{|dll| %Q{<file name="#{File.basename(dll)}"/>} }.join }
+      #{ core_dlls.map{|dll| %Q{<file name="#{File.basename(dll)}"/>} }.join }
     </assembly>
   EOT
 end

recipes/sandbox/80-copy-msys-files.rake

@@ -2,7 +2,11 @@ self.sandboxfiles.each do |destpath|
   directory File.dirname(destpath)
   unless Rake::Task.task_defined?(destpath)
     file destpath => [destpath.sub(sandboxdir, unpackdirmgw), File.dirname(destpath)] do |t|
-      cp_r(t.prerequisites.first, t.name)
+      if File.file?(t.prerequisites.first)
+        cp(t.prerequisites.first, t.name)
+      elsif File.directory?(t.prerequisites.first) && !File.exist?(t.name)
+        mkdir t.name
+      end
     end
   end
 end