Rubygems and bundler

[c4lliope]

Hello again - I had some success running Ruby code in an online page,
as you can see on ://ruby.assembled.app.

This is good progress, and I'm now curious if there is an approach
for requiring ruby gem libraries inside the .wasm package,
so my code can use helpers such as 1.second and "hello".titlecase,
among others.

I recognize some libraries including C extensions, such as Nokogiri,
are unlikely to be runnable inside .wasm unless specially compiled.
For non-C gems, I'm curious if there is a non-compilation approach,
though if compiling a new .wasm binary is necessary, I'd be glad for directions.

[dphilla]

I've been working on a couple of different approaches to this, if you'd like to talk about it more, feel free to dm me (on twitter, in bio).

Basically, I've made it (kind of) work by building from source after adding gems to the default gems in 'ruby-wasm32-wasi/usr' as mounted to '/usr' which is kind of clunky 🙈 , and that's why I am also trying to find a more elegant solution.

I would guess the maintainers here have some better thoughts on how to approach this 😄 .

[kateinoigakukun]

It still needs some hacks, but here is how to use bundler and rubygems on Wasm.
https://gist.github.com/kateinoigakukun/5caf3b83b2732b1653e91b0e75ce3390

The main important points are:

• Tell the location of Gemfile by BUNDLE_GEMFILE
• Call require "bundler/setup" instead of bundle exec

[c4lliope]

@kateinoigakukun - can you also lend some direccions on running C-compiled libraries?
Nokogiri especially could be really usable in online pages.

[c4lliope]

And looks like your branch #15 merged;
does your approach change much?

[kateinoigakukun]

I'm not well familiar with nokogiri, but it seems using mkmf. So what needs to be done to support wasm target are:

1. Support cross-compilation with mkmf and RubyGems builder
2. Figure out a way to link the compiled extension library with pre-compiled ruby.wasm (wasm doesn't support dynamic linking, so it would be difficult). Or provide an easy way to link 3rd party ext libraries while building. It would be done by generalizing how ruby.wasm links js interop libraries.
   c.f. ext directory in this repo, how they are built, and how they are linked

[kateinoigakukun]

And looks like your branch #15 merged;
does your approach change much?

I updated the instruction, but not much changed 👍

[c4lliope]

I began running your recipe using a couple changes:

I cloned today's release of 3.2.0-dev:

curl -LO https://github.com/ruby/ruby.wasm/releases/download/2022-07-05-a/ruby-head-wasm32-unknown-wasi-full.tar.gz
tar xfz ruby-head-wasm32-unknown-wasi-full.tar.gz
rbenv local 3.2.0-dev

I prepared an example running a simpler rubygem,
ox xml parser.

bundle init
bundle add ox
bundle config set --local path 'vendor/bundle'
bundle install

I added a simple program, ox.rb:

require 'bundler/setup'
require 'net/http'
require 'ox'

Ox.default_options = { mode: :generic, effort: tolerant, smart: true, }

page = Ox.parse Net::HTTP.get URI.parse 'http://example.com'
puts page.locate('*/a')[0].attributes['href']

And some helper code in ./run.sh:

#!/bin/bash

run() {
  wasmtime run \
    --mapdir /usr::./head-wasm32-unknown-wasi-full/usr \
    --mapdir /root::./ \
    --mapdir /dev::/dev \
    --env BUNDLE_GEMFILE=/root/Gemfile \
    head-wasm32-unknown-wasi-full/usr/local/bin/ruby -- \
    "$@"
}

run --version
run '-e' 'puts Dir.glob("/root/vendor/bundle/ruby/3.2.0+1/gems/*")'
run /root/ox.rb

I seemed prepared enough, only by some quirk my rubygems had gone missing.

[grace@chesapeake ruby]$ ./run.sh 
ruby 3.2.0dev (2022-07-06) [wasm32-wasi]
/root/vendor/bundle/ruby/3.2.0+1/gems/ox-2.14.11
Could not find ox-2.14.11 in any of the sources
Run `bundle install` to install missing gems.

Perhaps in a side discussion, can you explain the socket error I see
as I load net/http inside wasmtime?

[grace@chesapeake ruby]$ run -e "require 'net/http'"
Ignoring debug-1.5.0 because its extensions are not built. Try: gem pristine debug --version 1.5.0
Ignoring rbs-2.6.0 because its extensions are not built. Try: gem pristine rbs --version 2.6.0
<internal:/usr/local/lib/ruby/3.2.0+1/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- socket (LoadError)
        from <internal:/usr/local/lib/ruby/3.2.0+1/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /usr/local/lib/ruby/3.2.0+1/net/protocol.rb:22:in `<top (required)>'
        from <internal:/usr/local/lib/ruby/3.2.0+1/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/local/lib/ruby/3.2.0+1/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /usr/local/lib/ruby/3.2.0+1/net/http.rb:23:in `<top (required)>'
        from <internal:/usr/local/lib/ruby/3.2.0+1/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/usr/local/lib/ruby/3.2.0+1/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from -e:1:in `<main>'

[kateinoigakukun]

@c4lliope It looks like ox gem has a native extension. You need to manage to build extensions as wasm object files and link them with CRuby statically.

[c4lliope]

I had been guessing my problem is misaligned ruby version numbers;
e.g. the rubygems path /root/vendor/bundle/ruby/3.2.0+1/gems/*
is unrecognized by the 2022-07-05a release of ruby.wasm?

[kateinoigakukun]

@c4lliope You can see which paths are in the search path by:

wasmtime run \
  --mapdir /usr::./head-wasm32-unknown-wasi-full/usr \
  --mapdir /root::./ --mapdir /dev::/dev \
  --env BUNDLE_GEMFILE=/root/Gemfile \
  head-wasm32-unknown-wasi-full/usr/local/bin/ruby -- -e 'require "bundler/setup"; puts $:'

And you will see /root/vendor/bundle/ruby/3.2.0+1/gems/* is recognized well.

Also you can see more verbose log by replacing require "bundler/setup" by require "bundler"; Bundler.setup and add --env DEBUG=1, then you will see:

$ wasmtime run \
  --mapdir /usr::./head-wasm32-unknown-wasi-full/usr \
  --mapdir /root::./ --mapdir /dev::/dev \
  --env BUNDLE_GEMFILE=/root/Gemfile --env DEBUG=1 \
  head-wasm32-unknown-wasi-full/usr/local/bin/ruby -- /root/my_app.rb

Found no changes, using resolution from the lockfile
Source locally installed gems is ignoring #<Bundler::StubSpecification name=ox version=2.14.11 platform=ruby> because it is missing extensions
/usr/local/lib/ruby/3.2.0+1/bundler/definition.rb:486:in `materialize': Could not find ox-2.14.11 in any of the sources (Bundler::GemNotFound)
        from /usr/local/lib/ruby/3.2.0+1/bundler/definition.rb:191:in `specs'
        from /usr/local/lib/ruby/3.2.0+1/bundler/definition.rb:239:in `specs_for'
        from /usr/local/lib/ruby/3.2.0+1/bundler/runtime.rb:18:in `setup'
        from /usr/local/lib/ruby/3.2.0+1/bundler.rb:151:in `setup'
        from /root/my_app.rb:2:in `<main>'

Please note that Source locally installed gems is ignoring #<Bundler::StubSpecification name=ox version=2.14.11 platform=ruby> because it is missing extensions

Again, ox has C extension library and it's not compiled in your build script. That's why it fails.

[c4lliope]

Ah, incredible description; these commands are useful for digging deeper in my build.
May be a number of days before I can come back and learn this compilation approach more.

[c4lliope]

Hello! I'm seeing some progress, and some remaining errors.

I had an example going with the simple and dependency-less colorize gem.
The approach described in @kateinoigakukun 's gist had been good background.

I began seeing an odd error with the gem he chose as an example, though - syntax_tree.

$ wasmtime run \
  --mapdir /usr::3_2-wasm32-unknown-wasi-full/usr \
  --mapdir /src::./src \
  --env BUNDLE_GEMFILE=/src/Gemfile \
  ruby.wasm -- /src/tree.rb
Could not find fiddle-1.1.1 in locally installed gems
Run `bundle install` to install missing gems.

Looking closer, the fiddle gem appears to be a "default" gem,
and is therefore packaged inside the 3.2.0 ruby installation.

$ bundle install
Using bundler 2.4.1
Using colorize 0.8.1
Using fiddle 1.1.1
Using prettier_print 1.2.0
Using syntax_tree 5.3.0
Bundle complete! 3 Gemfile dependencies, 5 gems now installed.
Bundled gems are installed into `./bundle`
$ bundle show syntax_tree
/home/grace/src/ruby-wasm/src/bundle/ruby/3.2.0/gems/syntax_tree-5.3.0
$ bundle show fiddle
/home/grace/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/fiddle-1.1.1

Since my Ruby versions are the same, I'd imagine fiddle should be a "default" gem in ruby.wasm also:

$ ruby -v
ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-linux]
$ wasmtime ruby.wasm -- -v
ruby 3.2.0 (2022-12-25 revision a528908271) [wasm32-wasi]

Sadly, no fiddle in ruby.wasm:

$ ls 3_2-wasm32-unknown-wasi-full/usr/local/lib/ruby/gems/3.2.0/gems/
abbrev-0.1.1           etc-1.4.2          net-pop-0.1.2       psych-5.0.1           singleton-0.1.1
base64-0.1.1           fcntl-1.0.2        net-protocol-0.2.1  racc-1.6.2            stringio-3.0.4
benchmark-0.2.1        fileutils-1.7.0    net-smtp-0.3.3      rake-13.0.6           strscan-3.0.5
bigdecimal-3.1.3       find-0.1.1         nkf-0.1.2           rbs-2.8.2             syntax_suggest-1.0.2
bundler-2.4.1          forwardable-1.3.3  observer-0.1.1      rdoc-6.5.0            tempfile-0.1.3
cgi-0.3.6              getoptlong-0.2.0   open3-0.1.2         readline-0.0.3        test-unit-3.5.7
csv-3.2.6              ipaddr-1.2.5       openssl-3.1.0       reline-0.3.2          time-0.2.1
date-3.3.3             irb-1.6.2          open-uri-0.3.0      resolv-0.2.2          timeout-0.3.1
debug-1.7.1            json-2.6.3         optparse-0.3.1      resolv-replace-0.1.1  tmpdir-0.1.3
delegate-0.3.0         logger-1.5.3       ostruct-0.5.5       rexml-3.2.5           tsort-0.1.1
did_you_mean-1.6.3     matrix-0.4.2       pathname-0.2.1      rinda-0.1.1           typeprof-0.21.3
digest-3.1.1           minitest-5.16.3    power_assert-2.0.3  rss-0.2.9             un-0.2.1
drb-2.1.1              mutex_m-0.1.2      pp-0.4.0            ruby2_keywords-0.0.5  uri-0.12.0
english-0.7.2          net-ftp-0.2.0      prettyprint-0.1.1   securerandom-0.2.2    weakref-0.1.2
erb-4.0.2              net-http-0.3.2     prime-0.1.2         set-1.0.3             yaml-0.2.1
error_highlight-0.5.1  net-imap-0.3.4     pstore-0.1.2        shellwords-0.1.0      zlib-3.0.0

Here are the gems in my normal Ruby 3.2.0:

$ ls ~/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/
abbrev-0.1.1           getoptlong-0.2.0              ostruct-0.5.5         ruby2_keywords-0.0.5
addressable-2.8.1      io-console-0.6.0              pathname-0.2.1        rubygems-update-3.4.5
base64-0.1.1           io-nonblock-0.2.0             power_assert-2.0.3    securerandom-0.2.2
benchmark-0.2.1        io-wait-0.3.0                 pp-0.4.0              set-1.0.3
bigdecimal-3.1.3       ipaddr-1.2.5                  prettyprint-0.1.1     shellwords-0.1.0
bundler-2.4.1          irb-1.6.2                     prime-0.1.2           singleton-0.1.1
bundler-2.4.5          json-2.6.3                    pry-0.14.2            stringio-3.0.4
capybara-3.38.0        logger-1.5.3                  pstore-0.1.2          strscan-3.0.5
cgi-0.3.6              matrix-0.4.2                  psych-5.0.1           syntax_suggest-1.0.2
coderay-1.1.3          method_source-1.0.0           public_suffix-5.0.1   syslog-0.1.1
csv-3.2.6              mini_mime-1.1.2               racc-1.6.2            tempfile-0.1.3
date-3.3.3             minitest-5.16.3               rack-3.0.4.1          test-unit-3.5.7
debug-1.7.1            mutex_m-0.1.2                 rack-test-2.0.2       time-0.2.1
delegate-0.3.0         net-ftp-0.2.0                 rake-13.0.6           timeout-0.3.1
did_you_mean-1.6.3     net-http-0.3.2                rbs-2.8.2             tmpdir-0.1.3
digest-3.1.1           net-imap-0.3.4                rdoc-6.5.0            torch-rb-0.12.0
drb-2.1.1              net-pop-0.1.2                 readline-0.0.3        torch-rb-0.12.1
english-0.7.2          net-protocol-0.2.1            readline-ext-0.1.5    tsort-0.1.1
erb-4.0.2              net-smtp-0.3.3                regexp_parser-2.6.2   typeprof-0.21.3
error_highlight-0.5.1  nkf-0.1.2                     reline-0.3.2          un-0.2.1
etc-1.4.2              nokogiri-1.14.0-x86_64-linux  resolv-0.2.2          uri-0.12.0
fcntl-1.0.2            observer-0.1.1                resolv-replace-0.1.1  weakref-0.1.2
fiddle-1.1.1           open3-0.1.2                   rexml-3.2.5           xpath-3.2.0
fileutils-1.7.0        openssl-3.1.0                 rice-4.0.4            yaml-0.2.1
find-0.1.1             open-uri-0.3.0                rinda-0.1.1           zlib-3.0.0
forwardable-1.3.3      optparse-0.3.1                rss-0.2.9

There's much ruby.wasm can do alone, though real adoption can only happen
once people can add and use gems as easily as on their base machine.
If we can agree on these 'default' gems included in each distribution,
this could be a good launchpad for more gem access on wasm.

[kateinoigakukun]

In short, fiddle is unavailable on the WASI platform due to a lack of runtime dynamic linking. I know it's one of the most crucial library to get real-world adoption, so we should have a way to mitigate this situation. But I think it will be happened after supporting extension libraries in RubyGems

[jtippett]

I tried the example in your gist without success, in fact I'm not sure I even understand it (shouldn't it need to build with the vfs first?). Anyway, I attached my experience to that. Has anyone actually managed to do this at all, or should I just inline code (pure ruby only, obviously) to add additional libraries?

FYI the lib I was trying with was a pure-ruby msgpack gem.