async IO support via kqueue and epoll #216

bradgessler · 2024-12-06T17:14:14Z

I'm working with async-websockets to make a connection from a tebako app to a server. When the connection starts, the application halts on https://github.com/socketry/async/blob/367745ddf9494698327971b7f843f01a053a3662/lib/async/scheduler.rb#L386.

I'm pressing the binary with the latest version of ruby via -R 3.3.6 to make sure it's using the latest Ruby has to offer for Fibers, but it still halts on that LOC.

I don't know enough about epoll, kqueue, and compiling native apps to add much beyond that, but I'll keep updating this as I go deeper and find out more.

The text was updated successfully, but these errors were encountered:

bradgessler · 2024-12-06T17:33:05Z

Probably a related issue that solves this problem for all native gems: #72

maxirmx · 2024-12-07T07:45:01Z

Thank you, @bradgessler
You reference to #72 is most likely correct.
If your project includes gems that have native extensions calling other native libraries such libraries shall be treated indvidually.
It done by tebako-runtime gem

I will look at async-websockets when I close current PR

bradgessler · 2024-12-07T17:47:43Z

Interesting, I completely missed the purpose of that gem from the docs.

I'm certain the issue is actually in the io-select gem, which calls out to epoll on Linux and kqueue on macOS/BSD. Not sure what "call out" means since I'm not familiar with these libraries, but I do see C code in that repo that needs to refer to file descriptors.

maxirmx · 2024-12-13T19:46:30Z

Hi @bradgessler

I adopted an example from async-websockets:

Gemfile

# frozen_string_literal: true

source "https://rubygems.org"

gem "async"
gem "async-http"
gem "async-websocket"

binance.rb

#!/usr/bin/env ruby
# frozen_string_literal: true

# Released under the MIT License.
# Copyright, 2021-2024, by Samuel Williams.

require "async"
require "async/http"
require "async/websocket"

URL = "wss://stream.binance.com:9443/ws/btcusdt@bookTicker"

Async do |task|
	endpoint = Async::HTTP::Endpoint.parse(URL, alpn_protocols: Async::HTTP::Protocol::HTTP11.names)

	Async::WebSocket::Client.connect(endpoint) do |connection|
		while message = connection.read
			$stdout.puts message.parse
		end
	end
end

placed them in async folder
and processed with tebako press -r async -e binance.rb -o tebako-binance-package

resulting tebako package worked:

maxirmx@MSS-WS-N:~/Projects/tebako-samples$ ./tebako-binance-package
{:u=>56455063902, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"9.82012000", :a=>"101264.81000000", :A=>"42.99478000"}
{:u=>56455063903, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"9.82112000", :a=>"101264.81000000", :A=>"42.99478000"}
{:u=>56455063933, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"9.82012000", :a=>"101264.81000000", :A=>"42.99478000"}
{:u=>56455063934, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"9.82012000", :a=>"101264.81000000", :A=>"42.99379000"}
{:u=>56455063949, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.87937000", :a=>"101264.81000000", :A=>"42.99379000"}
{:u=>56455063970, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.87937000", :a=>"101264.81000000", :A=>"42.99478000"}
{:u=>56455063971, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.88036000", :a=>"101264.81000000", :A=>"42.99478000"}
{:u=>56455063972, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.88036000", :a=>"101264.81000000", :A=>"42.81865000"}
{:u=>56455063979, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.88036000", :a=>"101264.81000000", :A=>"42.66176000"}
{:u=>56455063989, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.88036000", :a=>"101264.81000000", :A=>"42.66077000"}
{:u=>56455063991, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.87937000", :a=>"101264.81000000", :A=>"42.66077000"}
{:u=>56455063992, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.86442000", :a=>"101264.81000000", :A=>"42.66077000"}
{:u=>56455063996, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.87906000", :a=>"101264.81000000", :A=>"42.66077000"}
{:u=>56455064011, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.87906000", :a=>"101264.81000000", :A=>"42.66028000"}
{:u=>56455064016, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.87906000", :a=>"101264.81000000", :A=>"42.65785000"}
{:u=>56455064022, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.88006000", :a=>"101264.81000000", :A=>"42.65785000"}
{:u=>56455064023, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.88006000", :a=>"101264.81000000", :A=>"42.65884000"}
{:u=>56455064043, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.88006000", :a=>"101264.81000000", :A=>"42.65785000"}
{:u=>56455064044, :s=>"BTCUSDT", :b=>"101264.80000000", :B=>"11.87906000", :a=>"101264.81000000", :A=>"42.65785000"}
^C/__tebako_memfs__/lib/ruby/gems/3.2.0/gems/async-2.21.1/lib/async/scheduler.rb:386:in `select': Interrupt
        from /__tebako_memfs__/lib/ruby/gems/3.2.0/gems/async-2.21.1/lib/async/scheduler.rb:386:in `run_once!'
        from /__tebako_memfs__/lib/ruby/gems/3.2.0/gems/async-2.21.1/lib/async/scheduler.rb:425:in `run_once'
        from /__tebako_memfs__/lib/ruby/gems/3.2.0/gems/async-2.21.1/lib/async/scheduler.rb:498:in `block in run'
        from /__tebako_memfs__/lib/ruby/gems/3.2.0/gems/async-2.21.1/lib/async/scheduler.rb:461:in `block in run_loop'
        from /__tebako_memfs__/lib/ruby/gems/3.2.0/gems/async-2.21.1/lib/async/scheduler.rb:458:in `handle_interrupt'
        from /__tebako_memfs__/lib/ruby/gems/3.2.0/gems/async-2.21.1/lib/async/scheduler.rb:458:in `run_loop'
        from /__tebako_memfs__/lib/ruby/gems/3.2.0/gems/async-2.21.1/lib/async/scheduler.rb:497:in `run'
        from /__tebako_memfs__/lib/ruby/gems/3.2.0/gems/async-2.21.1/lib/kernel/async.rb:34:in `Async'
        from /__tebako_memfs__/local/binance.rb:13:in `<main>'

maxirmx · 2024-12-13T19:49:02Z

So there may be an issue with tebako and this gem of course but it does not look easily reproducable
If you can share your code or at least some example I may be able to provide better support

bradgessler · 2024-12-18T17:55:45Z

I pushed that code up to https://github.com/terminalwire/tebako-async so I could run it and link to stuff; however I ran into an issues with the boost package on macOS because I think boost was updated from Brew.

Here's the error:

dyld[77545]: Library not loaded: /opt/homebrew/opt/boost/lib/libboost_chrono-mt.dylib
  Referenced from: <24FA4B4B-D617-39C9-8AE8-C440CC85DE26> /Users/bradgessler/.tebako/deps/bin/mkdwarfs
  Reason: tried: '/opt/homebrew/opt/boost/lib/libboost_chrono-mt.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/boost/lib/libboost_chrono-mt.dylib' (no such file), '/opt/homebrew/opt/boost/lib/libboost_chrono-mt.dylib' (no such file), '/opt/homebrew/Cellar/boost/1.87.0/lib/libboost_chrono-mt.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/Cellar/boost/1.87.0/lib/libboost_chrono-mt.dylib' (no such file), '/opt/homebrew/Cellar/boost/1.87.0/lib/libboost_chrono-mt.dylib' (no such file)
make[3]: *** [CMakeFiles/packaged_filesystem] Abort trap: 6
make[2]: *** [CMakeFiles/packaged_filesystem.dir/all] Error 2
make[1]: *** [CMakeFiles/tebako.dir/rule] Error 2
make: *** [tebako] Error 2

Here's the contents of the 1.87.0 boost package:

terminalwire/tebako-async [main] → ls /opt/homebrew/Cellar/boost/1.87.0/lib
cmake					libboost_locale.a			libboost_regex.a
libboost_atomic.a			libboost_locale.dylib			libboost_regex.dylib
libboost_atomic.dylib			libboost_log.a				libboost_serialization.a
libboost_charconv.a			libboost_log.dylib			libboost_serialization.dylib
libboost_charconv.dylib			libboost_log_setup.a			libboost_stacktrace_addr2line.a
libboost_chrono.a			libboost_log_setup.dylib		libboost_stacktrace_addr2line.dylib
libboost_chrono.dylib			libboost_math_c99.a			libboost_stacktrace_basic.a
libboost_container.a			libboost_math_c99.dylib			libboost_stacktrace_basic.dylib
libboost_container.dylib		libboost_math_c99f.a			libboost_stacktrace_noop.a
libboost_context.a			libboost_math_c99f.dylib		libboost_stacktrace_noop.dylib
libboost_context.dylib			libboost_math_c99l.a			libboost_system.a
libboost_contract.a			libboost_math_c99l.dylib		libboost_system.dylib
libboost_contract.dylib			libboost_math_tr1.a			libboost_test_exec_monitor.a
libboost_coroutine.a			libboost_math_tr1.dylib			libboost_thread.a
libboost_coroutine.dylib		libboost_math_tr1f.a			libboost_thread.dylib
libboost_date_time.a			libboost_math_tr1f.dylib		libboost_timer.a
libboost_date_time.dylib		libboost_math_tr1l.a			libboost_timer.dylib
libboost_exception.a			libboost_math_tr1l.dylib		libboost_type_erasure.a
libboost_fiber.a			libboost_nowide.a			libboost_type_erasure.dylib
libboost_fiber.dylib			libboost_nowide.dylib			libboost_unit_test_framework.a
libboost_filesystem.a			libboost_prg_exec_monitor.a		libboost_unit_test_framework.dylib
libboost_filesystem.dylib		libboost_prg_exec_monitor.dylib		libboost_url.a
libboost_graph.a			libboost_process.a			libboost_url.dylib
libboost_graph.dylib			libboost_process.dylib			libboost_wave.a
libboost_iostreams.a			libboost_program_options.a		libboost_wave.dylib
libboost_iostreams.dylib		libboost_program_options.dylib		libboost_wserialization.a
libboost_json.a				libboost_random.a			libboost_wserialization.dylib
libboost_json.dylib			libboost_random.dylib

The history of that package on Homebrew shows that it was updated 2 days ago (at the time of this writing): https://github.com/Homebrew/homebrew-core/commits/87ed609740968c1904035263d621c13d1451e1c4/Formula/b/boost.rb

Finally, when I try to install an older version I get this message from brew:

terminalwire/tebako-async [main] → brew install boost@1.76
Error: boost@1.76 has been disabled because it is a versioned formula! It was disabled on 2024-12-14.

bradgessler · 2024-12-18T18:04:51Z

One question I have that might help me better debug and troubleshoot issues: when an application is running inside of Tebako, is it completely containerized? In other words, does my Ruby code have full access to the outside world?

I read the docs for DwarFS, but as far as I could tell that's a binary blob of read-only files that a running program reads. What's unclear to me is what Tebako looks like when it's actually running.

The diagram at https://github.com/tamatebako/tebako-runtime is my mental model of what the program looks like at rest, but I'm not sure what it looks like when it's running. I think it looks like this:

Some sort of runtime is launched
Said runtime loads DwarFS
The runtime finds the entry-point within DwarFS
The runtime also finds the embedded Ruby version
The runtime passes the entry-point file into the embedded ruby binary and the application starts running.

I assume this runtime has some sort of boundary between the Ruby code that's running and a view of the outside world. I thought this was true for files when I looked at the PATH patches in tebako-runtime gem, but then I ran some basic tests like File.write("foo.txt", "hi") and saw that it wrote directly to the host operating systems' file system, which wrecked my mental model of Tebako's runtime.

Note that I might be applying what I know about Docker too heavily towards Tebako. I know they're very very different, but I don't know to what degree.

maxirmx · 2024-12-18T18:25:32Z

One question I have that might help me better debug and troubleshoot issues: when an application is running inside of Tebako, is it completely containerized? In other words, does my Ruby code have full access to the outside world?

No, the application is not containerized at all. Your Ruby code has full access to the outside world, no boundaries.

The application is patched Ruby interpreter. When it starts it mounts user-space device that contains your code and executes your code.

Practically it is achieved by implementation of ~10 functions. This implementation is aware of dwarfs filesystem and routes requested either to dwarfs or to the original versions.

Have a look at the code snippets, nothing to be really proud of :)
https://github.com/tamatebako/libdwarfs/blob/main/include/tebako-defines.h
https://github.com/tamatebako/libdwarfs/blob/main/src/file-io.cpp

Ruby is compiled and linked against these functions

maxirmx · 2024-12-18T18:30:38Z

I pushed that code up to https://github.com/terminalwire/tebako-async so I could run it and link to stuff; however I ran into an issues with the boost package on macOS because I think boost was updated from Brew.

Here's the error:

dyld[77545]: Library not loaded: /opt/homebrew/opt/boost/lib/libboost_chrono-mt.dylib
  Referenced from: <24FA4B4B-D617-39C9-8AE8-C440CC85DE26> /Users/bradgessler/.tebako/deps/bin/mkdwarfs
  Reason: tried: '/opt/homebrew/opt/boost/lib/libboost_chrono-mt.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/boost/lib/libboost_chrono-mt.dylib' (no such file), '/opt/homebrew/opt/boost/lib/libboost_chrono-mt.dylib' (no such file), '/opt/homebrew/Cellar/boost/1.87.0/lib/libboost_chrono-mt.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/Cellar/boost/1.87.0/lib/libboost_chrono-mt.dylib' (no such file), '/opt/homebrew/Cellar/boost/1.87.0/lib/libboost_chrono-mt.dylib' (no such file)
make[3]: *** [CMakeFiles/packaged_filesystem] Abort trap: 6
make[2]: *** [CMakeFiles/packaged_filesystem.dir/all] Error 2
make[1]: *** [CMakeFiles/tebako.dir/rule] Error 2
make: *** [tebako] Error 2

Yesterday I saw this issue as well and it does not make any sense. boost library on MacOS is libboost_chrono.dylib without -mt suffix.
Today the issue just dissolved. On GHE I am doing brew update, not sure if it is critical or not.
I will be releasing the next version of the gem in 1-2 days. It is tested against updated homebrew.

bradgessler · 2024-12-18T19:17:41Z

I'll run press again after you release the gem and see how far I get.

ronaldtse · 2024-12-19T01:44:46Z

but I'm not sure what it looks like when it's running.

Thanks @bradgessler , we are definitely missing some explanatory documentation on how Tebako works.

As @maxirmx explained, the goal of Tebako is to allow interpreted executables run with an interpreter runtime "just like" if it was available on the system, so there are no boundaries enforced.

It is certainly possible to implement a "chroot"-style jail with explicit host mounts (locking the interpreter inside the DwarfFS volume), but so far we haven't had that use case (and whether that use case is something we should invest in is another question).

maxirmx · 2024-12-20T22:25:20Z

Ref terminalwire/tebako-async#1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

async IO support via kqueue and epoll #216

async IO support via kqueue and epoll #216

bradgessler commented Dec 6, 2024 •

edited

Loading

bradgessler commented Dec 6, 2024

maxirmx commented Dec 7, 2024

bradgessler commented Dec 7, 2024

maxirmx commented Dec 13, 2024

maxirmx commented Dec 13, 2024

bradgessler commented Dec 18, 2024 •

edited

Loading

bradgessler commented Dec 18, 2024

maxirmx commented Dec 18, 2024

maxirmx commented Dec 18, 2024

bradgessler commented Dec 18, 2024

ronaldtse commented Dec 19, 2024

maxirmx commented Dec 20, 2024

async IO support via kqueue and epoll #216

async IO support via kqueue and epoll #216

Comments

bradgessler commented Dec 6, 2024 • edited Loading

bradgessler commented Dec 6, 2024

maxirmx commented Dec 7, 2024

bradgessler commented Dec 7, 2024

maxirmx commented Dec 13, 2024

maxirmx commented Dec 13, 2024

bradgessler commented Dec 18, 2024 • edited Loading

bradgessler commented Dec 18, 2024

maxirmx commented Dec 18, 2024

maxirmx commented Dec 18, 2024

bradgessler commented Dec 18, 2024

ronaldtse commented Dec 19, 2024

maxirmx commented Dec 20, 2024

bradgessler commented Dec 6, 2024 •

edited

Loading

bradgessler commented Dec 18, 2024 •

edited

Loading