ARM64/AARCH64 Support

[AndyObtiva]

I got a new Mac M1 CPU (ARM64/AARCH64) computer and the first thing I did was try LibUI, and it didn't work (clearly noting have 'x86_64', need 'arm64e'):

/Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/fiddle.rb:61:in `initialize': dlopen(/Users/andymaleh/.rvm/gems/ruby-3.1.0@glimmer-dsl-libui/gems/libui-0.0.13/vendor/libui.dylib, 0x0009): tried: '/Users/andymaleh/.rvm/gems/ruby-3.1.0@glimmer-dsl-libui/gems/libui-0.0.13/vendor/libui.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/libui.dylib' (no such file), '/usr/lib/libui.dylib' (no such file) (Fiddle::DLError)
	from /Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/fiddle.rb:61:in `new'
	from /Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/fiddle.rb:61:in `dlopen'
	from /Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/fiddle/import.rb:86:in `block in dlload'
	from /Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/fiddle/import.rb:77:in `collect'
	from /Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/fiddle/import.rb:77:in `dlload'
	from /Users/andymaleh/.rvm/gems/ruby-3.1.0@glimmer-dsl-libui/gems/libui-0.0.13/lib/libui/ffi.rb:10:in `<module:FFI>'
	from /Users/andymaleh/.rvm/gems/ruby-3.1.0@glimmer-dsl-libui/gems/libui-0.0.13/lib/libui/ffi.rb:5:in `<module:LibUI>'
	from /Users/andymaleh/.rvm/gems/ruby-3.1.0@glimmer-dsl-libui/gems/libui-0.0.13/lib/libui/ffi.rb:4:in `<top (required)>'
	from /Users/andymaleh/.rvm/gems/ruby-3.1.0@glimmer-dsl-libui/gems/libui-0.0.13/lib/libui.rb:19:in `require_relative'
	from /Users/andymaleh/.rvm/gems/ruby-3.1.0@glimmer-dsl-libui/gems/libui-0.0.13/lib/libui.rb:19:in `<module:LibUI>'
	from /Users/andymaleh/.rvm/gems/ruby-3.1.0@glimmer-dsl-libui/gems/libui-0.0.13/lib/libui.rb:4:in `<top (required)>'
	from <internal:/Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:/Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /Users/andymaleh/.rvm/gems/ruby-3.1.0@glimmer-dsl-libui/gems/glimmer-dsl-libui-0.4.20/lib/glimmer-dsl-libui.rb:35:in `<top (required)>'
	from <internal:/Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:160:in `require'
	from <internal:/Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
	from <internal:/Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
<internal:/Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- glimmer-dsl-libui (LoadError)
	from <internal:/Users/andymaleh/.rvm/rubies/ruby-3.1.0/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'

It would be cool if you could figure out a way to compile LibUI for ARM64 on Mac and AARCH64 on Linux.

[kojix2]

I didn't know that LibUI doesn't work on new Macs
I rented an Apple silicon server on Macincloud and confirmed that I do indeed get the above error.

git clone https://github.com/kojix2/LibUI
cd LibUI
bundle
bundle exec rake vendor:mac_x64
bundle exec rake test # ERROR

However, after compiling libui-ng from source code and creating a shared library, the error no longer occurred.

git clone https://github.com/libui-ng/libui-ng
cd libui-ng
meson setup build
ninja -C build

Copy the created shared library libui.A.dylib to vendor directory.
Don't forget to rename the file to remove A.

cp libui-ng/build/meson-out/libui.A.dylib LibUI/vendor/libui.dylib

Then run

bundle exec rake test

The test will now pass.

glimmer-dsl-libui works in a similar way. (Replaced vendor/bundle/ruby/3.0.0/gems/libui-0.0.13/vendor/libui.dylib.)

[kojix2]

Of course, we don't want to go to all this trouble.
LibUI should work on various platforms just by installing the gem.

The easiest way to do this is for libui-ng to release binary files for the various platforms, and LibUI to include them in the gem.

This is an issue for the future.

[rubyFeedback]

Having libui work EVERYWHERE would be pretty cool - imagine people using arduino + libui + mruby. :D

Hopefully libui-ng can improve stuff there.

kojix2 wrote:

LibUI should work on various platforms just by installing the gem.

I think so too. That's by far LibUI's most compelling advantage - you just "gem install" it. Imagine if we could
do this for ruby-gtk on windows too ... (I kind of gave up trying to compile the gtk stack on windows ... on
linux it works very well though).

[kojix2]

Another practical way to include shared files for various platforms in your Gem is to automatically build LibUI when you install the Gem. You may think of a C-extended Gem, but it is also possible to do this with FFI.

For example, ruby-minimap2 does this; it uses FFI, but compiles minimap2 at installation time.

The method is as follows
First, place minimap2 (or libui) as a Git submodule in the ext directory, and write a build task as the default task in the Rakefile. Next, write in gemspec.

spec.extensions = %w[ext/Rakefile]`

This will fire the build task at installation time.

It looks good.

However, I think this method may not work as expected with LibUI, because LibUI compilation depends on meson and ninja. In contrast, all you need to build minimap2 is make, and unlike make and cc, meson and ninja are not supposed to be installed in all environments. There are a few people in my office who use Macs, but they are not programmers, so they do not install these tools. It is quite possible that they have Microsoft Office installed, but not Homebrew or MacPorts.

So I guess it might be practical to prepare binary files for major platforms and build libui from source code for minor environments. I don't want to put a lot of binary files and libui code in LibUI. From the user's point of view, binary files that are not used in their environment are just a waste of disk space. I want libui to be a small gem.

Another option would be to release a new gem to build LibUI at installation time (e.g. libui-build), or simply add a task to the Rakefile to help build libui. It is also possible to add a task to build libui using only make or cc, without relying on meson or ninja. However, this may be difficult to maintain for me.

[cody271]

The easiest way to do this is for libui-ng to release binary files for the various platforms, and LibUI to include them in the gem.

ARM builds is definitely a priority, at least for macOS and Linux. See issue here

[AndyObtiva]

@kojix2 it would be cool if you could make one more release with the andlabs libui that includes binaries for ARM64 on Mac and AARCH64 on Linux before you consider moving to libui-ng going forward. Sorry, I didn't mention this before your just released 0.0.14 (I did not expect you to make that release). Maybe you could push libui-ng changes to a temporary branch while you make one more release for ARM64/AARCH64 with andlabs libui and then bring the libui-ng changes back when it is a good time.

[kojix2]

@AndyObtiva
I admit that the fact that LibUI does not support M1 is not a small problem. However, I don't know the best practices for making fat gems compatible with both Intel64-based and ARM64-based Macintosh computers.

• In v0.0.14, three binary files (one for Windows, one for Mac, and one for Linux) are packed in the vendor directory, the same way ankane is doing it.

• Nokogiri provides a different gem for each platform.

nokogiri (1.13.1 ruby aarch64-linux arm64-darwin java x64-mingw-ucrt x64-mingw32 x86-linux x86-mingw32 x86_64-darwin x86_64-linux, 1.6.1 x86-mswin32-60, 1.4.4.1 x86-mswin32)

• My personal mentor, kou, don't like fat gems. He used to create ruby-gnome as a fat gem, but now he thinks it is a best practice to avoid using fat gems.

Probably v0.0.14 will be alive for a long time, so a workaround is needed. The easiest way I can think of is to compile libui in my macincloud and include it in the vendor directory. When I have time, I will see if this will work. And hopefully I will be able to release a new gem.

However, even if I can find a way to manually create binaries in Macincloud, it's just a workaround. cody said "GitHub Actions are x86 only...". I think this is the real bottleneck.

So we need to support libui-ng and find a way to deploy with less pain.

If there are any skilled...or not-so-skilled, Mac users watching this thread, please consider whether you can contribute to the continuous deployment of libui-ng...

[AndyObtiva]

I can give you a few tips since I had to support arm64 recently in Glimmer DSL for SWT, which is a sort of a JRuby fat gem (includes Java JAR files within).

I use the OS gem that senses what platform the user is on and outputs x86_64 or arm64, and based on that, I activate the SWT Mac libraries for x86_64 or arm64, both of which are included in the glimmer-dsl-swt gem.

In fact, you don't need the OS gem (that's just an extra convenience). If you run this command, you can do the same thing:

uname -m

It returns arm64 or x86_64 depending on the system. You would sense that in addition to the other variable that is the platform of Mac, Windows, or Linux (which is easy to detect in Ruby as far as I know):

RbConfig::CONFIG['host_os']

You can also sense the architecture using RbConfig now that I am thinking about it (probably simpler than shelling out with the uname command):

RbConfig::CONFIG['host_cpu'] # returns "arm64" on Mac ARM64 and "x86_64" on Mac x86_64

Adding Mac ARM64 in a release to start (without Linux AARCH64) is a great idea! That way, you break the problem down bit by bit instead of tackling everything at once.

However, even if I can find a way to manually create binaries in Macincloud, it's just a workaround.

A workaround is better than nothing for the short term. I have a Mac ARM64 machine I can verify your workaround on.

We'll cross the libui-ng bridge when we get to it. But, I think given that you can compile the andlabs libui on a Macincloud machine, if that works successfully, it would be a big win for the time being until libui-ng makes a release (still has no release yet, so it is too early to support).

[kojix2]

I created libui.dylib in universal binary format and uploaded it to the v0.0.14 release page.

This libui.dylib is a simple merge of the official dylib distributed by andlabs with the dylib compiled on MacinCloud M1 Mac server.

compilation:

git clone https://github.com/andlabs/libui
git checkout alpha4.1
mkdir -p release
cd release
cmake -DCMAKE_BUILD_TYPE=Release ..
make

merge:

lipo -create libui.x86_64.dylib libui.arm64.dylib -output libui.dylib

More precisely I have created a new universal shared library by adding new binary files to the original binary files distributed by andlabs. I believe this is a permitted activity under the MIT license. However, those who take the position that the word "software" in the MIT license means purely source code may find problems with editing and redistribution of binary files.

Finally, I added a task to the Rakefile to download the shared library for this M1 Mac.

(While working on a Mac, I noticed that when I call UI.init, I get an error message. However, I don't think this has anything to do with the new libui.dylib, because it happens with the current version of the gem. )

@AndyObtiva

A workaround is better than nothing for the short term. I have a Mac ARM64 machine I can verify your workaround on.

Thanks.
You can try it with the command below.

git clone https://github.com/kojix2/LibUI
git fetch
git checkout andlabs
rake -T
rake vendor:mac_arm
bundle
bundle exec ruby examples/control_gallery.rb

This is my first attempt to create and distribute software in binary format. I'm not sure if it will really work, so I'm not going to release it for a while and see how it goes. Please give it a try if you like.

[AndyObtiva]

The instructions worked (on Mac ARM64)!

Just a few minor modifications to the instructions:

git clone https://github.com/kojix2/LibUI
cd LibUI
git fetch
git checkout andlabs
bundle
rake -T
rake vendor:mac_arm
bundle exec ruby examples/control_gallery.rb

By the way, I also tested the instructions on my Mac x86_64, and they worked too (using the vendor:mac_arm)!

[kojix2]

I have released v0.0.15.pre so that you don't have to use the rake task to fetch the shared library.
https://github.com/kojix2/LibUI/releases/tag/v0.0.15.pre

gem install libui --pre

If no one reports a bug, I'll upload it to rubygem as v0.0.15 next weekend or the weekend after.

[AndyObtiva]

I just tested 0.0.15.pre, and it worked on both Mac arm64 and Mac x86_64. It also worked on Windows 10 and Linux.

[kojix2]

@AndyObtiva
Released v0.0.15. Thanks for the report and your strong support!!

[AndyObtiva]

That's excellent news!!!

I just upgraded the required libui version in glimmer-dsl-libui to 0.0.15 and released v0.5.5:
https://github.com/AndyObtiva/glimmer-dsl-libui/tree/v0.5.5

Cheers!

[AndyObtiva]

I think I closed this issue too soon forgetting arm64 on Linux, aka aarch64. I am re-opening due to this new issue in glimmer-dsl-libui:

AndyObtiva/glimmer-dsl-libui#23

[cody271]

I think I closed this issue too soon forgetting arm64 on Linux, aka aarch64. I am re-opening due to this new issue in glimmer-dsl-libui:

AndyObtiva/glimmer-dsl-libui#23

We now have an official Linux package that supports both ARM and x86, at least on Arch:

$ yay -S libui-ng-git

[cody271]

CI now builds for ARM and x86 on macOS, so you could use the prebuilt binaries from here.

[kojix2]

Thank you @cody271.
Now I can use the latest pre-built binaries.

It would be better if the github actions Artifact data were persistent and there were URLs that will always download the latest build.

[cody271]

Thank you @cody271. Now I can use the latest pre-built binaries.

It would be better if the github actions Artifact data were persistent and there were URLs that will always download the latest build.

This isn't supported by the API for GitHub Actions CI unfortunately..

[cody271]

There are a few workarounds out there though, I am investigating which is best.

[cody271]

@kojix2 Try this: master/macOS-x64-shared-debug.zip

[kojix2]

@cody271

Thanks.
I just added a rake task that allow you to download development builds for Mac and Ubuntu from the links you provided.
Now it is very easy to try out the development version of libui-ng.
It would be even better if a shared library for Windows were available :)

[cody271]

It would be even better if a shared library for Windows were available :)

Absolutely. Improvements to the Windows CI builds are in progress, see PR here.

[kojix2]

As of November 12, 2023, Ruby's LibUI includes new shared libraries built using GitHub Actions. These libraries are created in the .github/workflows/pre-build.yml located in the pre-build branch of the kojix2/libui-ng repository. The workflow is a modification of the original build.yml, set up to produce shared libraries in release mode.

For Raspberry Pi, the project uses guyot/arm-runner-action@v2, which builds libraries using Qemu. However, these libraries have not been tested.

The main developer, kojix2, primarily works with Ubuntu (x64) and MacOS (M2). Issues related to these systems are likely to be found quickly. However, for other systems, the project relies on reports from volunteers.

[kojix2]

The build for Raspberry Pi did not work on my old Raspberry Pi, the link to glibc is not working.I'm starting to think that for arm-linux, it's better to encourage people to build it themselves rather than distribute binaries.It's not that hard, and Raspberry Pi users can do it. There are many variations of arm-linux, and there are many cases where it won't work, I think.

[cody271]

QEMU for CI is just not reliable. I am still hoping that public support for GitHub Actions on ARM is added soon!
Otherwise we will have to switch to a much more portable CI provider..

See https://github.com/orgs/community/discussions/73040