libsmpack Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

[CAMOBAP]

https://github.com/metanorma/packed-mn/actions/runs/7836671011/job/21384812122#step:7:443

No changes in:

• ruby version
• bundler version
• runner version

But bundler decide to rebuild native extension instead using prebuilt one(

[ronaldtse]

@CAMOBAP do you need @maxirmx 's help here? Thanks.

[CAMOBAP]

I think yes, generally is there an answer to why bundler decides that native extension must be rebuilt and prebuild cannot be used?

As I noted in the topic there are no changes between success/failure run. Also when I re-executed previous successful run it also fails, which looks confusing

[maxirmx]

libmspack gem does not have native extension published so it is probably built each time.

It looks like ffi-compiler2 gem invocation got broken and it now inserts '' to compiler command line

gcc -fexceptions -O -fno-omit-frame-pointer -fno-strict-aliasing -fPIC
-I./libmspack/mspack/ -D_FILE_OFFSET_BITS\=64 -DHAVE_MEMCMP\=1
-DHAVE_TOWLOWER\=1 -DHAVE_TOLOWER\=1 -DHAVE_FSEEKO\=1 -DHAVE_CTYPE_H\=1
-DHAVE_LIMITS_H\=1 -DHAVE_STDLIB_H\=1 -DHAVE_STRING_H\=1 -DHAVE_WCTYPE_H\=1 -o
x86_64-windows/cabc.o -c ./libmspack/mspack//cabc.c

There are no direct changes to configuration since the last successful run but this workflow indirectly installs the latest bundler. Now it is 2.5.6, the last successful run was with 2.5.5.
Also MSys may have some components updated.

I would start with trying bundler 2.5.5 as the last good run was with it

[maxirmx]

@CAMOBAP
I can take this issuse if you like

[CAMOBAP]

@maxirmx Yes please, really appreciate the help

A few notes from my side:

1. there was a successful run with 2.5.6 https://github.com/metanorma/packed-mn/actions/runs/7805778709/job/21290718167#step:7:330
2. yep there is no prebuilt ext https://rubygems.org/gems/libmspack/versions, so previous compilation worked
3. Actually runner image changes:
   • Successful one: https://github.com/actions/runner-images/blob/win19/20240128.1/images/windows/Windows2019-Readme.md
   • Failed one: https://github.com/actions/runner-images/blob/win19/20240204.1/images/windows/Windows2019-Readme.md

[ronaldtse]

Is libmspack a dependency from fontist? Is that a better place to fix this, and potentially we can reuse the issue here? Eg have prepackaged dependencies that we can reuse?

[maxirmx]

There is a fork already -- https://github.com/metanorma/ruby-libmspack
Pls give me write permissions
We shall transfer this issue to this repo and, probably, transfer the repo to fontist organization.

[ronaldtse]

@maxirmx I’ve fixed the access rights on that repo. Please feel free to proceed there. Thanks!

[maxirmx]

@maxirmx I’ve fixed the access rights on that repo. Please feel free to proceed there. Thanks!
@ronaldtse
Thank you. Could you please allow issues on that fork ?

[ronaldtse]

Done!

[maxirmx]

@ronaldtse @CAMOBAP
There is ffi-compiler2 update to 2.2.1 that breaks Windows build. If I pin ffi-compiler2 version to 2.0.1 libmspack installation works.

There is no option to open an issue against https://github.com/davispuh/ffi-compiler and https://github.com/davispuh/ffi-compiler looks dead.

The issue with ffi-compiler2 is that it now uses shelljoin instead of join and shelljoin creates a mess im MSys environment.

Not sure what to do with it.

• we can probably fix metanorma with ffi-compiler2, '= 2.01' if applied before ruby-libmspack in gmemspec. If not, people do monkey patching of shellescape, shelljoin (Shellwords.shellescape is not cross platform safe fastlane/fastlane#13213)

• we can use mini_portile with libmspack instead of ffi-compiler2 which will be more robust, but it will require to maintain a fork of libmspack

• we can ping @davispuh and hope he will read it :)

[CAMOBAP]

'freeze' ffi-compiler2 version to 2.0.1 in packed-mn, looks like a solution to me

[davispuh]

Turns out shell escaping is incredibly complex... Real fix would be for Ruby to support it natively. Alternative could be to rewrite ffi-compiler so it invokes programs directly without going thru shell which is stupid anyway.

Anyway I wrote small escaper for Windows (it might not work correctly for all cases) and pushed ffi-compiler2 v2.2.2

You can always fix things and submit your PRs ffi/ffi-compiler / ffi-compiler2

Also btw it's not true that libmspack doesn't include prebuilt binaries, in fact it does, they are in ext/{x86_64-linux, x86_64-windows} you can inspect gem and see it there. If for some reason it's not picked up it's a bug somewhere.

[maxirmx]

Thank you very much, @davispuh, for the quick response.
I did not feel competent to suggest changes to shelljoin.
I have created a separate discussion about prebuilt binaries.