Windows build - info/discussion

[avih]

Hi,

I've built txiki.js for windows, here are some thoughts and data points which others might find useful. This could also be a basis for a readme file or wiki page, or some patches.

curl

On windows, the TLS backend in curl can be either schannel (windows store) where Windows manages the certificates, or a third party TLS lib, like OpenSSL. The latter requires certificates in files/bundles.

A single [lib]curl binary may support more than one backend.

using mingw in MSYS2

The MSYS2 mingw libcurl is configured to use a certificate bundle in a dir relative to the curl dll (DLL-DIR/../ssl/certs/), and so if one builds tjs.exe using the MSYS2 mingw packages then a sequence similar to this could make the bundle portable:

# MSYS2: build txiki.js and then package it at ./dist/ with dll deps and certs
make \
&& rm -rf dist \
&& mkdir -p dist/bin \
&& cp -a build/tjs.exe dist/bin/ \
&& cp -a $(ldd build/tjs.exe | awk '/mingw/ { print $3 }') dist/bin/ \
&& mkdir -p dist/ssl/certs \
&& cp -a /$MSYSTEM/ssl/certs/* dist/ssl/certs/

It's possible that the curl dll also support schannel and/or configurable certs path (maybe using env vars), but I've not looked into it.

cross compiling using MXE

I've managed to build tjs.exe using MXE - https://github.com/mxe/mxe on debian. MXE builds a mingw toolchain from source and has recipes for many third party packages (e.g. including curl and libffi). It can also be configured to default to static libs and linkage, which helps with creating a single tjs.exe without dependencies. It also helps that the curl recipe in MXE doesn't seem to need a certs bundle file for https to work out of the box with tjs.exe (defaults to schannel?).

However, using MXE, especially for a one-off build, can be a long process, as building the mingw toolchain and then curl (and its many dependencies) can take quite a while. However, it one already has an MXE setup then it's a viable option.

I did have to work around two issues though: capital includes names in wasm3 (see below), and the curl dependencies were not picked automatically by cmake, so I added them manually at CMakeLists.txt.

cross compiling using debian mingw

Another approach is to use mingw from the distro. I've used debian 11.5 live DVD with the script below to build a 32/64 portable txiki.js package.

This does not build curl because it can be complex, and instead downloads the official reproducible curl-for-windows binary - https://github.com/curl/curl-for-win . All the other deps (quickjs, libuv, wasm3, libffi) are built from the txiki.js git repo and linked statically.

Some data points:

• The debian mingw package includes only the toolchain (for 32/64), and there are no additional mingw libs or helpers. Specifically, cmake needs a custom toolchain file and a wrapper which uses this file where applicable. The script creates both files and uses them for the build.
• The wasm3 sources have two windows includes with capitalized file names, while the mingw headers are lower-case. That's not an issue in MSYS2 mingw because on windows the fs is case-insensitive, but on linux it's not. So the script patches it to use lower-case header names (todo: report upstream).
• The script always downloads the latest curl-for-windows (from a fixed URL). The zip includes headers and static/shared libs. It should be possible to use the static lib, but the script uses the shared lib because it's easier to setup - it doesn't need the additional static libs for linkage (openssl etc).
• the libcurl binary is configured to use OpenSSL without any pre-configured certs path, and so by default https fails. The script patches txiki.js to setup a cert bundle file name (and takes the bundle from the zip). The binary also support schannel - without a cert bundle file, but it was not explored further, and the default is OpenSSL.
• CMakeLists.txt is patched to use a static libpthread.a instead of shared lib (dll).
• The resulting package has tjs.exe, curl.dll, and the certs bundle (and few info files).

Attached are the build script, and the resulting 64 bits package from current git master, built on debian 11.5 live DVD in a VM (the package inclues the script too at info/).

To use the script on a freshly booted Debian 11 live DVD: save it e.g. build-tjs.sh and chmod +x build-tjs.sh, then ./build-tjs.sh to build the 64 bits package, or BITS=32 ./build-tjs.sh for 32 bits (it first installs the required packages). zipped package path is reported at the end of the build.

txiki.debian-mingw-build-script.zip
txiki.js.x86_64-w64-mingw32.zip

[saghul]

Thanks a lot for this! Some comments inline.

Hi,

I've built txiki.js for windows, here are some thoughts and data points which others might find useful. This could also be a basis for a readme file or wiki page, or some patches.

Certainly. Currently there is a Windows section on the README, maybe you could improve that?

curl

On windows, the TLS backend in curl can be either schannel (windows store) where Windows manages the certificates, or a third party TLS lib, like OpenSSL. The latter requires certificates in files/bundles.

A single [lib]curl binary may support more than one backend.

using mingw in MSYS2

The MSYS2 mingw libcurl is configured to use a certificate bundle in a dir relative to the curl dll (DLL-DIR/../ssl/certs/), and so if one builds tjs.exe using the MSYS2 mingw packages then a sequence similar to this could make the bundle portable:

# MSYS2: build txiki.js and then package it at ./dist/ with dll deps and certs
make \
&& rm -rf dist \
&& mkdir -p dist/bin \
&& cp -a build/tjs.exe dist/bin/ \
&& cp -a $(ldd build/tjs.exe | awk '/mingw/ { print $3 }') dist/bin/ \
&& mkdir -p dist/ssl/certs \
&& cp -a /$MSYSTEM/ssl/certs/* dist/ssl/certs/

It's possible that the curl dll also support schannel and/or configurable certs path (maybe using env vars), but I've not looked into it.

I think that's the winssl libcurl backend, isn't it? That's how I build it in the CI:

txiki.js/.github/workflows/ci.yml

Line 117 in 8a521df

curl-winssl:p

[snip on cross-compiling]

While useful, GH actions can also do MSYS2 so I guess this would be useful for those who want to cross-compile it themselves? At any rate, another nice addition to the README!

[avih]

Currently there is a Windows section on the README, maybe you could improve that?

Sure, anything specific you had in mind? I think maybe just adding this shell snippet to package the files and the certs in MSYS2?

I guess this would be useful for those who want to cross-compile it themselves?

Well, my hope was to be able to create a statically linked tjs.exe, preferably without additional required files (no concrete reasons other than some subjective sense of elegance of being 100% self-contained).

The msys2 build has quite a lot of dll deps, and the cert bundle file seem to be expected at an awkward place relative to the binary. Also, the MSYS2 mingw packages are quite bleedng edge, and one might wish to depend on maybe more stable packages (like the official curl-for-windows).

This (debian mingw) build procedure can generate tjs.exe which depends only on the curl dll and cert bundle at the same dir. I believe it should be possible to link statically to libcurl (but the zip doesn't have a .pc file), and possibly also use schannel if a cert bundle is unavailable (didn't try it yet).

So it's just another method to build. Not better or worse, just another option.

[saghul]

Well, my hope was to be able to create a statically linked tjs.exe, preferably without additional required files (no concrete reasons other than some subjective sense of elegance of being 100% self-contained).

This was my goal too. I might have missed something in your previous (and detailed!) post. Are you able to get a statically linked libcurl so the end result if a fully static binary?

and the cert bundle file seem to be expected at an awkward place relative to the binary.

Not with the winssl variant, since it uses the system cert store. I think this is the way to go, is it possible to make such a build in the environments you tested?

So it's just another method to build. Not better or worse, just another option.

Got it now. So I'd suggest just adding these options to the README around the windows section.

[avih]

Are you able to get a statically linked libcurl so the end result if a fully static binary?

I am, but so far only using the MXE method. Also, I think the MXE [lib]curl recipe uses schannel - because https in tjs.exe works without a certs file (that I know of).

With the debian mingw build it's close, but not quite there yet. The official curl-for-windows zip does include libcurl.a (and the additional static libs it needs to link with, like libgsasl.a and few more), but it does not include a .pc file, and in a single attempt to specify the libs manually by editing CMakeList.txt I think I got the order wrong and it failed, but it should be possible.

So for now the debian mingw script links with libcurl.dll.a, and needs the curl dll.

Not with the winssl variant, since it uses the system cert store.

Right, I'll admit I didn't actually follow the instructions at the README, and didn't install curl-winssl. I just started the mingw64 shell and ran make with whatever mingw libs I already had installed, and it worked, but https failed unless the certs were copied too. I guess I had plain (mingw) curl package.

I think this is the way to go

I don't know enough to have a strong opinion here. IIRC curl recommends using OpenSSL instead of schannel, and I think this is the reason the official binary defaults to OpenSSL.

It shoud be possible to switch the runtime backend to schannel, but I didn't try it yet.

Personally I think it would be nice if it defaults to schannel, unless it finds a certs bundle and then use it instead. This way one can both let windows handle it, or override it with custom certificates.

EDIT - Im attaching the MXE statically linked (stripped) tjs.exe with my build notes:
txiki.js-win32---x86-64--2022-11-11--built-avih-deban-11.5-MXE-.zip

[saghul]

Thank you! This is very helpful. I'll certainly take a look at MXE for distributing binary build for people to try.

[KopiNinja]

any news on this? it would be awesome to just download from Github releases and run everything in a single binary.

[saghul]

No news, sorry. Happy to take patches if anyone can submit them.

[ahaoboy]

The msys2 environment can compile txiki correctly, but you may need to install some dependent packages

[ahaoboy]

Is it possible to use CI build to publish commonly used binary files to GitHub release

[saghul]

Yes, I'm planning on doing that for the next release

[ahaoboy]

I tried to add a ci, you can directly download the windows file

https://github.com/ahaoboy/txiki.js/releases

[saghul]

Great start!

If you want to push it over the finish line there are a few things that need to be taken care of:

• Pack all the necessary DLLs on Windows. Placing them on the same ZIP file is enough. The README contains instructions on how to list them.

• Make sure the macOS version is multi-arch, so both x64 and arm64.

• For Linux I think it's best to leave it out until we can have a fully static build, since we cannot make a build what will work everywhere.

If you want to go for it, I'd take a PR!