Consider building layout library with CMake and using K(eyboard)L(ayout)I(nstaller)

[davidebeatrici]

I originally found this repository while looking up whether Microsoft Keyboard Layout Creator supports ARM64.
Upon discovering that it doesn't, I decided to:

1. Use MSKLC to generate C source files.
2. Use CMake to build the layout library.
3. Write an installer from scratch: https://github.com/davidebeatrici/kli

For reference: thomasfaingnaert/win-us-intl-altgr#6

Would you be interested in setting up something similar?

[rimas-kudelis]

@davidebeatrici somehow I never noticed your comment until I did just now by accident.

Ugh, that's a difficult question to answer. On one hand, I'm totally for switching build infrastructure of these layouts to something "more open". On the other hand though, I have no plans to change them further, and I don't think Windows on ARM is prominent enough already for me to worry about it. I would only be able test amd64 and wow64, and would have to just blindly trust that the other builds work.

By the way, the driver archive you have in davidebeatrici/kli#1 doesn't even have a wow64 build. I wonder if that was intentional. Probably not?

Also, note that different platforms require different compile flags for layouts to work, so I'm not sure you can have a single CMake config file work for all platforms.

Also some more questions/thoughts about KLI:

• Does it create an uninstall entry in the registry for layout removal?
• It would be nice if its messages were localizable. Right now they seem to be hardcoded to English.
• Why did you decide against MSI? MSKLC did produce installable MSI files, so (I think) you could install the layouts even without running setup.exe.
• One thing I always disliked about MSKLC output files is that you got a bunch of files scattered around which you then had to pack into an actual distributable yourself (e.g. a self-extracting archive). As I see, KLI does exactly the same. I understand that KLI is just an installer and not a package builder (right?), but this bit is still lacking.

---

The rest of this comment is kinda offtopic.

The abandonware status of MSKLC made me think that it would be super awesome to have an open-source replacement for both MLKLC and KbdEdit. KLI could be its nice "behind the hood" tool for generating the installers. However I never even attempted to do anything in that direction because:

1. Like I said above, I have no plans to make further modfications to my layouts
2. Like I said above, Windows on ARM doesn't seem to be much of a thing (yet?)
3. I'm not really a Windows developer
4. I don't use Windows much

There is one thing that bothers me significantly enough and would certainly cause me to rebuild my layouts: it's the "new" layout features from Win8+. I do check from time to time whether anyone has figured out how to make custom keyboard layouts support them, but that doesn't seem to be the case so far. :(

[davidebeatrici]

No worries, thanks for the detailed reply!

By the way, the driver archive you have in davidebeatrici/kli#1 doesn't even have a wow64 build. I wonder if that was intentional. Probably not?

WOW64 is the 32 bit compatibility layer used on 64 bit Windows, am I missing something?

Also, note that different platforms require different compile flags for layouts to work, so I'm not sure you can have a single CMake config file work for all platforms.

Even if a particular architecture required specific flags it would not be a problem, as CMake supports if statements.

Does it create an uninstall entry in the registry for layout removal?

It would be a nice addition, right now you can run KLI (with the same configuration) in order to uninstall the layout.

It would be nice if its messages were localizable. Right now they seem to be hardcoded to English.

Agreed.

Why did you decide against MSI? MSKLC did produce installable MSI files, so (I think) you could install the layouts even without running setup.exe.

One thing I always disliked about MSKLC output files is that you got a bunch of files scattered around which you then had to pack into an actual distributable yourself (e.g. a self-extracting archive). As I see, KLI does exactly the same. I understand that KLI is just an installer and not a package builder (right?), but this bit is still lacking.

I confirm layouts can be installed by running the MSI file, setup.exe simply chooses the correct one depending on the architecture.

Achieving a single independent MSI file for each architecture would indeed be ideal and something worth pursuing.

The abandonware status of MSKLC made me think that it would be super awesome to have an open-source replacement for both MLKLC and KbdEdit. KLI could be its nice "behind the hood" tool for generating the installers. However I never even attempted to do anything in that direction because:

1. Like I said above, I have no plans to make further modfications to my layouts
2. Like I said above, Windows on ARM doesn't seem to be much of a thing (yet?)
3. I'm not really a Windows developer
4. I don't use Windows much

I mainly use Linux, but I write code for other platforms as well.

Windows on ARM is indeed kinda "experimental", to the point that not even Visual Studio is built natively for it yet: https://developercommunity.visualstudio.com/t/native-arm-support-for-visual-studio/1161018

There is one thing that bothers me significantly enough and would certainly cause me to rebuild my layouts: it's the "new" layout features from Win8+. I do check from time to time whether anyone has figured out how to make custom keyboard layouts support them, but that doesn't seem to be the case so far. :(

Which features are you referring to, in particular?

[rimas-kudelis]

@davidebeatrici:

By the way, the driver archive you have in davidebeatrici/kli#1 doesn't even have a wow64 build. I wonder if that was intentional. Probably not?

WOW64 is the 32 bit compatibility layer used on 64 bit Windows, am I missing something?

Yes, and it requires a separate keyboard layout DLL. The two links below discuss that.

I suppose this might also be the case when running x86 apps on arm64, as I read that that emulation also uses a similar WoW64 emulation mechanism.

Also, note that different platforms require different compile flags for layouts to work, so I'm not sure you can have a single CMake config file work for all platforms.

Even if a particular architecture required specific flags it would not be a problem, as CMake supports if statements.

Okay, that's cool!

Why did you decide against MSI? MSKLC did produce installable MSI files, so (I think) you could install the layouts even without running setup.exe.

One thing I always disliked about MSKLC output files is that you got a bunch of files scattered around which you then had to pack into an actual distributable yourself (e.g. a self-extracting archive). As I see, KLI does exactly the same. I understand that KLI is just an installer and not a package builder (right?), but this bit is still lacking.

I confirm layouts can be installed by running the MSI file, setup.exe simply chooses the correct one depending on the architecture.

Achieving a single independent MSI file for each architecture would indeed be ideal and something worth pursuing.

And apparently it is possible, as I just stumbled upon this link. It requires having WiX Toolset installed, but it's not a big price to pay, if everything else works. I might try it out next weekend or some other time.

The abandonware status of MSKLC made me think that it would be super awesome to have an open-source replacement for both MLKLC and KbdEdit. KLI could be its nice "behind the hood" tool for generating the installers. However I never even attempted to do anything in that direction because:

1. Like I said above, I have no plans to make further modfications to my layouts
2. Like I said above, Windows on ARM doesn't seem to be much of a thing (yet?)
3. I'm not really a Windows developer
4. I don't use Windows much

I mainly use Linux, but I write code for other platforms as well.

Oh, same here, except I'm a web developer. These keyboard layouts are pretty much my only pet projects on Windows/Mac, and even those are rather dormant.

There is one thing that bothers me significantly enough and would certainly cause me to rebuild my layouts: it's the "new" layout features from Win8+. I do check from time to time whether anyone has figured out how to make custom keyboard layouts support them, but that doesn't seem to be the case so far. :(

Which features are you referring to, in particular?

I've listed them in the README. Stock Windows 8 and 10 keyboard layouts are shipped with several new features, namely:

• abbreviated layout name (e.g. US), shown in the task bar when more than one layout is enabled for a single language,
• the QWERTY/AZERTY/* descriptor string in language options,
• touch-optimized layouts.

However, the keyboard layout dll file format seems to have remained unchanged, and this new functionality was instead placed elsewhere. I was unable to find ways to add these features to my layouts when I looked, and it doesn't seem like anyone else would've had any success there either.

[rimas-kudelis]

@davidebeatrici so I booted to Windows and used burn.exe from WiX toolset to decompile the MSI file.
What it creates from it is a source WXS file and a single DLL binary containing custom actions for the installer. The custom actions that it provides are:

1. RegisterKeyboard (entrypoint CA01)
2. UnregisterKeyboard (entrypoint CA02)
3. AddKeyboardToLangBar (entrypoint CA03)
4. RemoveKeyboardFromLangBar (entrypoint CA04)

So the only thing that currently prevents us from having a completely open-source MSI version of that MSKLC-generated installer is this DLL.

With this extension at hand, proper per-platform (and maybe even cross-platform, if MSI allows that) MSI files could be generated.

Would you be interested in transforming KLI into such WiX extension?

A setup.exe could then also be created as a thin wrapper, although I guess it would only be useful if/when platform-specific MSIs were used. On the other hand, if a single MSI file would be be generated, it would be more practical to just package that MSI in a self-extracting archive instead and launch msiexec automatically after extracting.

What do you think?

[rimas-kudelis]

Further findings:
the "stock" custom actions DLL is available in the MSKLC installation directory as KbdMsi.dll. Although it's slightly smaller than what dark.exe extracts, all DLL info (including original file name and version number) matches, so I assume dark.exe just adds some padding at the end. I inspected both the original and the extracted files using DLL Export Viewer, and they both export those same four entry points at same addresses.

I md5sum'ed all three of these dll's extracted by burn.exe (one from each platform msi), and they are all identical, so it's indeed the same file in all cases (BTW, apparently, this is how WiX 3 works, and it's also mentioned here, although in this case even ia64 didn't get a separate build).

Apparently, this might be an issue when installing on a machine without WoW64 (apparently, some server editions now ship without this layer), so here's one more reason (in addition to licensing) to rewrite it instead of just copying it.

[davidebeatrici]

Please note that I documented the relevant WinAPI functions: https://github.com/davidebeatrici/kli/wiki/WinAPI-reference

Transforming KLI into a WiX extension is an excellent idea, I will attempt that.

As for the single MSI file in a self-extracting package: it would be ideal.

[rimas-kudelis]

@davidebeatrici It's really great that you have these functions documented already!

I'm currently trying to recompile the KLC installer using wix, just to see that the resulting file works.
When it does, it would be cool to be able to easily replace the current KbdMsi.dll with an open-source one (although I wouldn't mind if exported entry points had better names than CA01 – CA04).

[rimas-kudelis]

Less good news: you cannot have multiplatform MSI files. However, a single wix source file could be used to generate msi packages for all platforms.

I've diffed all three decompiled wxs files, and their differences are neglible (randomly-generated identifiers differ, and the 32-bit version has two extra conditions specified which prevent it from being installed on 64-bit systems), so I guess a single wxs with multiple msi's is the way to go...

[rimas-kudelis]

@davidebeatrici I've committed my current progress to the new-installer branch.

Things are looking good. :) I've cleaned up the wxs file and wrote a batch file to build all three of the current installers. Interestingly, the resulting MSI files are even smaller than the ones produced by MSKLC, even though they already embed the DLLs they are installing (although I admit, dark did warn me during decompilation that some functionality would be left out).

I've tested, and all three MSIs work as expected on my platform (amd64): the ia64 one is refused by MSI, the i386 one rejects installation with a custom message, and the amd64 one gets installed and enables the layout.

[rimas-kudelis]

To avoid spamming this "issue" page further, I created a wiki page too, to collect whatever info I might find relevant to any effort of rewriting/replacing (parts of) MSKLC and/or working around its issues.

[rimas-kudelis]

Transforming KLI into a WiX extension is an excellent idea, I will attempt that.

@davidebeatrici, do you have an idea when you'd be willing to attempt it?

[davidebeatrici]

Sorry for the delay, I've been busy with other projects. The wiki page you created is awesome!

I unfortunately don't have an ETA for when I'll attempt to write the WiX extension, feel free to step in if you want to work on it instead.

[rollingmoai]

Windows on ARM is indeed kinda "experimental", to the point that not even Visual Studio is built natively for it yet: developercommunity.visualstudio.com/t/native-arm-support-for-visual-studio/1161018

Visual Studio 2022 17.3 Preview 2 is now available and is the first version of Visual Studio to run natively on ARM devices running Windows 11.
It would be great to see ARM versions of the keyboard layouts.

[davidebeatrici]

Awesome!

https://developercommunity.visualstudio.com/t/native-arm-support-for-visual-studio/1161018

It was actually already possible to build keyboard layouts for ARM64, with a cross compiler.

[rimas-kudelis]

A little status update.

• I got myself a cheap ARM64 Windows PC and dug around a little. Some notes here.
• Also, I stubbed the binaries MSKLC 1.4 uses to build the layouts, and now have the exact commands and arguments used by kbdutool supplied with that version of MSKLC. For all platforms it supported.
• I also had a look at CMake (on an Intel machine) and, honestly, I'm not sure I like it in the context of this project. I want to be able to use a single target to build the layout for all architectures, but CMake requires to specify an architecture when building, so at the very least, I would need to wrap its magic in a batch file or shell script. On the other hand, if it allowed me to build the driver using Linux, that probably wouldn't be such a bad deal after all. Then again, I'm just not sure how I would build the layout (possibly even two versions of it) for WOW64, for example.
• What I did like about CMake though was its ability to produce text files from templates, replacing stuff like version numbers and dates.
• I also tried going the NMake route instead. I installed Visual Studio, wrote a Makefile and tried compiling my layout. But I could only compile it for x86 and x64 and not for ARM (even though I had ARM support installed), and I haven't tested whether these compiled drivers even work. This will obviously need some more work.

[davidebeatrici]

Thanks for the update!

I'm surprised you couldn't get the layout to compile for ARM using NMake, since it's what CMake uses (unless Ninja is set).

Building for different platforms without explicitly calling CMake multiple times is possible, but it's far more complex than it should be with Visual Studio:

https://github.com/mumble-voip/mumble/blob/eb01d1b6d607cb79531bf359500c07bfa4a329e6/overlay/CMakeLists.txt#L150-L196

https://github.com/mumble-voip/mumble/blob/master/overlay/scripts/build_overlay_xcompile.cmd

On Linux it would just be a matter of selecting the corresponding cross-compiler, ideally through the use of a specific toolchain.
Technically this can be done with Visual Studio as well, but as far as I know it can turn out to be even more cumbersome than the method we're using in Mumble right now.

As for the commands executed by kbdutool, it's very useful info! I'm pretty sure the need for most parameters is now a thing of the past, but we can definitely use them in an attempt to fix any issues that may come up, especially at runtime.

[rimas-kudelis]

I'm surprised you couldn't get the layout to compile for ARM using NMake, since it's what CMake uses (unless Ninja is set).

I mean I couldn't do that from a single VS "Tools Command Prompt" the way I wanted (although yes, I wasn't able to do that at at all, in the end). Maybe I just started the wrong one? Either way, if kbdutool can do it with only a few files from MSVC installed, so should we. I think.

I was thinking, perhaps I should go dump environment variables that these build commands are started with, in addition to just the command line...

[davidebeatrici]

Yes, the environment variables are mandatory.

[rimas-kudelis]

I've updated the wiki page with environment vars. Not much to see there, as it doesn't seem like kbtutool would add many. I only noticed one addition when it's calling link.exe for 64-bit platforms.