Errors installing v1.5 via pip on multiple distros

[RedBearAK]

@sde1000 @flacjacket @mbey-mw

Somebody using a project of mine that installs xkbcommon in a Python virtual environment filed a report this morning about a weird installation error on Ubuntu 20.04. I was able to replicate on Ubuntu, and just kind of assumed the libxkbcommon on Ubuntu 20.04 was too old for the latest xkbcommon release, even though that didn't make too much sense. So I pinned the version to something older for Ubuntu 20.04 and it started working again.

But then I tested other distros and I've been running into the same error, where previously I was installing xkbcommon without any issue on distros as old as CentOS 7, as long as the libxkbcommon development package was installed.

For now, I'm pinning the version to avoid the new 1.5 release.

xkbcommon<1.5

This makes the install use the 1.0.1 version that was released on the same day. Yesterday.

https://pypi.org/project/xkbcommon/#history

Since I don't know exactly what is going wrong, I'm hesitant to rely on that. I'm thinking about pinning to the 1.0.1 release explicitly until I know more about the cause of this problem.

The error messages go like this, during installing packages in the venv with pip.

Collecting xkbcommon
  Downloading xkbcommon-1.5.tar.gz (79 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.9/79.9 kB 384.0 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done

...
[lots of other pip packages downloading/installing]
...

Building wheels for collected packages: dbus-python, systemd-python, pygobject, xkbcommon, inotify-simple, evdev
  Building wheel for dbus-python (pyproject.toml) ... done
  Created wheel for dbus-python: filename=dbus_python-1.3.2-cp38-cp38-linux_x86_64.whl size=125039 sha256=79164b04108e2524fc2b2836bb6c38b7b208f4d4b2a185fe658ba3e27909e968
  Stored in directory: /home/testuser/.cache/pip/wheels/ee/6d/3b/ba4374f33fc6b6c2147f5434a897800f99a6eb2a2250714829
  Building wheel for systemd-python (setup.py) ... done
  Created wheel for systemd-python: filename=systemd_python-235-cp38-cp38-linux_x86_64.whl size=185184 sha256=94ad68b89d02d752b0bcb4cef5f0af74ddc4a8e39c6ac11a0cd6cf893ebd179e
  Stored in directory: /home/testuser/.cache/pip/wheels/40/5b/21/f6b14253e0aa61edffdac63d5ddeeb25f1c092f2259ec6ab6e
  Building wheel for pygobject (pyproject.toml) ... done
  Created wheel for pygobject: filename=PyGObject-3.44.1-cp38-cp38-linux_x86_64.whl size=895199 sha256=241f5af85470028ca1c280258c5227058182e0ab891c0a77e82d72774929af90
  Stored in directory: /home/testuser/.cache/pip/wheels/b0/b5/51/317e6588f7bda212431c770cc26c930e88673899b6f372b254
  Building wheel for xkbcommon (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for xkbcommon (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [30 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-38
      creating build/lib.linux-x86_64-cpython-38/xkbcommon
      copying xkbcommon/__init__.py -> build/lib.linux-x86_64-cpython-38/xkbcommon
      copying xkbcommon/ffi_build.py -> build/lib.linux-x86_64-cpython-38/xkbcommon
      copying xkbcommon/xkb.py -> build/lib.linux-x86_64-cpython-38/xkbcommon
      warning: build_py: byte-compiling is disabled, skipping.
      
      running build_ext
      generating cffi module 'build/temp.linux-x86_64-cpython-38/xkbcommon._ffi.c'
      creating build/temp.linux-x86_64-cpython-38
      building 'xkbcommon._ffi' extension
      creating build/temp.linux-x86_64-cpython-38/build
      creating build/temp.linux-x86_64-cpython-38/build/temp.linux-x86_64-cpython-38
      x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/testuser/.config/toshy/.venv/include -I/usr/include/python3.8 -c build/temp.linux-x86_64-cpython-38/xkbcommon._ffi.c -o build/temp.linux-x86_64-cpython-38/build/temp.linux-x86_64-cpython-38/xkbcommon._ffi.o
      build/temp.linux-x86_64-cpython-38/xkbcommon._ffi.c: In function ‘_cffi_const_XKB_CONTEXT_NO_SECURE_GETENV’:
      build/temp.linux-x86_64-cpython-38/xkbcommon._ffi.c:1017:12: error: ‘XKB_CONTEXT_NO_SECURE_GETENV’ undeclared (first use in this function); did you mean ‘_cffi_const_XKB_CONTEXT_NO_SECURE_GETENV’?
       1017 |   int n = (XKB_CONTEXT_NO_SECURE_GETENV) <= 0;
            |            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
            |            _cffi_const_XKB_CONTEXT_NO_SECURE_GETENV
      build/temp.linux-x86_64-cpython-38/xkbcommon._ffi.c:1017:12: note: each undeclared identifier is reported only once for each function it appears in
      build/temp.linux-x86_64-cpython-38/xkbcommon._ffi.c: In function ‘_cffi_d_xkb_keymap_key_get_mods_for_level’:
      build/temp.linux-x86_64-cpython-38/xkbcommon._ffi.c:2760:10: warning: implicit declaration of function ‘xkb_keymap_key_get_mods_for_level’; did you mean ‘xkb_keymap_key_get_syms_by_level’? [-Wimplicit-function-declaration]
       2760 |   return xkb_keymap_key_get_mods_for_level(x0, x1, x2, x3, x4, x5);
            |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            |          xkb_keymap_key_get_syms_by_level
      error: command '/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for xkbcommon
  Building wheel for inotify-simple (setup.py) ... done
  Created wheel for inotify-simple: filename=inotify_simple-1.3.5-py3-none-any.whl size=7685 sha256=fc56c2245113704c99c39176c96f15d7d913fe99201644bcd03144e58f5ce156
  Stored in directory: /home/testuser/.cache/pip/wheels/85/b2/be/354e28439e9b15a9a77924041be045e499f11bb03493529246
  Building wheel for evdev (pyproject.toml) ... done
  Created wheel for evdev: filename=evdev-1.7.1-cp38-cp38-linux_x86_64.whl size=108600 sha256=eff409dc98f3734f75572e4c319d4f72ee68eda01064f4dbcd9785bd1b9eb311
  Stored in directory: /home/testuser/.cache/pip/wheels/40/a7/e8/d118cb622045826cc85b320ab9f6ebe89d9b88f2e8150be72d
Successfully built dbus-python systemd-python pygobject inotify-simple evdev
Failed to build xkbcommon
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (xkbcommon)

I have a wide selection of different distros I have to keep this working on, if I'm going to use this Python module. Just want to know what's happening here.

[sde1000]

From the README:

From release 0.5 onwards, the version numbering of this package will relate to releases of libxkbcommon as follows:

If the Python package version is major.minor[.patch] then it requires at least release major.minor.0 of libxkbcommon to build and run, and should work with any subsequent release. The patch version of the Python package is unrelated to the patch version of libxkbcommon.

So release 1.0.1 requires libxkbcommon-dev >= 1.0, release 1.5 requires libxkbcommon-dev >= 1.5, and so on. I don't know how to express this requirement in the setup.cfg file, though.

If you don't use any features of releases 1.5 and 1.6 then it is safe to require <1.5 as you have done.

[RedBearAK]

Yeah, I saw that note, but the version of libxkbcommon-dev on Ubuntu 20.04 was 0.10.1 or something, so I really didn't know what to make of it in relation to the note.

For now I've gone ahead and pinned xkbcommon to <=1.0.1 for everything. The fact that two different versions were released at the same time had me a little confused, and concerned that another parallel release between 1.0.1 and 1.5 might cause the same problem. (?)

At the moment I have no understanding of which distros and distro versions the 1.5 release would even work on. I'd have to try to test the version of libxkbcommon-dev available on different distro types (under different package names), and on Ubuntu 20.04 it definitely wasn't 1.0 or greater. So I don't really understand the relationship.

I also have no clue if I would ever use features in 1.5 or later. I have only been using this module for accessing the list of keyboard layouts, and the keymaps. Don't really know much more about what this is for.

The plan is to use the extracted keymaps to help an evdev keymapper be smarter about working with non-US layouts. But I haven't gotten very far with that.

[sde1000]

I think I'll have to address this with a documentation update, listing relevant versions of libxkbcommon and the API features introduced in them and recommending that people always specify a maximum version when depending on python-xkbcommon. For your case I'd recommend depending on xkbcommon<1.1 so you will pick up any bug fixes released in the 1.0 series.

If you are also using python-xkbregistry then I'd recommend depending on xkbregistry<1.1 as well.

Ubuntu 20.04 has libxkbcommon-0.10 — I am surprised python-xkbcommon 1.0.1 still works on it, I removed ubuntu-20.04 from the CI after release 0.8!

• libxkbcommon-0.8 added xkb_keysym_to_upper() and xkb_keysym_to_lower()
• libxkbcommon-1.0 added xkb_keymap_key_get_mods_for_level()
• There were no relevant API changes in libxkbcommon-1.1 to libxkbcommon-1.4
• libxkbcommon-1.5 added XKB_CONTEXT_NO_SECURE_GETENV
• libxkbcommon-1.6 added an API to iterate over compose tables
• There are no relevant API changes in libxkbcommon-1.7

I plan to maintain branches for 1.0, 1.5 and 1.6 going forward, at least until there are no supported distros left that have earlier libxkbcommon than 1.5.

[RedBearAK]

I think I'll have to address this with a documentation update, listing relevant versions of libxkbcommon and the API features introduced in them and recommending that people always specify a maximum version when depending on python-xkbcommon.

For those who read the documentation like they should, that would be good. Wouldn't have helped me, I didn't really expect this to be something that would change much. Shows what I know.

For your case I'd recommend depending on xkbcommon<1.1 so you will pick up any bug fixes released in the 1.0 series.

Sounds like a good idea. I'll do that. A lot of Linux users are stuck on LTS distros that probably won't be updating libxkbcommon-dev anytime soon.

If you are also using python-xkbregistry then I'd recommend depending on xkbregistry<1.1 as well.

OK, but I actually kind of had the opposite problem with xkbregistry. It was so new that it had no supporting package on AlmaLinux 8.x (my proxy for RHEL 8 and clones). I gave up on using it. Seems like I can get what I need from xkbcommon.

Unless xkbregistry has the miraculous ability to tell me the user's currently active layout at any given time, and when it changes. That would be incredibly helpful. I have found some D-Bus signals in KDE Plasma and a dconf key that can be watched with dconf watch in GNOME, but all other methods I've found on the web for knowing the active layout have failed to show anything but a list of enabled layouts. Not the active one, or when the user switches layouts.

It is my current understanding that neither xkbcommon nor xkbregistry has the ability to know the currently active layout. I have to find some way to talk to X11 or the Wayland compositor, or use some other native method like those I recently discovered for Plasma and GNOME. From Python.

I plan to maintain branches for 1.0, 1.5 and 1.6 going forward, at least until there are no supported distros left that have earlier libxkbcommon than 1.5.

That's... gonna be a while, I think.

Thanks for the info. I should be good from now on, I hope. 👍🏽

[sde1000]

You should be able to get the currently active layout for Wayland reasonably easily — the way keyboard handling works in Wayland clients is that the compositor passes the keymap to the client (by sending an fd for them to mmap), and the client then uses something like libxkbcommon to interpret the keymap and make sense of the keycodes being sent.

There's an example of this in the demo app for my pure Python Wayland implementation but please don't base any new code on that — use pywayland instead if you want to go down that route.

Under X, you would probably want to use libxkbcommon-x11 as described in #14. Someone will have to write the Python bindings for that, though — it'll be a separate project that depends on python-xkbcommon and also xcffib.

[sde1000]

I've released version 1.5.1 with the documentation update so it will show on the project page on pypi.

I will hold off releasing version 1.6 because that will catch out even more people (it's only present in Ubuntu 24.04 as far as I can tell — Debian bookworm only has libxkbcommon-1.5). Let's give it six months to a year before a release...