xkbcommon supports no way for a prefix pattern to override a longer pattern matching the prefix

[mcclure]

I am not sure this is a "bug", however it is an inconvenience which other XCompose parsers do not seem to have.

CONTEXT

I like emdashes— a lot. My custom XCompose contains a custom sequence:

<Multi_key> <minus> <minus>: "—" # mcc's emdash

This is because all standard Compose files, including the /usr locale one on my system, contain

<Multi_key> <minus> <minus> <minus>     : "—"   U2014 # EM DASH

And that's too many minuses for me— I'd prefer two.

TLDR

In my testing, in programs based on xkbcommon, the three-minus emdash code always overrides the two-minus emdash code; if anywhere my custom XCompose includes a file containing the three-minus emdash, then there appears to be no way to get an xkbcommon program to accept the two-minus emdash. This differs from the behavior of non-xkbcommon programs.

"REPRO"

I am on Ubuntu 23.10. My way of testing xkbcommon is I run the "Editor" example program from the Iced repo. With Rust 1.73.0 I check out that repo and run (cd examples/editor && cargo run). The way I know this is using libxkbcommon is if there are errors in the custom XCompose it is libxkbcommon that prints an error. I don't know of a better/less Rusty way of testing libxkbcommon.

When I run this editor program, it does not recognize my <minus> <minus> emdash code. I also cannot get it to recognize any other code that is a prefix of another, for example I can't make a <Multi_key> <parenleft> <parenright> due to the existence in usr locale Compose of <Multi_key> <parenleft> <parenright> <parenright>: "🄯" U1F12F # COPYLEFT SYMBOL. It doesn't matter where in the file I place it, before or after the include "%L". Leaving out include "%L" entirely, of course, means my <minus> <minus> emdash code works.

This behavior must be a conscious choice, because it is documented. See "Conflicting Sequences" here.

This behavior differs from that of other Compose implementations I have tested. Testing with WinCompose for Windows, or on Ubuntu: any ibus (GNOME) application; Firefox, which I think is using ibus; or Google Chrome (which has its own internal Compose implementation) in X or XWayland— in all of these, my custom <minus> <minus> code is able to take priority. I don't know exactly what rule is being followed, but I notice all those other programs load sequences from the system Compose file even if no include "%L" is included, implying those programs are bending over backward to honor user intent even if intent is unclear or the file is "incorrect".

I am not aware of any "controlling" documentation indicating whether it is xkbcommon or the other apps which are correct— IE I'm not aware if any body sets a standard for XCompose. The "XCompose" man page on my system says nothing about conflict resolution and doesn't indicate any way to forcibly "forget" a sequence that has already been registered.

EXPECTED BEHAVIOR

I would expect that XCompose would offer some practical way for me to register my <minus> <minus> sequence. Currently, the only way to do this is to not include system Compose (obviously undesirable), or to make a copy of system XCompose, delete the emdash code, and include that (possible, but undesirable because it means I will not pick up new sequences on future OS updates).

(I think my personal expectation would have been that later sequences override earlier ones, but I don't know if I can defend this.)

Although the documented xkbcommon behavior is well-reasoned, it excludes a reasonable user case because it means the system XCompose (which the user cannot easily control) can prevent the user from being able to program certain sequences.

[wismill]

As a matter of fact, I faced this issue giving the recent update system Compose file in libX11.

I can confirm the different handling of conflicting sequences. I do not use ibus. Given the following:

<A> <B> <C> : "X"
<A> <B> : "Y"
<A> <B> <C> <D> : "Z"
<A> : "✨"

• xkbcommon

  To avoid ambiguity, a sequence is not allowed to be a prefix of another. In such a case, the conflict is resolved thus:

  1. A longer sequence overrides a shorter one.
  2. An equal sequence overrides an existing one.
  3. A shorter sequence does not override a longer one.

  Results:

  • <A>: nothing
  • <A> <B>: nothing
  • <A> <B> <C>: nothing
  • <A> <B> <C> <D> : Z

• VSCode/Firefox: intermediate composing result possible.

  Results:

  • <A>: ✨ (still composing)
  • <A> <B>: Y (still composing)
  • <A> <B> <C>: X (still composing)
  • <A> <B> <C> <D> : Z (composing finished)

I have to say GTK handling is much more intuitive and handy.

[wismill]

I am working on a patch to allow GTK/Ibus behavior.

@whot @bluetech I think the patch would be backward compatible. Is there anything that stops us to propose this new (optional) behavior? By optional, I mean code that is not updated to support the new feature will work the same as before.

[wismill]

@mcclure see #398 for fix. You can try it with

meson devenv -C build xkbcli-interactive-wayland --enable-compose --enable-compose-overlapping

or replace wayland with x11 if relevant.

[whot]

Is there anything that stops us to propose this new (optional) behavior? By optional, I mean code that is not updated to support the new feature will work the same as before.

tbh, I don't know why compose sequences cannot be overlapping but I suspect it just dates back decades with "too hard" being the reason back then. Maybe @fooishbar knows?

[fooishbar]

Afraid it long predates me …

[wismill]

I suspect it just dates back decades with "too hard" being the reason back then.

@whot then I suppose the hard part is not the handling in libxkbcommon but rather the handling in the client. If we reach an overlapping sequence, then:

1. The overlapping result should be displayed to the user. In Gtk apps, it is underlined to inform it is still composing.
2. If the next keysym does not advance the sequence, or the input looses focus, then mark the sequence composed. If a keysym was fed, output it as well.
3. If the next keysym advances the sequence, continue composing.

This looks much like a basic input method. That’s why I asked feedback to the Qt devs in #398.

[bluetech]

I support facilitating overlapping/prefix matches for clients which are able to support basic input method like choice.

@wismill I'll review your PR next time I get a chance.

BTW, unfortunately I am a lazy to test it myself ATM but I'm curious how does GTK handle this, from a user perspective. In the <A> <B> <C>: X (still composing) state, what does the user see (any visual indication)? How does the user commit the X - I suppose any character other than D would do it? And if the user actually does happen to want XD, I guess they should do <A> <B> <C> <ESC> <D>?

[omentic]

@bluetech If your .XCompose has the following lines in it:

<Multi_key> <a> : "α"
<Multi_key> <a> <l> : "∀"
<Multi_key> <a> <l> <p> <h> <a> : "α"
<Multi_key> <b> <e> <t> <a> : "β"
<Multi_key> <b> <e> <t> <h> : "ℶ"

then when you type the following:

• <Multi_key>: a dot with an underscore appears
• <Multi_key> a: an α with an underscore
• <Multi_key> a l p h a: α
• <Multi_key> a: α, with an underscore
• <Multi_key> b: b, with an underscore
• <Multi_key> b e t: b, with an underscore
• <Multi_key> b e t a: β
• <Multi_key> b e t h: ℶ

I think the GTK system is rather good. All of your assumptions about it are correct, yes.

[wismill]

@omentic I quite like too how overlaps are handled in Gtk. We do have a MR to fix the issue: #398, but my proposal to use it in Qt stalled. Not much interest from the Qt devs.