Auto-type Extended ASCII in FireFox prints incorrect/delayed characters

[TheBeardOfTruth]

Linux-5.2.5-x86_64-ARCH, Ryzen 7 2xxx series CPU

Upon invoking auto-type using global hotkey on the website reddit.com we get an incorrect password printout, changing the text field from 'password' type to 'text' type reveals the exact issue - the password in question gets reproduced incorrectly (quickly testing this with sublime text 3 reveals that it's not not necessarily an issue with KeePassXC's ability to type these characters, rather how it types them into firefox)

I have prepared a dummy entry for use as an example [{DELAY 1500}{PASSWORD}{ENTER}].

Expected output:
òöMèÁ[¦r´Ö~èb^ø?(ããùôÁYhËpÛ(ÓÖkÑÞYhåå$%¾D«Ô½µ´ñ*xߦ4ÐOï5ó«É¥ñÒ½ù

Actual output when auto-typed into reddit.com's password field:
Á´Mèø[¦rãÁ~Ëb^Ö?(åå¾´ßYhïpÉ(ñùkùùYhùù$%ùDùùùùùù*xù¦4ùOù5ùùùùùÒùù

Actual output when auto-typed into github.com's issues field 3x to illustrate the ways in which it fails:

ÖãMËÞ[¦rÔÐ~ñb^ù?(ùùùùùYhùpù(ùùkùùYhùù$%ùDùùùùùù*xù¦4ùOù5ùùùùùÒùù
´ãMËÖ[¦rå´~Ðb^¥?(ùùùùùYhùpù(ùùkùùYhùù$%ùDùùùùùù*xù¦4ùOù5ùùùùùÒùù
´ãMÖ«[¦rïñ~ùb^ù?(ùùùùùYhùpù(ùùkùùYhùù$%ùDùùùùùù*xù¦4ùOù5ùùùùùÒùù

As we can see they break in similar ways, chiefly with two major issues: Dropped characters (standard delay of 25ms, typing normally does not exhibit the same noticeable drop in responsiveness. This is probably related) and incorrectly entered characters.

The same test as above with this [{DELAY 1500}{DELAY=150}{PASSWORD}{ENTER}] auto-entry line:

öÁMÁÖ[¦rèè~øb^ã?(ôôÁËÛYhÓpÖ(ÑÞkååYh¾¾$%ÔDµññßÐï*xó¦4ÉOñ5ùùùùùÒùù
öÁMÁÖ[¦rèø~øb^ù?(ÁÁÁÛÓYhÖpÞ(å¾k«½Yhµµ$%ñDßßïïÉñ*x½¦4ùOù5ùùùùùÒùù
öÁm´è[¦rèø~ôb^Á?(ÛÛÖÞåYh¾p½(ñßkßïYhóó$%¥D½ùùùùù*xù¦4ùOù5ùùùùùÒùù

We can, after these debug things infer the following:

1. The issue is not site-specific (or is site-specific to reddit and github).
2. The issue is probably application-specific (FireFox, probably others) as it does not happen in sublime text.
3. The issue has clearly observable symptoms. When using an extended ASCII password it will sometimes drop characters and will enter characters incorrectly.
4. Additionally we note that both KeePassXC and FireFox become very unresponsive (chiefly due to upwards of 4-5 seconds of lag before entered characters show up and they show up in groups upwards of 15 at a time, rather than one at a time as expected), even with slow keystrokes. Noticeably this causes significant CPU spikes from idle ~0.7% upwards of 28-30% CPU usage.
5. This issue is not present under Microsoft Windows 10 Professional 1903 either, it would be safe to assume that the issue lies in either only the Linux backend or in all non-Windows backends (though that's not my place to speculate about, all we can safely say is that this issue is reproducible on my machine)

Normal ASCII-range characters do not exhibit this issue, auto-type appears to work correctly for every other password.

The relevant specs, in detail:

Arch Linux, Kernel 5.2.5 x86_64
Ryzen 7 2700 (μcode patch level 0x0800820b)
16GB DDR4-3200MT/s
GTX 1070

Full debug info output:

KeePassXC - Version 2.4.3
Revision: 5d6ef0c

Qt 5.13.0
Debugging mode is disabled.

Operating system: Arch Linux
CPU architecture: x86_64
Kernel: linux 5.2.5-arch1-1-ARCH

Enabled extensions:

• Auto-Type
• Browser Integration
• SSH Agent
• KeeShare (signed and unsigned sharing)
• YubiKey

Cryptographic libraries:
libgcrypt 1.8.4

Notably browser integration is loaded, but not set for any current browser due to a preference of more loosely interacting programs. I believe this should not inherently be an issue.

[droidmonkey]

Considering sublime text does not exhibit issues I would squarely place the blame on Firefox. Have you reported it to the Firefox developers?

[TheBeardOfTruth]

I would argue that it is, at least partially, an issue with KeePassXC since KeePass2 enters the characters correctly (as do similar tools such as XDOTOOL, I'm not entirely familiar with what KeePass2 uses as a backend, it may be piggybacking on XDOTOOL but even so this at least proves that FireFox can receive special character input, at least in some limited context).

i,e the method used by KeePassXC to enter extended ascii is not as compatible as one might want. While I believe that FireFox perhaps should also look into the cause for this on their end... it isn't outside the realm of plausibility that other applications may exhibit similar incompatibilities, therefore I still believe that it is prudent to investigate the cause from the perspective of KeePassXC's implementation.

But yes, I definitely should FWD this report to the firefox bug tracker as well (though I expect it'll be ignored or flagged WONTFIX, they tend not to spend overmuch effort on smaller things like this, at least in my experience).

Not that this is a huge priority issue either way, a mere annoyance with migrating to an overall more well-behaved implementation.

[droidmonkey]

I ran some qualitative tests myself. Firefox on Windows 10 does not exhibit this behavior at all. Firefox on Ubuntu 19.04 exhibits the described behavior no matter what the typing delay is set to. Also observed that TextEditor exhibited a significant delay between the characters being typed and them appearing in the editor (ext ascii), but the characters were all represented correctly. Normal ASCII passwords typed as usual in both Firefox and TextEditor. There is no special "handling" of extended ASCII in the auto-type code, however this may be an artifact of X11's passing of extended ASCII keystrokes to firefox.

I will note that using xdotool (namely xdotool search --name '<tab name>' | xargs xdotool windowfocus && xdotool type '<ext ascii>') produces the same (and even worse) results as KeePassXC.

[TheBeardOfTruth]

Right, so it's reproducible (and it's interesting that I have zero-delay in text editors and terminals but not in firefox, where as you appear to have the delay universally).

Notably firefox and xorg spike to ~90%+ cpu usage on the cores they live on which feels anomalous

That command line works horribly when running under gnome it definitely produces less than ideal results (additional x11 latency because of gnome or gnome interfering? Note to self: test under 2bwm or i3 or something else that's a bit less intrusive) in either case it doesn't exhibit the incorrect characters issue (merely the dropped characters issue, though a lot more of them which is arguably worse)

sleep 0.5 && xdotool windowactivate --sync $windowid type --delay 75 '<INPUT>' 

This produces very slightly better results for me under gnome.

And after locking up my machine for ~25 minutes and some regex I have this data:

For the input string (repeated 5x, interleaved): €‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ¡ ¢£¤¥¦§¨©ª«¬®¯±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ we find that it really doesn't like some characters.

What we get out is 4€Žóë‘’“”™*¡ ¢£¥¦§©ª«¬®±²³µ¶·¹º»¼½¾¿ÆÐÒ×ØÞßæðò÷øùþ

Naturally everything that appears here (with the exceptions of Žóë™, I will cover these separately) also appeared interleaved 5x (i.e €€€€€).

You may notice two anomalies. the "4", I have no idea where the "4" came from and the "*", again, no idea from where. My best guess is that the bits sent over got shifted by some number of bits one way or another and ended up on 00110100 and 00101010 respectively but honestly I'm not entirely sure how, the bit patterns don't line up so that shouldn't happen unless someone, somewhere, is incorrectly writing their data (10000000 and 10011001 respectively for the euro and trademark symbols)

We find here that firefox doesn't like most accented characters at all and the ones it does seems arbitrary. I'm not sure if it's because of these specific characters or just an artifact of this trial run, but time to process increases exponentially with string length (seconds for a string of 32 characters, over 25 minutes for 605, that's a couple orders of magnitude longer per character, taking well over 2.4 seconds to process a single character)

As for 4€Žóë‘’“”™*¡, here's what it looked like interleaved: 4€€€€Žóë‘‘‘‘‘’’’’’“““““”””””™™*™™¡¡¡¡¡ It could, honestly, mean a lot of things but my guess is that the first bit is the most "unstable" (in that we dropped a trademark for no reason and we dropped 4 of {Ž, ó, ë} respectively (though that begs the question, why were all the euro signs preserved?)

Either way, that's some intense data loss, as an aside xdotool seems to work slightly better with -key KEYSYM entries but that doesn't sound like a tenable solution.

As a slight footnote: you may notice the string I used only contains 121 characters, that's because 5 extended ASCII characters aren't printable/unavailable on my machine for whatever reason {129, 141, 143, 144, 157} (I included non-breaking space for completeness) and one behaved poorly when trying to add it to my string 173 (&shy) "Soft hyphen". Other than that it's a complete set of extended ASCII (not that these exclusions will matter much, best case the output would have 6 more characters but 38/121 vs 44/127 isn't a meaningful difference, ratio-wise since it's still about 70% data loss)

I don't even know if this adds much of value, I wager it doesn't aside from knowing that whatever mechanism lies behind it scales even worse with long passwords... whatever, thanks for taking a look at it though.

[droidmonkey]

Given that copy/paste works perfectly fine I am not keen on blaming FireFox anymore. I think that FireFox is just such a monster of an application that it exacerbates the problem. The real problem is, in my opinion, X11. I doubt this will ever get fixed. I recommend using the KeePassXC Browser Plugin or copy/paste for Ext ASCII passwords.

I also suspect that KeePass2 running under Mono does not use the X11 typing interface at all and does something completely different.

[furai]

So for a normal end-user like me - is there anything I can do about it without installing browser plugins?
I've noticed that upping the delay to around 150ms stops the issue for me.

[TheBeardOfTruth]

I'd reckon the only real options are to either not use extended ascii passwords, to manually copy/paste them (double click pw field), or to use the browser plugins.

No other solution exists as far as I'm aware.

[phoerious]

I can only caution against the use of such passwords anyway. Use ASCII and make the password longer instead.

[furai]

I can only caution against the use of such passwords anyway. Use ASCII and make the password longer instead.

Maybe that's an option. Is there a way to find quickly those entries using extended ASCII? Or will I have to change it successively when I notice it in auto-type?

Would be cool if auto copy&paste was an option but I guess that's what the plugins are for.

[TheBeardOfTruth]

I can only caution against the use of such passwords anyway. Use ASCII and make the password longer instead.

Why? A larger character-space equals a larger searchspace, more bits = better. More relevant, this may be 2019 -- you'd think people wouldn't enforce character limits on passwords (at least not really low ones like 8 or 16 characters)... but alas far too many places restrict the length of your password, even relatively major websites (e.g google)

Longer isn't always an option, and maybe 1300 bits of entropy is excessive, but I'd rather have a password too strong than too weak, no?

[furai]

I can only caution against the use of such passwords anyway. Use ASCII and make the password longer instead.

Why? A larger character-space equals a larger searchspace, more bits = better. More relevant, this may be 2019 -- you'd think people wouldn't enforce character limits on passwords (at least not really low ones like 8 or 16 characters)... but alas far too many places restrict the length of your password, even relatively major websites (e.g google)

Longer isn't always an option, and maybe 1300 bits of entropy is excessive, but I'd rather have a password too strong than too weak, no?

That's exactly why I'm using extended ASCII, I've run into passwords being limited to certain number of characters.
Ever since I've been applying it to every password I generate if it's an acceptable character to use.

[admirabilis]

I have a very similar problem with GNOME, perhaps because of ibus. Accented Portuguese characters take a very long time to be typed by KeePassXC, xdotool is even slower. The solution is immediate by setting the X keyboard manually with eg. setxkbmap jp, but it seems GNOME resets that after some time, so I'm thinking of running this command in a cron or something. There could be side effects... In notes from years ago, running setxkbmap from a script had the exact opposite effect in KeePassXC and xdotool.

[hifi]

Can someone test 2.6.4 and #6247 if this issue is still present? Thanks.

[hifi]

Now that #6247 is merged we have changed the way Auto-Type works on X11. It requires one of your active keyboard layouts to include the characters that are being typed and it shouldn't cause slowdowns anymore. It will also prefer your current active layout to avoid changing the layout group unnecessarily.

This new requirement will permanently break the pattern in OPs post unless you can find enough layouts to satisfy every special character. I consider this an edge case for testing purposes but feedback is welcome.

If no one steps up to test this with current nightlies it will end up in 2.7.0 as described.

[admirabilis]

Your description seems perfectly fine. I haven't stepped up to test it for two reasons: I changed all passwords that have localized chars, which will also avoid any issues in the future with bad websites, and I'm temporarily using Solus, where building packages is such a pain..., I gave up and I'm waiting for some spare time to go back to Debian/Ubuntu land.

[hifi]

Please open a new issue against develop if this continues to be an issue. Closing, thanks.