Rework keyboard management to allow device-side tracking of modifier states

[IDisposable]

This is the beginnings of the rework of the keyboard support.

Goals:

1. Remove client/browser-side tracking of meta key state (Shift, Control, Alt, Super)
2. Add device-side (go code) to track the meta key state and communicate it back to the client (for Virtual Keyboard layout tracking)
3. Remove option to select LED tracking for latching key state (Caps Lock, Num Lock, Scroll Lock, Compose, Kana) since we have to trust the device
4. Ensure the device tracks the LED state at all times
5. Change the handling of keys in WebRTCVideo to use reportKeypress to pass all the keydown / keyup events so it can track the meta state and LED state
6. Allow for multiple-key-down (up to the standard 6 key-rollover of HID keyboards)

[IDisposable]

@adamshiervani looks like this is mostly complete to get the state managed device-side. There was a lot of work done to make the type-checking better because I had trouble working out some issues, so I just hardened up the code to get it to the point where the error was "obvious". Sadly in the end it was some odd interaction between Go's default serialization of uint8/byte values to JSON, which if given an array slice of uint8/byte will emit the slice as a base-64 encoded blob... on the typescript side, that doesn't get recognized as an array, so passing the keys-down wasn't working. In the end, I added a ByteSlice type on the Go side, and forced it to serialize as an array... sigh...

Now that I'm mostly done, might want to chat through if I did this the way we want. One thing I've realized just today is that if the client UI tries to make the new rpc calls to the device and it's not current.. well. Might have to restore the code I elided in commit 0c76634 and do some device-version detection?

[IDisposable]

Rebased and conflicts resolved

[adamshiervani]

Hell yeah, amazing work!!! Let's get this merged now.

[IDisposable]

Whew...

[Caedis]

Trying this out locally and seems like LMeta is not able to be held down. When I press and hold it down, the status bar only shows it pressed for a split second

[IDisposable]

Trying this out locally and seems like LMeta is not able to be held down. When I press and hold it down, the status bar only shows it pressed for a split second

LMeta is pretty hard to manage because it pops up the OS menus when the browser/remote is on Windows and MacOS... mostly the only way I can get things to work with LMeta+something is to use Macros or Virtual Keyboard, but I would like to know more about how it acts on your system. Can you record a video or screen-share?

[Caedis]

Before this pr, i was able to easily use Windows+Shift+S, but now I have to basically press all 3 at once and release quickly

Jetkvm connected to Windows
Browser is on Linux

[IDisposable]

Before this pr, i was able to easily use Windows+Shift+S, but now I have to basically press all 3 at once and release quickly

Jetkvm connected to Windows Browser is on Linux

Trying to get to the bottom of this! Please bear with me...

Do you mean pressing keys on the physical keyboard?

What OS is the browser/client running on? I think you mean you've got

flowchart LR
    PC[Linux] -->|press physical key | LMeta
    LMeta --> Browser[fa:fa-browser Chrome]
    Browser -->|event| JetKVM[client]
    JetKVM -->|RPC| Device
    Device -->|HID| Host[Windows]

Loading

Also, are you keyboard trapped, fullscreen + running the JetKVM on an HTTPS (not HTTP)?

[IDisposable]

Before this pr, i was able to easily use Windows+Shift+S, but now I have to basically press all 3 at once and release quickly

Oh, do you mean that you before could press-and-release LMeta, press-and-release Shift, press S and it would be seen as a Windows+Shift+S on the host?

If that's the case I suspect that's because the old keyboard stack wasn't processing things in order or synchronously (which broke all kinds of input things and basically had to be fixed to be in-order processing.

[Caedis]

No, i could press shift, then meta, then s, Windows would register it, then I would release all 3.

Yes, pressing and holding the physical key would only show as a quick press and release of the LMeta key.

I have 0 issues holding down LMeta in dwm or hyprland to move stuff around or executing other shortcuts that use LMeta.

[Caedis]

Non-fullscreen and absolute mouse so not keyboard trapped. Wish that worked but it doesnt even with an https reverse proxy with an LE cert.

[Caedis]

Client = Arch in Firefox
Host/Jetkvm is attached to Windows.

I have 0 shortcuts set to LMeta+Shift+S

[Caedis]

Preview:

20250828_235857.mp4

[IDisposable]

Great repro, thanks...will look into that ASAP

[alexballas]

One thing I noticed:
Over https://app.jetkvm.com/ Holding LeftShift down and then pressing MetaLeft doesn't even trigger this momentary "MetaLeft" press. Nothing appears.

Over IP (without HTTPS) I do see this momentary press

[alexballas]

      if (!isKeyboardLockActive && hidKey === keys.MetaLeft) {
        // If the left meta key was just pressed and we're not keyboard locked
        // we'll never see the keyup event because the browser is going to lose
        // focus so set a deferred keyup after a short delay
        setTimeout(() => {
          console.debug(`Forcing the left meta key release`);
          handleKeyPress(hidKey, false);
        }, 100);
      }

this part here is the one causing the momentary MetaLeft press

[Caedis]

Nice find

[alexballas]

On Linux when I press Alt key and change the focus of the window without releasing it, the browser will force release it. So I dont know if the following statement is correct.

        // If the left meta key was just pressed and we're not keyboard locked
        // we'll never see the keyup event because the browser is going to lose
        // focus so set a deferred keyup after a short delay

One for @IDisposable to confirm

[Caedis]

Commented out that 2 blocks and deployed and everything works for me
(meta and shift being on at the start was due to my shortcut to record a selection being Meta + Shift + PrintSc)

20250829_083600.mp4

[IDisposable]

On review post folks beginning to test, I think I should have released the Lock here to ensure we're not holding through the entire callback for onKeysDownChange. Thoughts @adamshiervani ?

[IDisposable]

Over https://app.jetkvm.com/ Holding LeftShift down and then pressing MetaLeft doesn't even trigger this momentary "MetaLeft" press. Nothing appears.

As far as I know, the app.jetkvm.com has not been updated yet since this isn't fully released. Thus it is still using the ReportKeys (old stack/Macros) instead of the ReportKeyPress (new stack)

      if (!isKeyboardLockActive && hidKey === keys.MetaLeft) {
        // If the left meta key was just pressed and we're not keyboard locked
        // we'll never see the keyup event because the browser is going to lose
        // focus so set a deferred keyup after a short delay
        setTimeout(() => {
          console.debug(`Forcing the left meta key release`);
          handleKeyPress(hidKey, false);
        }, 100);
      }

this part here is the one causing the momentary MetaLeft press

Correct, not sure how to handle it now, but I certainly can remove that code in favor of a lazy timer. The comment remains true, though, because if someone is not in fullscreen on an HTTPS connection, we can't grab a keyboard lock, and thus the LMeta kicks in for the browser's OS (and thus we would never see the key up event. This code (in a different form existed before) see https://github.com/jetkvm/kvm/pull/725/files#diff-503edceb49a3a14452eb4663ab40040a5ff04a2a89f13e48be02539e2c939f08L441-L450

I will try to find a better way that doesn't force the key-up event unless we've lost focus and some interval passes... but that's going to be fun because it's possible for the key state to get out of sync with the device and host... so have to protect that.

[IDisposable]

On Linux when I press Alt key and change the focus of the window without releasing it, the browser will force release it. So I dont know if the following statement is correct.

        // If the left meta key was just pressed and we're not keyboard locked
        // we'll never see the keyup event because the browser is going to lose
        // focus so set a deferred keyup after a short delay

One for @IDisposable to confirm

Interesting... maybe I should just not special-case the LMeta at all...

[IDisposable]

Over https://app.jetkvm.com/ Holding LeftShift down and then pressing MetaLeft doesn't even trigger this momentary "MetaLeft" press. Nothing appears.

As far as I know, the code on app.jetkvm.com is the old stack still (since 0.5 hasn't actually released).

To get the new stack for now, you will have to be running against the device directly (via IP or your local DNS name)

In fact I had to rework some stuff in this PR to support the opposite case (when app has been upgraded but the device hasn't)... wheeee

[IDisposable]

Commented out that 2 blocks and deployed and everything works for me (meta and shift being on at the start was due to my shortcut to record a selection being Meta + Shift + PrintSc)

Awesome, thanks for confirming... I'm all in favor of just not special casing things :)

What's the "other" block you deleted?

[Caedis]