Sending keyboard events as physical keyboard instead of virtual keyboard

[AlynxZhou]

Current it send keyboard events as virtual keyboard (like input method keyboard), however google pinyin input supports physical keyboard with OTG to input, physical can use numbers to choose words which is really convenient and saves half of screen, if it can send as physical keyboard it will be better (because actually we are using a physical keyboard on computer).

[rom1v]

See #87 (comment) about OTG.

[lasek101]

Hello
First thanks for Your work.
Now to my problem.
I use polish programmer phisical keyboard. It is normal qwerty keyboard with polish national characters obtaining by pressing right alt with specific letter.
I have compiled, on linux box, client and server fom dev and rawalpha branch but can't get any polish diacritical letter. Pressing ctrl+k does nothing.
What I'm doing wrong?

[rom1v]

@lasek101 Unfortunately, I have no solution to generate such keys on the device.
See #193 and #194.

[AlynxZhou]

@rom1v what about letting computer dealing with input method and just pass input string to phone?

[npes87184]

Hi, @AlynxZhou

Does the problem become to be able to input (Chinese) words from computer?
If the answer is yes, it may be a simple workaround to cover the problem which is some character cannot be prefectly handled. However, there are still some problems to deal.
For example, pass input string to where? clipboard or pass directly to the edittext?

If we pass the string to clipboard, there will be inconvenience to use. We need to paste again and again in phone. On the other hand, if we want to directly pass to input field. As I know, there are many restricts. For instance, how about webview?

In my opinion, unless we can cover these problems. It is not an elegant solution yet.

Thanks.

[lasek101]

Maybe there is a hope.
If I connect physical keyboard in OTG mode I can choose layout in Android (settings-languages and input methods - physical keyboard - then I choose keyboard I've connected and choose correct layout). After that I can get polish diacritical letters by pressing right alt and certain letter.
Mayby there is a way to tell android that keyboard used by scrcpy is like OTG one.

[rom1v]

@AlynxZhou

what about letting computer dealing with input method and just pass input string to phone?

This is what the client sends to the server (except for letters and space since v1.4, see #87).

The problem is that the device does not support injecting characters other than ASCII. See handle text inputs.

@lasek101

If I connect physical keyboard in OTG mode

See #87 (comment).

After that I can get polish diacritical letters by pressing right alt and certain letter.

When you press AltGr+letter on your OTG keyboard, or on your computer?

[lasek101]

I press AltGr+letter on phisical keyboard and get correct diacritical letter on android.
see https://youtu.be/QEmYMImkNnQ

[rom1v]

@lasek101 OK, so this comment totally applies: #87 (comment)

This works because an OTG keyboard sends HID events over USB directly, which is not possible from a computer (except using HID over AOA).

[lasek101]

So, this is definitely not possible in scrcpy?

[lasek101]

Alternativly mayby it is possible to use phisical keyboard like virtual in a way that: when I long press letter 'a' I get option to choose what character I want to send i.e. 'ą'.
Now when I long press letter I get many reps of that letter.

[AlynxZhou]

@rom1v I think HID events is the best way however I know little about Android development and USB HID, if I know I can help...

[rom1v]

@lasek101

Alternativly mayby it is possible to use phisical keyboard like virtual in a way that: when I long press letter 'a' I get option to choose what character I want to send i.e. 'ą'.

The problem is not how to generate the key, but how to inject it without root access into the system. See Handle accented characters.

@AlynxZhou

I think HID events is the best way

It would be great, but it may be a bit complicated.

Since you don't necessarily use a USB keyboard on your computer, you need to retrieve the key events via SDL, and "convert" them into raw USB HID events (is the mapping easy?).

And you need to enable AOA (which does not work on all devices, and I don't know how to communicate over AOA from Windows). See discussion about audio forwarding: #14.

if I know I can help...

Generating HID events from SDL key events would be great 😉

[AlynxZhou]

I understand what we need, but I am questioning about whether we have a proper way to simulate a "virtual device" on Android without root... It seems AOA is not proper way for this?
Plus did you see KDE Connect remote keyboard? This seems a way that send text from PC as an input method, I am trying this, but unfortunately GSConnect cannot work after GNOME 3.30 upgrade...

[rom1v]

@AlynxZhou Yes, this is another alternative. It would require to install an APK though, so this may be "optional" (install the additional keyboard if you want to support "special characters").
See #37 (comment).

[AlynxZhou]

Installed KDE Plasma and test failed: KDE Connect cannot forward Chinese character too...

[amosbird]

Since you don't necessarily use a USB keyboard on your computer, you need to retrieve the key events via SDL, and "convert" them into raw USB HID events (is the mapping easy?).

Is there a PoC for this?

[rom1v]

@amosbird No.

[amosbird]

Ok. I asked that after coming across this https://github.com/rom1v/aoa-hid-bug . Looking forward for a PoC :D

[amosbird]

Finished a dirty PoC. It works as expected. https://github.com/amosbird/scrcpy/blob/master/app/src/usb_hid_keys.h#L482

[rom1v]

Cool, thank you for the PoC 👍

I had to add the thread dependency to make it work:

diff --git a/app/meson.build b/app/meson.build
index 319548a..8b2456c 100644
--- a/app/meson.build
+++ b/app/meson.build
@@ -29,6 +29,7 @@ if not get_option('crossbuild_windows')
         dependency('libavutil'),
         dependency('sdl2'),
         dependency('libusb-1.0'),
+        dependency('threads'),
     ]
 
 else

(and of course change my pid:vid)

Currently, it does not handle special chars (e.g. if I press the key è, it sends 7), and inputs are interleaved with normal text input events (so it's hard to tell which method sends what).

Could you link some HID doc/spec you used to generate these tables, please? https://github.com/amosbird/scrcpy/blob/master/app/src/usb_hid_keys.h#L27

Do you know how to use USB on Windows with libusb (and make AOA work on Windows, too)?

[amosbird]

Currently, it does not handle special chars (e.g. if I press the key è, it sends 7), and inputs are interleaved with normal text input events (so it's hard to tell which method sends what).

yeah, that commit is broken. Now it works fine.

Could you link some HID doc/spec you used to generate these tables, please?

https://gist.github.com/MightyPork/6da26e382a7ad91b5496ee55fdc73db2

Do you know how to use USB on Windows with libusb (and make AOA work on Windows, too)?

Sorry I have no idea.

[amosbird]

@rom1v hmm, the media control is very unreliable via AKEYCODE_MEDIA_PLAY_PAUSE. How can I correctly fake the control signals generated by my bluetooth earphone, which works in all music applications?

[rom1v]

https://gist.github.com/MightyPork/6da26e382a7ad91b5496ee55fdc73db2

Thanks. The AOSP doc is there: https://source.android.com/devices/input/keyboard-devices

yeah, that commit is broken. Now it works fine.

Cool, text input are not interleaved anymore. However, I still can't press è (on the 7 key on AZERTY keyboard).

[amosbird]

I still can't press è

hmm, I'm not sure where that issue resides. Can you input è on a real external usb keyboard via otg?

[amosbird]

I've iterated all the combinations of the 4 bytes HID data. None of them play/pause the music for me....

[amosbird]

finally achieved media control. The HID interface is really dark.

[AlynxZhou]

Another issue is HID mode depends on USB, so it also needs to check whether wireless is used...

[rom1v]

Then what should be the default option?

Good question :)

[AlynxZhou]

Then what should be the default option?

Good question :)

Personally I prefer HID.

And some related questions: If user choose HID explicitly, and HID set up failed, scrcpy should exit, that's OK, but if HID is default, user adds no options, and it failed, now should scrcpy quit, or fallback to inject automatically?

[AlynxZhou]

Any ideas on this?

Currently I am trying to make a --input-mode argument, if hid, use HID, and quit if failed, if inject, use the old way, and if not specified, first try HID and fallback to old way. That's both explicit and user friendly.

Your advice is very important and I hope we can have an agreement on this soon so a PR can be made.

[rom1v]

Yes, you could do something like:

--input-mode=auto    # the default if not specified, which attempts to use HID then fallback
--input-mode=hid     # force HID
--input-mode=inject  # force injection (or a better name)

[AlynxZhou]

Yes, you could do something like:

--input-mode=auto    # the default if not specified, which attempts to use HID then fallback
--input-mode=hid     # force HID
--input-mode=inject  # force injection (or a better name)

I've already done this, and what's your opinion about the third choice's name? I'm not so familiar to it.

[AlynxZhou]

Yes, you could do something like:

--input-mode=auto    # the default if not specified, which attempts to use HID then fallback
--input-mode=hid     # force HID
--input-mode=inject  # force injection (or a better name)

I've already done this, and what's your opinion about the third choice's name? I'm not so familiar to it.

I chose inject because the code just call it INJECT_KEYCODE:

https://github.com/Genymobile/scrcpy/blob/master/app/src/input_manager.c#L90
https://github.com/Genymobile/scrcpy/blob/master/app/src/input_manager.c#L575

[AlynxZhou]

Yes, you could do something like:

--input-mode=auto    # the default if not specified, which attempts to use HID then fallback
--input-mode=hid     # force HID
--input-mode=inject  # force injection (or a better name)

I've already done this, and what's your opinion about the third choice's name? I'm not so familiar to it.

Check AlynxZhou@c2f337b for the implementation

[AlynxZhou]

In order to make scrcpy() function cleaner: AlynxZhou@5140be9

(sorry, wrong url)

HID keyboard is now more like a module, and makes the code clear.

[AlynxZhou]

I currently feels OK for hid_keyboard itself, now I have an idea of using HID for mouse, too. But currently touch works just fine, except it seems that scroll cannot trigger drag to refresh.

[AlynxZhou]

Another question - seems controller which send key events and mouse events to android is running on a controller thread, do we also need a thread for hid keyboard?

[AlynxZhou]

Another question - seems controller which send key events and mouse events to android is running on a controller thread, do we also need a thread for hid keyboard?

I'd like to try this, however, it seems a lot of refactor is needed.

[rom1v]

do we also need a thread for hid keyboard?

Yes, I/O to send the HID events must not be done from the UI thread 👍

[AlynxZhou]

do we also need a thread for hid keyboard?

Yes, I/O to send the HID events must not be done from the UI thread +1

Working on this, I'll create another thread for AoA, like controller.

[AlynxZhou]

do we also need a thread for hid keyboard?

Yes, I/O to send the HID events must not be done from the UI thread +1

Done via AlynxZhou@e4ca327

I decide not to update DEVELOP.md myself because if I want to add link to code snippets, I first need to merge my change to upstream 😄 so this will be done after PR.

[AlynxZhou]

Implemented HID mouse today, however it works awful (HID mouse events only contains relative x and y change, so if you move cursor outside scrcpy's window, you lose cursor sync, and also an awful delay)(delay maybe is problem of my code, but still too hard to sync cursors).

I've plugged my mouse into my Android phone, the only useful things is that I can use side buttons for forward and backward in chrome and select text in text entries. Because scrcpy's existing mouse experience is good enough so I won't submit my HID mouse code.

I'll do some cleaning and send PR of HID keyboard soon.

[AlynxZhou]

I wrote an article about the detailed logic of this PR in my blog, English version can be visit on https://sh.alynx.one/posts/Simulate-Physical-Keyboard-in-Scrcpy-via-USB-HID-over-AOAv2/#EnglishVersion.

[rom1v]

Implemented HID mouse today, however it works awful (HID mouse events only contains relative x and y change, so if you move cursor outside scrcpy's window, you lose cursor sync

Isn't it possible to send absolute positions, by changing this line in the mouse report descriptor (hid1_11.pdf, Appendix E, §E.10 Report Descriptor (Mouse)):

Input (Data, Variable, Relative),  ; 2 position bytes (X & Y)  81 06

to

Input (Data, Variable, Absolute),  ; 2 position bytes (X & Y)  81 02

Btw, in absolute mode, one byte might not be sufficient, so maybe 82 02 instead of 81 02.

(cf Input section in the table §6.2.2.4 Main items)

[AlynxZhou]

Implemented HID mouse today, however it works awful (HID mouse events only contains relative x and y change, so if you move cursor outside scrcpy's window, you lose cursor sync

Isn't it possible to send absolute positions, by changing this line in the mouse report descriptor (hid1_11.pdf, Appendix E, §E.10 Report Descriptor (Mouse)):

Input (Data, Variable, Relative),  ; 2 position bytes (X & Y)  81 06

to

Input (Data, Variable, Absolute),  ; 2 position bytes (X & Y)  81 02

Yes, this should work, but I never try it, because most mice uses relative position, and another problem is what's the unit of absolute position? Pixels? Then different devices have different resolutions.

Btw, in absolute mode, one byte might not be sufficient, so maybe 82 02 instead of 81 02.

(cf Input section in the table §6.2.2.4 Main items)

No problem, actually my own mouse uses 2 bytes for relative position. 1 byte is the minimal requirement.

[rom1v]

Closed by #2632.

[krishtoautomate]

@rom1v can we use OTG mode on iphone as well?