scrcpy without the need for ADB

[JonasLoeffelholz]

Hello,
my name ist Jonas and I work at 9elements (more precisely [https://9esec.io/], over the last months we developed an application inspired by scrcpy but under the premise not to use ADB for Data transfer. Instead we opted for a solution based on the AOA-Protocol using a standalone Android app that grabs the screen using androids Mediarecorder.
On the Desktop side we use libusb to receive the media data and send Mouse and Keyboard commands to the Phone (AOA includes options to actually act as a HID to the Phone).

I write this because we want to now if there would be interest in the community to either get the features of using AOA into the current scrcpy or maybe even put the code out in the open (maybe a seperate repo) so people could contribute.

Best Regards :)

[rom1v]

Hi,

Thank you for reaching out.

over the last months we developed an application inspired by scrcpy but under the premise not to use ADB for Data transfer.

Interesting :)

For scrcpy, there is a recent PR implementing HID over AOA: #2632. It sends keyboard HID events via libusb. As far as I tested, it works well.

One limitation is that it only works on Linux, which is fine to me, but does your application sending HID events also work on Windows/macOS? If so, how?

After this PR is merged, I'd like to also allow this HID control (keyboard and mouse) without mirroring (so even with adb disabled).

I write this because we want to now if there would be interest in the community to either get the features of using AOA into the current scrcpy or maybe even put the code out in the open (maybe a seperate repo) so people could contribute.

Publishing the app source code would be great, indeed 😉

Cheers

[AlynxZhou]

Instead we opted for a solution based on the AOA-Protocol using a standalone Android app that grabs the screen using androids Mediarecorder.

So I think this is by using AOAv1 and send video stream as USB data packets? I guess we need to create a protocol for this.

On the Desktop side we use libusb to receive the media data and send Mouse and Keyboard commands to the Phone (AOA includes options to actually act as a HID to the Phone).

I've currently send a PR that use AOAv2 for keyboard. For mouse, a problem is that HID mouse has no absolute X or Y position data in event, so if you move your cursor out of scrcpy window from one edge and back from another edge, you lose cursor sync.

Please also note https://libusb-devel.narkive.com/vENuKzdR/getting-the-serial-number-of-the-camera-fails if you are using libusb, I've tested it on Windows, it cannot open my Android phone and read USB serial. How do you make it work on Windows?

I write this because we want to now if there would be interest in the community to either get the features of using AOA into the current scrcpy or maybe even put the code out in the open (maybe a seperate repo) so people could contribute.

We'll be happy if you could make your changes open-source so we can learn how to do this.

[rom1v]

For mouse, a problem is that HID mouse has no absolute X or Y position data in event, so if you move your cursor out of scrcpy window from one edge and back from another edge, you lose cursor sync.

SDL_SetWindowGrab might be helpful.

[AlynxZhou]

For mouse, a problem is that HID mouse has no absolute X or Y position data in event, so if you move your cursor out of scrcpy window from one edge and back from another edge, you lose cursor sync.

SDL_SetWindowGrab might be helpful.

Nope, scrcpy is not a fullscreen game, I want to easily switch between scrcpy and my other works... Maybe some other ways can be found to sync mouse.

[AlynxZhou]

@rom1v For using AOAv1 and USB for video stream and other communicate, I personally like it, but it seems we need to re-implement code around USB instead of ADB, which is a big change.

[rom1v]

@AlynxZhou Yes, mirroring without adb is currently not a goal. It would require the server to be installed as an app, and use the MediaProjection API (with confirmation from the user to capture the screen every time).

[AlynxZhou]

@AlynxZhou Yes, mirroring without adb is currently not a goal. It would require the server to be installed as an app, and use the MediaProjection API (with confirmation from the user to capture the screen every time).

But it should be better experience, my phone warns me twice about "Payment Security" each time when I using online shopping or payments with USB debug on 😂.

[AlynxZhou]

OK, if we ask the Android device to start in AOAv1, it seems that Android device will re-connect with a new VID/PID, I am not sure whether this will fix the libusb windows issue: https://source.android.com/devices/accessories/aoa

[AlynxZhou]

@AlynxZhou Yes, mirroring without adb is currently not a goal. It would require the server to be installed as an app, and use the MediaProjection API (with confirmation from the user to capture the screen every time).

Install an app maybe not so bad because we could then integrate sndcpy.

[JonasLoeffelholz]

So I think this is by using AOAv1 and send video stream as USB data packets? I guess we need to create a protocol for this.

Yes, this is actually the case, right now we are sending just plain data without even wrapping it, wich might be a good idea in the future.

I've currently send a PR that use AOAv2 for keyboard. For mouse, a problem is that HID mouse has no absolute X or Y position data in event, so if you move your cursor out of scrcpy window from one edge and back from another edge, you lose cursor sync.

We Implemented a Function to calculate the absolute cursor position while the cursor is over the Application window wich actually works quite well, but things like Moving up the Lock screen to unlock the Phone are a little difficult sometimes.

Yes, mirroring without adb is currently not a goal. It would require the server to be installed as an app, and use the MediaProjection API (with confirmation from the user to capture the screen every time).

Yes, so we use the MediaProjection api and a Permanently installed App on the device the user has to give permission for screen recording, one thing about that is that Android will censor sensible data like banking information in an banking app.
The user Has to give permission just once until he completely closes the app, but he can disconnect the phone,
whithout having to allow screen recording again to restart the stream while the app ist still ideling.

One limitation is that it only works on Linux, which is fine to me, but does your application sending HID events also work on Windows/macOS? If so, how?

We only implemented it on Linux, as our customer wants it as a feature for their Debian based distribution, its also just a (relatively) stable proof of concept yet.

Publishing the app source code would be great, indeed wink

So our customer asked us to reach out in the community to ask if there would be interest, so I think this will be possible soon :)

[AlynxZhou]

So I think this is by using AOAv1 and send video stream as USB data packets? I guess we need to create a protocol for this.

Yes, this is actually the case, right now we are sending just plain data without even wrapping it, wich might be a good idea in the future.

Plain stream should be fine for video, but scrcpy supports control messages, so I think we may need a protocol (I believe currently we have one, not sure if it's via socket or adb).

I've currently send a PR that use AOAv2 for keyboard. For mouse, a problem is that HID mouse has no absolute X or Y position data in event, so if you move your cursor out of scrcpy window from one edge and back from another edge, you lose cursor sync.

We Implemented a Function to calculate the absolute cursor position while the cursor is over the Application window wich actually works quite well, but things like Moving up the Lock screen to unlock the Phone are a little difficult sometimes.

Nope, it's not about how to calculate the cursor position of the window, HID mouses (like my Zowie EC1B) reports only relative X and Y data, not absolute positions, I am not sure how you send absolute positions via HID event. I know we can set Absolute for Input tag in HID report descriptor, but I never tried because my mouse is relative.

Yes, mirroring without adb is currently not a goal. It would require the server to be installed as an app, and use the MediaProjection API (with confirmation from the user to capture the screen every time).

Yes, so we use the MediaProjection api and a Permanently installed App on the device the user has to give permission for screen recording, one thing about that is that Android will censor sensible data like banking information in an banking app.
The user Has to give permission just once until he completely closes the app, but he can disconnect the phone,
whithout having to allow screen recording again to restart the stream while the app ist still ideling.

I personally perfer this way, but @rom1v is the one who do the finally decision 😄.

One limitation is that it only works on Linux, which is fine to me, but does your application sending HID events also work on Windows/macOS? If so, how?

We only implemented it on Linux, as our customer wants it as a feature for their Debian based distribution, its also just a (relatively) stable proof of concept yet.

So there is some conflict things:

• the ADB way, works with WiFi/USB, input works on Windows via event injection.
• the USB way, works with USB, transfer all data via USB but will fail on Windows.

Publishing the app source code would be great, indeed wink

So our customer asked us to reach out in the community to ask if there would be interest, so I think this will be possible soon :)

Happy to see this.

Transfer all data via USB and a app looks good (and we can also integrate sndcpy into this app), but considering Windows support, maybe we should keep both method, and it need a lot of work to merge those 2 ways into one project...

[AlynxZhou]

gibsson/linux-adk#3

The developer of linux-adk said he has no problem for libusb and Windows, I am not sure where I wrong. Maybe related to my phone? If anyone can test their phone with my PR on windows, it will be good.

[JonasLoeffelholz]

So as promised here is a repository with the majority of the desktop side code:

aoakvm
The Project was Sponsored by secunet
We called it aoakvm for Android Open Accessory Keyboard Video Mouse ;)
I'll be Happy to Help.

Best Regards :)

[fleytman]

@rom1v
Might be helpful. There is a project for ios
https://github.com/libimobiledevice/usbmuxd
https://github.com/libimobiledevice/libusbmuxd