[HELP] Creating libscrcpy (input-only)

[srevinsaju]

Hello

I am currently under the process of creating libscrcpy; (although I have poor knowledge of C)
I am reimplementing scrcpy's input controller #1271

I came across a method

input_manager_process_key

If scrcpy is running; and scrcpy-server is on the device;
Can anyone provide a test method to see if the function is actually getting called.
By test method; I am looking forward to get the function signature and a sample call of the function; Also; if there are any other function to be bootstrapped before calling this method please let me know

@rom1v

[srevinsaju]

I tried calling the action_home function; but unfortunately, nothing happens to the device.

[rom1v]

I am currently under the process of creating libscrcpy

The idea of the library is to provide only the interaction with the device as a library, so that we can write an alternative UI (or other specific apps).

So it must be independant of the SDL UI. In particular, input_manager_process_key takes a SDL_KeyboardEvent * as parameter, so it cannot be in libscrcpy, it's the SDL UI part.

To create a proper lib, the protocol between the lib and the UI must be defined so that it is independant of any UI.

I tried calling the action_home function; but unfortunately, nothing happens to the device.

How do you call it? From where?

[srevinsaju]

@rom1v

Yes; I understand; I do not mind keeping SDL as a dependency. The SDL GUI is very fast; What I can do is to create custom window around it. The only problem now I face is the lag in adb shell tap events; because of what you have said in #231; scrcpy's controller.c and input manager is so fast; and I want to implement the same logic without rewriting them in Python (because I do not understand them)

So it must be independant of the SDL UI. In particular, input_manager_process_key takes a SDL_KeyboardEvent * as parameter, so it cannot be in libscrcpy, it's the SDL UI part.

Interesting;
I am calling action_home, but it requires an argument called controller for which I am not able to understand what is to be passed as a controller param. A possible problem with what I am doing might be because I do not run controller_init and other controller_* functions which bootstrap scrcpy's procress

edit: re reading what you have said; It seems like I cannot pass a keycode.
Is it possible to use scrcpy_server to execute the keycode, say 43?

To create a proper lib, the protocol between the lib and the UI must be defined so that it is independant of any UI.

What I am trying to achieve is to keep the SDL UI as is, and recreate the adb lib used by scrcpy; that is the controller.c and input_manager.c; I do not want to edit the SDL and the UI yet; because it works amazingly fast; and I do not want to disturb its order; Basically creating a lib which does the adb input taps faster than the original adb as mentioned in #231

How do you call it? From where?

As you might know; the necessity of the fast lib is because of the development of guiscrcpy which is in Python. Python, which is built directly over C enables me to directly use a shared library within a python script

So, what I have done:

• Run meson and ninja
• Before installing ninja,

cd x
cd app/a172ced@@scrcpy@exe

Now convert all the src_*.c.o files to libsrc_*.c.*.so files so that Python can use it as a shared library.

Recompile inputmanager.o to include all the necessary deps

gcc -L$(realpath .) -shared -o libinput_manager.so src_input_manager.c.o -lSDL -lSDL2 -lavcodec -lavformat -lsrc_cli.c.o -lsrc_scrcpy.c.o -lsrc_recorder.c.o -lsrc_file_handler.c.o -lsrc_opengl.c.o -lsrc_device.c.o -lsrc_stream.c.o -lsrc_util_net.c.o -lsrc_receiver.c.o -lsrc_sys_unix_command.c.o -lsrc_screen.c.o -lsrc_event_converter.c.o -lsrc_server.c.o -lsrc_util_str_util.c.o -lsrc_controller.c.o -lsrc_command.c.o -lsrc_fps_counter.c.o -lsrc_tiny_xpm.c.o -lsrc_input_manager.c.o -lsrc_main.c.o -lsrc_control_msg.c.o -lsrc_video_buffer.c.o -lsrc_device_msg.c.o -lsrc_decoder.c.o

Then open my python interpreter as a part of test

LD_LIBRARY_PATH=$(realpath .) python

Python 3.8.2 (default, Apr  8 2020, 14:31:25) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> scrcpy_input_manager = CDLL('./libinput_manager.so')
>>> scrcpy_input_manager.action_home()

Segmentation Fault


Python 3.8.2 (default, Apr  8 2020, 14:31:25) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> scrcpy_input_manager = CDLL('./libinput_manager.so')
>>> scrcpy_input_manager.action_home(1)
0

In the seconf case; it gives 0 which might be the void in C; This is my only progress; executing the above function should enable me to click the home button. But the signature of the action_home tells me that I have to pass controller *controller as a param. And that is where I am stuck

Any help from your side is appreciated. 😅

---

What all I have tried and found out

• Running adb reverse --list gives nothing while running scrcpy; creating a suspicion if I can ever write to the socket
• Running adb reverse --list at the exact moment of execution of scrcpy gives

host-16 localabstract:scrcpy tcp:27183

This started giving me the suspicion if I can write a command to localabstract:scrcpy since the port looks too abstract and is not a number; (I am not an expert Linux user; I am beginning to see what a socket and tcpip is)

• Even if I could write to the socket; scrcpy_server might or might not accept it as it follows a custom protocol as mentioned in Why your adb shell cmd execute so fast? #231. So I did not know how I can pass an information to scrcpy-server while its running

[srevinsaju]

I forgot to mention; I also converted all the static function definitions to functions without static (idk what to call non static functions)

[rom1v]

Oh, so what you need is a "remote control API" (#399).

The same as #1159 to receive the video, but in the other direction to inject input events.

With this, you could just communicate using some protocol over a socket.

[srevinsaju]

Oh, so what you need is a "remote control API"

Yes

The same as #1159 to receive the video, but in the other direction to inject input events.

Exactly

With this, you could just communicate using some protocol over a socket.

This is the part I am not understanding; What is protocol; and which socket?

In this case. I got this output
Few of my questions, if you have the time to answer:

• Regarding the output from adb reverse --list

host-16 localabstract:scrcpy tcp:27183

Does tcp:27183 change across sessions? In my case, it remained static; it has never changed. In such a case, can I write to my port 27183 and execute a back action or home action?

• If the above holds true, in which format can I write to port 27183 so that scrcpy-server understands it and be like, "Okay, I am going to press the back button"; like that

• Can I write to the socket using nc?

nc -l localhost 27183
55
555
keyevent

or something like that?

I apologize if I am disturbing you. Consider this as my poor curious questions. 😃
Thanks for the quick links; I will try checking them out

[leng-yue]

I made a cross-platform full-function client of the scrcpy server v1.18. py-scrcpy-client
This client can both receive live video feed and send control messages.