On-device OpenGL video filters

[rom1v]

This PR implements on-device OpenGL video filters and fixes all the following issues:

• Android 14 beta lock orientation doesn't work as expected when the device goes into landscape mode. #4011: --lock-video-orientation broken on Android >= 14
• Android 14 Beta 5 crop not working (Pixel 6 Pro) #4162: --crop broken on Android >= 14
• When casting Oculus Quest Pro to Windows, the screen content is rotated approximately 30 degrees. #4135, Support for Meta Quest 3 with angled source #4345, Add --scale, --position-x-offset, --position-y-offset and --rotation-offset #4658: apply a custom capture rotation, useful to mirror a Meta Quest 3
• Virtual Display Rotation Listener Problem #5428: support virtual display rotation
• Feature request: Allow to rotate the camera. #4426: rotate the camera capture for V4L2

Architecture

It first adds an architecture and tools to execute OpenGL filters:

• an OpenGL "runner" (OpenGLRunner), which will run a filter to be rendered to a Surface from an OpenGL-dedicated thread
• a simple OpenGL filter API (OpenGLFilter)
• a single "generic" OpenGL filter implementation to apply any 2D affine transformation (AffineOpenGLFilter)
• a tool to build affine transforms matrices from "semantic" transformations (rotate, scale…) (AffineMatrix)

Crop

To fix #4162, the --crop option is reimplemented using filters on all devices. For screen mirroring, the command is the same, the change is transparent for users.

But now, the specified crop also works for camera and virtual display mirroring (it worked only for screen mirroring before this PR).

Note: for sampling the input camera texture, I had to ignore the SurfaceTexture transform matrix (which is incorrect, it contains an additional 90° rotation which must not be used to sample the texture), and replace it by a vflip. See commit Apply filters to camera capture (and CameraCapture).

Orientation

Before the change, --lock-video-orientation performed 2 actions:

• it rotated the video capture
• it locked the orientation when the physical device rotated

# Now deprecated/removed
scrcpy --lock-video-orientation      # initial (current) orientation
scrcpy --lock-video-orientation=0    # natural orientation
scrcpy --lock-video-orientation=90   # 90° clockwise
scrcpy --lock-video-orientation=180  # 180°
scrcpy --lock-video-orientation=270  # 270° clockwise

This option was broken for devices with Android >= 14.

To fix #4011, now that we can apply our own filters, --lock-video-orientation is deprecated in favor of a more general option --capture-orientation, which supports all possible orientations (0, 90, 180, 270, flip0, flip90, flip180, flip270).

To capture the video with a specific orientation:

scrcpy --capture-orientation=0
scrcpy --capture-orientation=90       # 90° clockwise
scrcpy --capture-orientation=180      # 180°
scrcpy --capture-orientation=270      # 270° clockwise
scrcpy --capture-orientation=flip0    # hflip
scrcpy --capture-orientation=flip90   # hflip + 90° clockwise
scrcpy --capture-orientation=flip180  # hflip + 180°
scrcpy --capture-orientation=flip270  # hflip + 270° clockwise

This makes it consistent with client-side orientation (`--orientation).

The capture orientation can be locked by using a @ prefix, so that a physical device
rotation does not change the captured video orientation:

scrcpy --capture-orientation=@         # locked to the initial orientation
scrcpy --capture-orientation=@0        # locked to 0°
scrcpy --capture-orientation=@90       # locked to 90° clockwise
scrcpy --capture-orientation=@180      # locked to 180°
scrcpy --capture-orientation=@270      # locked to 270° clockwise
scrcpy --capture-orientation=@flip0    # locked to hflip
scrcpy --capture-orientation=@flip90   # locked to hflip + 90° clockwise
scrcpy --capture-orientation=@flip180  # locked to hflip + 180°
scrcpy --capture-orientation=@flip270  # locked to hflip + 270° clockwise

The capture orientation now also works for camera and virtual display mirroring (--lock-video-orientation only worked for screen mirroring), so it also works #4426.

Virtual display

#5370 implemented a virtual display feature, but the capture could not be rotated to follow the app rotation (#5428).

OpenGL filters are used to support virtual display rotation.

Rotation by a custom angle

A new option --angle rotates the video content by a custom angle in degrees (counter-clockwise).

The center of rotation is the center of the visible area (after cropping).

This allows to capture the Meta Quest 3 with the correct angle (#4135, #4345, #4658).

EDIT: I just realized that I forgot to implement it for camera and virtual display. I'll do it later.

---

Testing

There are many combinations to tests:

• devices & Android versions
• screen mirroring VS virtual display VS camera
• crop
• capture orientation (17 possible values, cf above)
• angle

Your help is welcome to report any problem.

Event positions

All mouse/touch events are mapped back to the device position, using the same (inverse) affine transform.

Scrcpy must work as expected with any combination: whatever the transform, clicking on the screen must click at the correct location on the device (you can enable "Pointer location" in developer options).

For example, if you enable "Pointer location" in developer options, then run:

scrcpy --crop=800:500:300:300 --angle=70 --capture-orientation=90 -m700

the click must be performed just under the cursor

Consistency

Some specific values of --capture-orientation directly match the old --lock-video-orientation feature:

 - --lock-video-orientation      ->  --capture-orientation=@
 - --lock-video-orientation=0    ->  --capture-orientation=@0
 - --lock-video-orientation=90   ->  --capture-orientation=@90
 - --lock-video-orientation=180  ->  --capture-orientation=@180
 - --lock-video-orientation=270  ->  --capture-orientation=@270

Running scrcpy v2.7 with the same --crop and a matching --lock-orientation-value must give the same result.

Also, the behavior between screen mirroring and virtual display must be consistent.

Special thanks to @yume-chan for unblocking me in #5444. 😉

---

old

Here is a binary of the current version:

• scrcpy-win64-glfilter.67.zip SHA-256: 88c0cdd94152d1dcf06cc76e3e589db9c5ce04bee9d4f16fac21e5ead331643

Thank you for your feedback.

[rom1v]

I've made some progress this week-end:

• I reimplemented --crop for screen capture, which now works on all devices, including Android >= 14
• I reimplemented --lock-video-orientation for screen capture, which now works on all devices, including Android >= 14
• I implemented virtual display rotation
• All the events are correctly mapped according to the affine transformations

Please review/test! In particular, make sure that everything works with any crop, locked video orientation, physical device rotation, resize… on devices with and without Android 14. Check that:

• the video content must matches the configuration
• all your clicks are correctly mapped

In a virtual display (scrcpy --new-display), an app which automatically rotates (or manually, with Alt+r) rotates the display.

TODO

I wanted to add support for --crop for camera capture and virtual display capture (because why not!?), but it fact there is a semantic issue: for these sources, -m/--max-size selects the source size, it does not resize a predefined source like for screen capture. So when we --crop and -m, it will not behave the same way (it will select another source, then crop). So I think I will not implement --crop for camera and virtual display (it's probably useless in practice anyway).
Or maybe we can just accept to crop the size selected by -m, it's smaller than the source anyway.

I will:

• implement --lock-video-orientation (and --crop?) for camera and virtual displays
• add --angle to specify a custom angle (e.g. 22°, for Meta Quest 3 mirroring), to be applied after cropping, from the center of the resulting region (I suggest --angle to avoid confusion with --rotation/--orientation which rotate by 90° and change the mirrored video size, while --angle will rotate the content by a custom angle inside the area.

[rom1v]

I submitted a new version where everything works (hopefully). I edited the original post and provided a binary for Windows.

Feedback welcome (please report any issue).

[Helaer]

I did a test on Xiaomi's Android 14 model, and after locking the direction, the screen was still incomplete when the phone app was rotated.

[rom1v]

Reproducible 100%? Could you retry after reverting 239553f please?

EDIT: ok, I reproduce, there was a bug in this commit (it resized to videoSize instead of inputSize).

[rom1v]

I updated the PR:

• I fixed the lock orientation issue (On-device OpenGL video filters #5455 (comment))
• I applied the given angle also for camera and virtual display
• I changed the way I workaround the incorrect transform matrix given by Android getTransformMatrix() (I replace it by an hardcoded vflip as a transform matrix, instead of using identity and applying a vflip from the filter)
• I fixed some matrices multiplication order

I published a new binary (glfilter.55) in the original post.

[Helaer]

I tested post.glfilter.55 on my Xiaomi Android 14 device and it worked fine, showing the screen correctly when rotated after the app.

[rom1v]

For consistency with orientation parameters, I changed --angle to interpret the value as clockwise (instead of counter-clockwise) (glfilter.56).

[4nric]

The rotation works on Android 9 although it looked glitchy (window rotates before the "content" does)

glfilter.55_A9.mp4

I have also tested S10+ running Android 14 custom rom, OneUI 6.1 port. Both main display and vd is broken:

glfilter.55_A14.mp4

glfilter.55_A14_vd.mp4

[rom1v]

@4nric Thank you for your tests!

Could you please re-test with Do not recreate display on every rotation reverted? (branch glfilter.56.revert)

I built a binary:

• scrcpy-win64-glfilter.56.revert.zip SHA-256: 3c98ca92d1fca0ae96d05d16ab34f7dfb180ef08686a134600f0d9e4d7e3777

The problem with the virtual display is different: the display listener is broken between Android 14_r1 and Android 14_r29. see e26bdb0. For normal screen capture, I use a fallback by listening to device rotation changes, but for vd I have no solution.

[rom1v]

For normal screen capture, I use a fallback by listening to device rotation changes, but for vd I have no solution.

Oh, actually, we could also use a fallback, but the rotation watcher may not be registered immediately, just after a few milliseconds:

diff --git server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
index 412eb8501..1a87a35ca 100644
--- server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
+++ server/src/main/java/com/genymobile/scrcpy/video/NewDisplayCapture.java
@@ -20,6 +20,8 @@ import android.hardware.display.VirtualDisplay;
 import android.os.Build;
 import android.os.Handler;
 import android.os.HandlerThread;
+import android.os.SystemClock;
+import android.view.IRotationWatcher;
 import android.view.Surface;
 
 import java.io.IOException;
@@ -191,6 +193,15 @@ public class NewDisplayCapture extends DisplayCapture {
                 }
             }, handler);
 
+            SystemClock.sleep(100); // hack: if we register immediately, the display id is not known by the system
+            IRotationWatcher rotationWatcher = new IRotationWatcher.Stub() {
+                @Override
+                public void onRotationChanged(int rotation) {
+                    Ln.i("NewDisplayCapture: onRotationChanged(" + rotation + ")");
+                    invalidate();
+                }
+            };
+            ServiceManager.getWindowManager().registerRotationWatcher(rotationWatcher, virtualDisplayId);
         } catch (Exception e) {
             Ln.e("Could not create display", e);
             throw new AssertionError("Could not create display");

[4nric]

@rom1v glad to test things 😊with scrcpy-win64-glfilter.56.revert.zip, rotation now works as expected on A14 without the "glitch". on A9 though, the window still rotates before the "content" does

[rom1v]

on A9 though, the window still rotates before the "content" does

How does it behave with scrcpy 2.7?

[anotheruserofgithub]

Very nice work! :)

Do you have a measure of the performance impact of the OpenGL runner, memory/GPU-wise and especially on mirroring latency? Is it negligible compared to v2.7?

[rom1v]

Do you have a measure of the performance impact of the OpenGL runner, memory/GPU-wise and especially on mirroring latency? Is it negligible compared to v2.7?

I did not measure. In practice, I haven't noticed an additional latency. I'm pretty sure the internal implementation of --lock-video-orientation and --crop with SurfaceControl in previous versions Android were also using OpenGL.

(And if no transform is needed, OpenGL is not used, the capture is directly encoded as before.)

[rom1v]

I changed the behavior of --max-size for virtual displays: #5506.