🐛 Frame orientation is in landscape-right when phone is held in portrait mode

[Marius-Adam]

What's happening?

I want to scan text in portrait mode but the camera does not pick up anything only if the text is sideways or the phone is in landscape mode.

When logging out the frame.orientation it comes back as landscape-right when the phone is held in portrait

RPReplay_Final1720178652.1.mp4

Reproduceable Code

import React from "react";
import { StyleSheet } from "react-native";
import { Camera, useCameraDevice, useFrameProcessor } from "react-native-vision-camera";
import { useTextRecognition } from "react-native-vision-camera-text-recognition";

function App() {
  const device = useCameraDevice("back");
  const options = { language: "latin"};
  const { scanText } = useTextRecognition(options);
  const frameProcessor = useFrameProcessor((frame) => {
    "worklet";
    console.log(frame.orientation)
    const data = scanText(frame);
   
  }, []);
  return (
    <>
      {!!device && (
        <Camera
          style={StyleSheet.absoluteFill}
          device={device}
          isActive
          frameProcessor={frameProcessor}
        />
      )}
    </>
  );
}
export default App;

Relevant log output

Running "main" with {"rootTag":1,"initialProps":{"concurrentRoot":false}}
Loading react-native-worklets-core...
Worklets loaded successfully
VisionCameraProxy: Creating Worklet Context...
VisionCameraProxy: Worklet Context Created!
🟢 Creating JS object for module 'ExpoKeepAwake'
Looking up Frame Processor Plugin "scanText"...
Frame Processor Plugin "scanText" found! Initializing...
Initialized TensorFlow Lite runtime.
INFO: Initialized TensorFlow Lite runtime.
13:27:21.231: [info] 📸 VisionCamera.didSetProps(_:): Updating 22 props: [onInitialized, cameraId, position, enableBufferCompression, onOutputOrientationChanged, onStarted, preview, top, onCodeScanned, right, isActive, isMirrored, onViewReady, onError, onStopped, onPreviewOrientationChanged, onPreviewStopped, enableFrameProcessor, onPreviewStarted, left, bottom, onShutter]
13:27:21.236: [info] 📸 VisionCamera.configurePreviewOrientation(_:): Updating Preview rotation: landscapeLeft...
13:27:21.237: [info] 📸 VisionCamera.configureOutputOrientation(_:): Updating Outputs rotation: landscapeLeft...
13:27:21.237: [info] 📸 VisionCamera.configure(_:): configure { ... }: Waiting for lock...
13:27:21.246: [info] 📸 VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: true, outputsChanged: true, videoStabilizationChanged: true, orientationChanged: true, formatChanged: true, sidePropsChanged: true, torchChanged: true, zoomChanged: true, exposureChanged: true, audioSessionChanged: true, locationChanged: true)
13:27:21.246: [info] 📸 VisionCamera.configureDevice(configuration:): Configuring Input Device...
13:27:21.246: [info] 📸 VisionCamera.configureDevice(configuration:): Configuring Camera com.apple.avfoundation.avcapturedevice.built-in_video:0...
13:27:21.261: [debug] 📸 VisionCamera.sensorOrientation: Sensor Orientation changed from landscapeLeft -> portrait
13:27:21.261: [info] 📸 VisionCamera.configurePreviewOrientation(_:): Updating Preview rotation: portrait...
13:27:21.261: [info] 📸 VisionCamera.configureOutputOrientation(_:): Updating Outputs rotation: portrait...
13:27:21.261: [info] 📸 VisionCamera.configureDevice(configuration:): Successfully configured Input Device!
13:27:21.261: [info] 📸 VisionCamera.configureOutputs(configuration:): Configuring Outputs...
13:27:21.261: [info] 📸 VisionCamera.configureOutputs(configuration:): Adding Video Data output...
13:27:21.263: [info] 📸 VisionCamera.configurePreviewOrientation(_:): Updating Preview rotation: portrait...
13:27:21.263: [info] 📸 VisionCamera.configureOutputOrientation(_:): Updating Outputs rotation: portrait...
13:27:21.263: [info] 📸 VisionCamera.configureOutputs(configuration:): Successfully configured all outputs!
13:27:21.265: [info] 📸 VisionCamera.setTargetOutputOrientation(_:): Setting target output orientation from device to device...
13:27:21.266: [info] 📸 VisionCamera.getPixelFormat(for:): Available Pixel Formats: ["420v", "420f", "BGRA", "&8v0", "&8f0", "&BGA"], finding best match... (pixelFormat="yuv", enableHdr={false}, enableBufferCompression={false})
13:27:21.266: [info] 📸 VisionCamera.getPixelFormat(for:): Using PixelFormat: 420f...
13:27:21.606: [info] 📸 VisionCamera.init(frame:session:): Preview Layer started previewing.
13:27:21.609: [info] 📸 VisionCamera.configure(_:): Beginning AudioSession configuration...
13:27:21.609: [info] 📸 VisionCamera.configureAudioSession(configuration:): Configuring Audio Session...
13:27:21.609: [info] 📸 VisionCamera.configure(_:): Beginning Location Output configuration...
13:27:21.611: [info] 📸 VisionCamera.configure(_:): Committed AudioSession configuration!
13:27:21.624: [info] 📸 VisionCamera.configure(_:): Finished Location Output configuration!

Camera Device

{
  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:0",
  "maxZoom": 120,
  "formats": [],
  "position": "back",
  "hardwareLevel": "full",
  "hasTorch": true,
  "minZoom": 1,
  "name": "Back Camera",
  "hasFlash": true,
  "minExposure": -8,
  "sensorOrientation": "portrait",
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "neutralZoom": 1,
  "supportsRawCapture": false,
  "supportsFocus": true,
  "isMultiCam": false,
  "supportsLowLightBoost": false,
  "maxExposure": 8,
  "minFocusDistance": 15
}

Device

iPhone 12 Pro Max (IOS 17.5.1)

VisionCamera Version

4.4.1

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

• I am using Expo
• I have enabled Frame Processors (react-native-worklets-core)
• I have read the Troubleshooting Guide
• I agree to follow this project's Code of Conduct
• I searched for similar issues in this repository and found none.

[maintenance-hans]

Guten Tag, Hans here.

Note

New features, bugfixes, updates and other improvements are all handled mostly by @mrousavy in his free time.
To support @mrousavy, please consider 💖 sponsoring him on GitHub 💖.
Sponsored issues will be prioritized.

[christophemenager]

I do have the same issue, only on iOS

[Marius-Adam]

I do have the same issue, only on iOS

IOS for me too

[emilje]

Same, iPhone 11.

[frabanca]

Same issue on iPhone 14 Pro and "react-native-vision-camera": "^4.5.0".

[Marius-Adam]

Blocking use of any frame processor with text recognition @mrousavy

[mrousavy]

Blocking use of any frame processor with text recognition @mrousavy
Please stop pinging me, I am not free 24h consultancy

---

The problem is with a Frame Processor Plugin you are using, not with my library.

I explained how Frame.orientation works in #3077, this is now finally the last change to the Frame.orientation prop and should give the plugin developer all they need to make it work.

In the case of react-native-vision-camera-text-recognition, he needs to update this switch statement with the correct values now. For example 0 is .up (as in the documentation) - not .right. With the current code it wouldn't have worked in selfie cameras or iPads.

If a Frame Processor Plugin wants to actually support rotation, they just need to add a parameter to their plugin where you can pass the current outputOrientation of the Camera.

cc @gev2002

[frabanca]

Hi @mrousavy , thanks for your reply and your time!

As often happens, your comments are very useful and allow us to understand the situation and in some cases manage it.

For example, your previous comment was very useful for me, because although I use another engine for OCR (this one: https://www.npmjs.com/package/@ismaelmoreiraa/vision-camera-ocr), it It was easy to find the file to edit in the plugin source (this file https://github.com/gev2002/react-native-vision-camera-text-recognition/blob/758041042d6608368fe2a41c38e3368c04f6b9c0/ios/VisionCameraTextRecognition.swift#L146-L151 a line 142) and insert the .right value instead of the .up value.

This small change - which I'm still testing - seems to be enough to solve the problem.

Thanks again for your time.

[mrousavy]

Thank you.

Yea the problem again was that Frame.orientation was already adjusted to output orientation before - and if you did that, it was impossible to reverse it for Skia based FPs. So we now just provide the raw value, and users can rotate it if they want to. Otherwise it only works in portrait :)

[frabanca]

Seems a good idea to give the plugin the raw value.
Maybe - to prevent incompatibility - you can pass both so you can allow legacy plugin to work like before.

[mrousavy]

Well it's both the same name unfortunately.

I could rename it to something like bufferOrientation and keep the old one (orientation)? I'll think about it, but to be fair, no frame processor plugin truly supported orientation before anyways.

[frabanca]

Considering how easy is to "fix" the plugin(s) code, maybe it's better to leave the property name "orientation" and let:

1. Plugin maintainers update their code
2. Plugin utilizers update/fix the code manually (as I did in less than 3 mins after your anware)

[alamothe]

We are using @ismaelmoreiraa/vision-camera-ocr and it has the same problem

[Flocurry]

Hello @mrousavy,

I've installed react-native-vision-camera-text-recognition@3.0.4

In the file android/src/main/java/com/visioncameratextrecognition/VisionCameraTextRecognitionModule.kt, I've modified the callback function and the getFrameRotation methods

override fun callback(frame: Frame, arguments: Map<String, Any>?): HashMap<String, Any>? {
        val data = WritableNativeMap()
        val mediaImage: Image? = frame.image
        mediaImage?.let {
            val rotation = getFrameRotation(frame.orientation)
            Log.d("callback FrameProcessor", "rotation: $rotation")
            val image = InputImage.fromMediaImage(it, rotation)
            val task: Task<Text> = recognizer.process(image)
            try {
                val text: Text = Tasks.await(task)
                if (text.text.isEmpty()) {
                    return WritableNativeMap().toHashMap()
                }
                data.putString("resultText", text.text)
                data.putArray("blocks", getBlocks(text.textBlocks))
                it.close()
                return data.toHashMap()
            } catch (e: Exception) {
                it.close()
                e.printStackTrace()
                return null
            }
        }
        return null
    }

    private fun getFrameRotation(orientation: Orientation): Int {
        Log.d("FrameProcessor", "Orientation: $orientation")
        return when (orientation) {
            Orientation.PORTRAIT -> 270
            Orientation.LANDSCAPE_LEFT -> 90
            Orientation.PORTRAIT_UPSIDE_DOWN -> 180
            Orientation.LANDSCAPE_RIGHT -> 270
        }
    }

You can see logs below (frame.orientation are same if I rotate the device)

I don't understand why my frame is not rotated in my app

You can see below my screenshots

Camera without frameProcessor props :

Camera with frameProcessor props :

Below is the frameProcessor code

import { useTextRecognition } from 'react-native-vision-camera-text-recognition';

const options = { language: 'latin' };
const { scanText } = useTextRecognition(options);

const frameProcessor = useFrameProcessor((frame) => {
    'worklet';
    const data = scanText(frame);
  }, []);

My app is locked in landscape mode

"react-native-vision-camera": "4.5.0",
"react-native-vision-camera-text-recognition": "^3.0.4",

[dalisalvador]

@Flocurry did you manage to solve this? I'm having the same issue and can't find a way around this without having to change the native code 🤔

Btw, I think this is not a plugin issue, so it doesn't matter if you make changes to the plugin code.
Aren't frames processed after the preview is rendered?