[camera] flip/change camera while recording (split out PR for cam_avf…

…oundation and cam_android) (#7109) * setDescription in Camera platform interface * example app setup to change description mid recording * AVFoundationCamera method call to setDescription * FLTCam setup to setDescription * captureSession split into video and audio so we will be able to switch cameras without breaking the audio * renamed setDescription to setDescriptionWhileRecording since it can only be used while recording * integration tests fixed * set description while recording integration test * throws error if device not recording and setDescriptionWhileRecording is called * set description while recording test * example project setup * camera preview can be changed while recording * camera switches and keeps surface pointed to mediarecorder * small change to set autofocus when switching while recording * android video record goes through VideoRenderer to apply matrix after switching camera * switch camera uses VideoRenderer * dont use video renderer until user switches camera while recording * rotate based on initial recording direction * VideoRenderer cleanup * flutter results for setDescriptionWhileRecording * error if you setDescriptionWhileRecording while device is not recording * android tests * integration tests * method channel test * main package tests * setDescriptionWhileRecording called while no video was recording test * integration tests * dependency overrides * update readme and version * removed old TODO * removed accidental dev team ID commit * renamed local variables * use captureSessionQueue * fixed local variable name * setupCaptureVideoOutput function * createConnectionWithInput * simplified configureConnection function to re-use code on switching camera * formatting * example project dependency overrides * fixed versioning * formatting * fixed some ios native tests * fixed small bug * dont emit initialized when switching camera * ios formatting * dependency overrides for camera/example * android formatting * ios test formatted * android tests formatted * android format that I missed * other android formatting * final formatting with flutter tool * formatted android again * android license in new file * update-excerpts ran * fixed changelog * removed development team * renames configureConnection to createConnection * renames unimplemented error message * renames setDescriptionWhileRecording error to match android and the other errors * fixes formatting * removes override dependencies from camera_web and camera_windows * removes camera_web override dependency in camera package * Update packages/camera/camera_android/android/src/main/java/io/flutter/plugins/camera/Camera.java Co-authored-by: Camille Simon <43054281+camsim99@users.noreply.github.com> * Update packages/camera/camera_android/android/src/main/java/io/flutter/plugins/camera/Camera.java Co-authored-by: Camille Simon <43054281+camsim99@users.noreply.github.com> * Update packages/camera/camera_android/android/src/main/java/io/flutter/plugins/camera/Camera.java Co-authored-by: Camille Simon <43054281+camsim99@users.noreply.github.com> * Update packages/camera/camera_android/android/src/main/java/io/flutter/plugins/camera/Camera.java Co-authored-by: Camille Simon <43054281+camsim99@users.noreply.github.com> * reformats camera.java * VideoRenderer uses surface texture timestamp instead of current system time * formats VideoRenderer.java * fixes comments in VideoRenderer.java * Update packages/camera/camera_platform_interface/lib/src/platform_interface/camera_platform.dart Co-authored-by: Maurice Parrish <10687576+bparrishMines@users.noreply.github.com> * Update packages/camera/camera/lib/src/camera_controller.dart Co-authored-by: Maurice Parrish <10687576+bparrishMines@users.noreply.github.com> * renames error typo * frees shaders after program linking * handles eglSwapBuffers errors * extension check guards eglPresentationTimeANDROID * cleans openGL resources * reverted timestamp to use uptimeMillis() * Tests for startPreviewWithVideoRendererStream * fixes exception not being caught * tests for correct rotation to be set * fixes versioning * tests method channel setDescriptionWhileRecording * adds forwarding getter on CameraController to its value's description * dummy commit to fix github test's not finding commit hash * adds override description for FakeController in camera tests * fixes versioning for avfoundation and android * fixes versioning * fixes pubspec versions * ios setDescription * setDescription * android setDescription * formatting * revert * nits and reverts * nits * fixes README * fixes other comments * fixes setDescription override in camera_preview_test * set description test * versions * removes changes on platform_interface_changes * points all packages to platform interface version 2.4 * points to the new platform interface * removes everything that isnt under camera_avfoundation and camera_android * removes dependency overrides in examples * removes version change on camera * removes camera changes that were missed * fixes android version --------- Co-authored-by: BradenBagby <braden.bagby@wavv.com> Co-authored-by: Camille Simon <43054281+camsim99@users.noreply.github.com> Co-authored-by: Maurice Parrish <10687576+bparrishMines@users.noreply.github.com>
flutter · Feb 14, 2023 · 9c312d4 · 9c312d4
1 parent f1a3fea
commit 9c312d4
Show file tree

Hide file tree

Showing 27 changed files with 1,014 additions and 125 deletions.
diff --git a/packages/camera/camera_android/CHANGELOG.md b/packages/camera/camera_android/CHANGELOG.md
@@ -1,3 +1,7 @@
+## 0.10.5
+
+* Allows camera to be switched while video recording.
+
 ## 0.10.4
 
 * Temporarily fixes issue with requested video profiles being null by falling back to deprecated behavior in that case.

diff --git a/packages/camera/camera_android/android/src/main/java/io/flutter/plugins/camera/Camera.java b/packages/camera/camera_android/android/src/main/java/io/flutter/plugins/camera/Camera.java
@@ -115,13 +115,28 @@ class Camera
    * Holds all of the camera features/settings and will be used to update the request builder when
    * one changes.
    */
-  private final CameraFeatures cameraFeatures;
+  private CameraFeatures cameraFeatures;
+
+  private String imageFormatGroup;
+
+  /**
+   * Takes an input/output surface and orients the recording correctly. This is needed because
+   * switching cameras while recording causes the wrong orientation.
+   */
+  private VideoRenderer videoRenderer;
+
+  /**
+   * Whether or not the camera aligns with the initial way the camera was facing if the camera was
+   * flipped.
+   */
+  private int initialCameraFacing;
 
   private final SurfaceTextureEntry flutterTexture;
+  private final ResolutionPreset resolutionPreset;
   private final boolean enableAudio;
   private final Context applicationContext;
   private final DartMessenger dartMessenger;
-  private final CameraProperties cameraProperties;
+  private CameraProperties cameraProperties;
   private final CameraFeatureFactory cameraFeatureFactory;
   private final Activity activity;
   /** A {@link CameraCaptureSession.CaptureCallback} that handles events related to JPEG capture. */
@@ -211,6 +226,7 @@ public Camera(
     this.applicationContext = activity.getApplicationContext();
     this.cameraProperties = cameraProperties;
     this.cameraFeatureFactory = cameraFeatureFactory;
+    this.resolutionPreset = resolutionPreset;
     this.cameraFeatures =
         CameraFeatures.init(
             cameraFeatureFactory, cameraProperties, activity, dartMessenger, resolutionPreset);
@@ -251,6 +267,7 @@ private void prepareMediaRecorder(String outputFilePath) throws IOException {
     if (mediaRecorder != null) {
       mediaRecorder.release();
     }
+    closeRenderer();
 
     final PlatformChannel.DeviceOrientation lockedOrientation =
         ((SensorOrientationFeature) cameraFeatures.getSensorOrientation())
@@ -279,6 +296,7 @@ private void prepareMediaRecorder(String outputFilePath) throws IOException {
 
   @SuppressLint("MissingPermission")
   public void open(String imageFormatGroup) throws CameraAccessException {
+    this.imageFormatGroup = imageFormatGroup;
     final ResolutionFeature resolutionFeature = cameraFeatures.getResolution();
 
     if (!resolutionFeature.checkIsSupported()) {
@@ -323,14 +341,16 @@ public void onOpened(@NonNull CameraDevice device) {
             cameraDevice = new DefaultCameraDeviceWrapper(device);
             try {
               startPreview();
+              if (!recordingVideo) // only send initialization if we werent already recording and switching cameras
               dartMessenger.sendCameraInitializedEvent(
-                  resolutionFeature.getPreviewSize().getWidth(),
-                  resolutionFeature.getPreviewSize().getHeight(),
-                  cameraFeatures.getExposureLock().getValue(),
-                  cameraFeatures.getAutoFocus().getValue(),
-                  cameraFeatures.getExposurePoint().checkIsSupported(),
-                  cameraFeatures.getFocusPoint().checkIsSupported());
-            } catch (CameraAccessException e) {
+                    resolutionFeature.getPreviewSize().getWidth(),
+                    resolutionFeature.getPreviewSize().getHeight(),
+                    cameraFeatures.getExposureLock().getValue(),
+                    cameraFeatures.getAutoFocus().getValue(),
+                    cameraFeatures.getExposurePoint().checkIsSupported(),
+                    cameraFeatures.getFocusPoint().checkIsSupported());
+
+            } catch (CameraAccessException | InterruptedException e) {
               dartMessenger.sendCameraErrorEvent(e.getMessage());
               close();
             }
@@ -340,7 +360,8 @@ public void onOpened(@NonNull CameraDevice device) {
           public void onClosed(@NonNull CameraDevice camera) {
             Log.i(TAG, "open | onClosed");
 
-            // Prevents calls to methods that would otherwise result in IllegalStateException exceptions.
+            // Prevents calls to methods that would otherwise result in IllegalStateException
+            // exceptions.
             cameraDevice = null;
             closeCaptureSession();
             dartMessenger.sendCameraClosingEvent();
@@ -756,7 +777,7 @@ public void startVideoRecording(
     if (imageStreamChannel != null) {
       setStreamHandler(imageStreamChannel);
     }
-
+    initialCameraFacing = cameraProperties.getLensFacing();
     recordingVideo = true;
     try {
       startCapture(true, imageStreamChannel != null);
@@ -768,6 +789,13 @@ public void startVideoRecording(
     }
   }
 
+  private void closeRenderer() {
+    if (videoRenderer != null) {
+      videoRenderer.close();
+      videoRenderer = null;
+    }
+  }
+
   public void stopVideoRecording(@NonNull final Result result) {
     if (!recordingVideo) {
       result.success(null);
@@ -778,6 +806,7 @@ public void stopVideoRecording(@NonNull final Result result) {
         cameraFeatureFactory.createAutoFocusFeature(cameraProperties, false));
     recordingVideo = false;
     try {
+      closeRenderer();
       captureSession.abortCaptures();
       mediaRecorder.stop();
     } catch (CameraAccessException | IllegalStateException e) {
@@ -786,7 +815,7 @@ public void stopVideoRecording(@NonNull final Result result) {
     mediaRecorder.reset();
     try {
       startPreview();
-    } catch (CameraAccessException | IllegalStateException e) {
+    } catch (CameraAccessException | IllegalStateException | InterruptedException e) {
       result.error("videoRecordingFailed", e.getMessage(), null);
       return;
     }
@@ -1070,13 +1099,51 @@ public void resumePreview() {
         null, (code, message) -> dartMessenger.sendCameraErrorEvent(message));
   }
 
-  public void startPreview() throws CameraAccessException {
+  public void startPreview() throws CameraAccessException, InterruptedException {
+    // If recording is already in progress, the camera is being flipped, so send it through the VideoRenderer to keep the correct orientation.
+    if (recordingVideo) {
+      startPreviewWithVideoRendererStream();
+    } else {
+      startRegularPreview();
+    }
+  }
+
+  private void startRegularPreview() throws CameraAccessException {
     if (pictureImageReader == null || pictureImageReader.getSurface() == null) return;
     Log.i(TAG, "startPreview");
-
     createCaptureSession(CameraDevice.TEMPLATE_PREVIEW, pictureImageReader.getSurface());
   }
 
+  private void startPreviewWithVideoRendererStream()
+      throws CameraAccessException, InterruptedException {
+    if (videoRenderer == null) return;
+
+    // get rotation for rendered video
+    final PlatformChannel.DeviceOrientation lockedOrientation =
+        ((SensorOrientationFeature) cameraFeatures.getSensorOrientation())
+            .getLockedCaptureOrientation();
+    DeviceOrientationManager orientationManager =
+        cameraFeatures.getSensorOrientation().getDeviceOrientationManager();
+
+    int rotation = 0;
+    if (orientationManager != null) {
+      rotation =
+          lockedOrientation == null
+              ? orientationManager.getVideoOrientation()
+              : orientationManager.getVideoOrientation(lockedOrientation);
+    }
+
+    if (cameraProperties.getLensFacing() != initialCameraFacing) {
+
+      // If the new camera is facing the opposite way than the initial recording,
+      // the rotation should be flipped 180 degrees.
+      rotation = (rotation + 180) % 360;
+    }
+    videoRenderer.setRotation(rotation);
+
+    createCaptureSession(CameraDevice.TEMPLATE_RECORD, videoRenderer.getInputSurface());
+  }
+
   public void startPreviewWithImageStream(EventChannel imageStreamChannel)
       throws CameraAccessException {
     setStreamHandler(imageStreamChannel);
@@ -1200,17 +1267,7 @@ private void closeCaptureSession() {
   public void close() {
     Log.i(TAG, "close");
 
-    if (cameraDevice != null) {
-      cameraDevice.close();
-      cameraDevice = null;
-
-      // Closing the CameraDevice without closing the CameraCaptureSession is recommended
-      // for quickly closing the camera:
-      // https://developer.android.com/reference/android/hardware/camera2/CameraCaptureSession#close()
-      captureSession = null;
-    } else {
-      closeCaptureSession();
-    }
+    stopAndReleaseCamera();
 
     if (pictureImageReader != null) {
       pictureImageReader.close();
@@ -1229,6 +1286,66 @@ public void close() {
     stopBackgroundThread();
   }
 
+  private void stopAndReleaseCamera() {
+    if (cameraDevice != null) {
+      cameraDevice.close();
+      cameraDevice = null;
+
+      // Closing the CameraDevice without closing the CameraCaptureSession is recommended
+      // for quickly closing the camera:
+      // https://developer.android.com/reference/android/hardware/camera2/CameraCaptureSession#close()
+      captureSession = null;
+    } else {
+      closeCaptureSession();
+    }
+  }
+
+  private void prepareVideoRenderer() {
+    if (videoRenderer != null) return;
+    final ResolutionFeature resolutionFeature = cameraFeatures.getResolution();
+
+    // handle videoRenderer errors
+    Thread.UncaughtExceptionHandler videoRendererUncaughtExceptionHandler =
+        new Thread.UncaughtExceptionHandler() {
+          @Override
+          public void uncaughtException(Thread thread, Throwable ex) {
+            dartMessenger.sendCameraErrorEvent(
+                "Failed to process frames after camera was flipped.");
+          }
+        };
+
+    videoRenderer =
+        new VideoRenderer(
+            mediaRecorder.getSurface(),
+            resolutionFeature.getCaptureSize().getWidth(),
+            resolutionFeature.getCaptureSize().getHeight(),
+            videoRendererUncaughtExceptionHandler);
+  }
+
+  public void setDescriptionWhileRecording(
+      @NonNull final Result result, CameraProperties properties) {
+
+    if (!recordingVideo) {
+      result.error("setDescriptionWhileRecordingFailed", "Device was not recording", null);
+      return;
+    }
+
+    stopAndReleaseCamera();
+    prepareVideoRenderer();
+    cameraProperties = properties;
+    cameraFeatures =
+        CameraFeatures.init(
+            cameraFeatureFactory, cameraProperties, activity, dartMessenger, resolutionPreset);
+    cameraFeatures.setAutoFocus(
+        cameraFeatureFactory.createAutoFocusFeature(cameraProperties, true));
+    try {
+      open(imageFormatGroup);
+    } catch (CameraAccessException e) {
+      result.error("setDescriptionWhileRecordingFailed", e.getMessage(), null);
+    }
+    result.success(null);
+  }
+
   public void dispose() {
     Log.i(TAG, "dispose");
 

diff --git a/...camera_android/android/src/main/java/io/flutter/plugins/camera/MethodCallHandlerImpl.java b/...camera_android/android/src/main/java/io/flutter/plugins/camera/MethodCallHandlerImpl.java
@@ -354,6 +354,18 @@ public void onMethodCall(@NonNull MethodCall call, @NonNull final Result result)
           result.success(null);
           break;
         }
+      case "setDescriptionWhileRecording":
+        {
+          try {
+            String cameraName = call.argument("cameraName");
+            CameraProperties cameraProperties =
+                new CameraPropertiesImpl(cameraName, CameraUtils.getCameraManager(activity));
+            camera.setDescriptionWhileRecording(result, cameraProperties);
+          } catch (Exception e) {
+            handleException(e, result);
+          }
+          break;
+        }
       case "dispose":
         {
           if (camera != null) {