[camerax] Implement Image Streaming #3454

camsim99 · 2023-03-13T18:36:40Z

Implements image streaming with all required image data, see here.

Fixes flutter/flutter#120463.
Fixes flutter/flutter#113792 along the way.

Special shoutout to @reidbaker for pair programming on this!

Pre-launch Checklist

…imgcap

…o/flutter/plugins/camerax/ImageCaptureHostApiImpl.java Co-authored-by: Maurice Parrish <10687576+bparrishMines@users.noreply.github.com>

Co-authored-by: Maurice Parrish <10687576+bparrishMines@users.noreply.github.com>

...ndroid_camerax/android/src/main/java/io/flutter/plugins/camerax/GeneratedCameraXLibrary.java

reidbaker · 2023-04-25T20:47:45Z

...ndroid_camerax/android/src/main/java/io/flutter/plugins/camerax/GeneratedCameraXLibrary.java

-    /** Constructor is private to enforce null safety; use Builder. */
-    private ResolutionInfo() {}
+    /** Constructor is non-public to enforce null safety; use Builder. */
+    ResolutionInfo() {}


Java behaves differently to dart non stated means package private which means that customer code can call it if they put their code in the right path. I think this should be protected.

This is generated.

Can you set the scope on the class it is generated from?

I don't think so

...ndroid_camerax/android/src/main/java/io/flutter/plugins/camerax/GeneratedCameraXLibrary.java

...ra/camera_android_camerax/android/src/test/java/io/flutter/plugins/camerax/AnalyzerTest.java

.../android/app/src/androidTest/java/io/flutter/plugins/cameraxexample/InstanceManagerTest.java

packages/camera/camera_android_camerax/lib/src/android_camera_camerax.dart

reidbaker · 2023-04-25T21:11:56Z

packages/camera/camera_android_camerax/lib/src/plane_proxy.dart

+  late final _PlaneProxyHostApiImpl _api;
+
+  /// Returns the pixels buffer containing frame data.
+  Future<Uint8List> getBuffer() {


Can you lay out pros/cons or maybe use cases that are harder/easier for getters vs the current implementation.

packages/camera/camera_android_camerax/test/android_camera_camerax_test.dart

bparrishMines

LGTM with a few comments and a question about the InstanceManager test

.../android/app/src/androidTest/java/io/flutter/plugins/cameraxexample/InstanceManagerTest.java

reidbaker · 2023-04-27T18:54:21Z

packages/camera/camera_android_camerax/lib/src/android_camera_camerax.dart

@@ -252,7 +266,8 @@ class AndroidCameraCameraX extends CameraPlatform {
  /// [cameraId] not used.
  @override
  Future<void> pausePreview(int cameraId) async {
-    _unbindPreviewFromLifecycle();
+    assert(preview != null);


This will call _unbindUseCase with a null preview in release builds. I suggest you either make this a hard check on preview nullability or remove this line.

packages/camera/camera_android_camerax/lib/src/android_camera_camerax.dart

reidbaker · 2023-04-27T18:57:09Z

packages/camera/camera_android_camerax/lib/src/android_camera_camerax.dart

+
+  /// Unbinds [useCase] from camera lifecycle controlled by the
+  /// [processCameraProvider].
+  Future<void> _unbindUseCaseFromLifecycle(UseCase useCase) async {


@bparrishMines does the ecosystem team have a guidance for code that is not used but might be useful? Most (but not all) of my past teams encouraged removing code like this.

reidbaker · 2023-04-27T18:59:12Z

packages/camera/camera_android_camerax/lib/src/image_analysis.dart

+  Future<void> clearAnalyzer() => _api.clearAnalyzerfromInstances(this);
+}
+
+/// Host API implementation of [ImageAnalysis].


Can you update the dart doc to describe when you expect callers to use a global and when you expect them to specify instanceManager?

@bparrishMines can you verify my understanding? Added at line 64

There should only be one scenario when we pass an InstanceManager to the HostApi. The summary could be shortened:

An [instanceManager] is typically passed when a copy of an instance contained by an `InstanceManager` is being created.

packages/camera/camera_android_camerax/lib/src/plane_proxy.dart

reidbaker · 2023-04-27T22:43:41Z

packages/camera/camera_android_camerax/lib/src/android_camera_camerax.dart

+
+  /// Unbinds [useCase] from camera lifecycle controlled by the
+  /// [processCameraProvider].
+  Future<void> _unbindUseCaseFromLifecycle(UseCase useCase) async {
    assert(processCameraProvider != null);


FYI you have another assert here followed by a use with !.

Going to remove all asserts in this PR.

Edit: all assert statements in the plugin API implementation (android_camera_camerax.dart). Done in this commit

reidbaker

Not blocking this on my review anymore. Please file issues or have some other way to track any unresolved comments.

* main: (46 commits) [go_router] Cleans up route match API and introduces dart fix (flutter#3819) [camerax] Add `LifecycleOwner` Proxy (flutter#3837) [file_selector] Add getDirectoryPaths implementation for Windows (flutter#3704) [various] Update Android example min SDKs (flutter#3847) Bump github/codeql-action from 2.2.12 to 2.3.2 (flutter#3838) [camerax] Implement Image Streaming (flutter#3454) [various] update agp and gradle for all examples in packages (flutter#3822) Update xcode to 14c18 (flutter#3774) [camera_android] Add NV21 as an image stream format flutter#3277 (flutter#3639) [go_router] Remove unused navigator keys (flutter#3708) [image_picker] Move I/O operations to a separate thread (flutter#3506) [pigeon]: Bump org.jetbrains.kotlin:kotlin-gradle-plugin from 1.8.20 to 1.8.21 in /packages/pigeon/platform_tests/test_plugin/android (flutter#3824) [pigeon] Reland: Add an initial example app (flutter#3832) Roll Flutter from c9004ff to 66fa4c5 (68 revisions) (flutter#3830) [various] Conditionalize the namespace in all Android plugins (flutter#3836) [in_app_pur]: Bump com.android.billingclient:billing from 5.1.0 to 5.2.0 in /packages/in_app_purchase/in_app_purchase_android/android (flutter#3672) [auick_action_ios] Retries multiple times to not fail ci when there is a flake (flutter#3823) Swap some iOS package CODEOWNERS (flutter#3793) [various] Add `targetCompatibility` to build.gradle (flutter#3825) [various] Set cmake_policy versions (flutter#3828) ...

Roll Packages from 7e3f5da to de6131d (41 revisions) flutter/packages@7e3f5da...de6131d 2023-05-01 reidbaker@google.com I122213 update non examples (flutter/packages#3846) 2023-04-29 47866232+chunhtai@users.noreply.github.com [go_router] Cleans up route match API and introduces dart fix (flutter/packages#3819) 2023-04-29 43054281+camsim99@users.noreply.github.com [camerax] Add `LifecycleOwner` Proxy (flutter/packages#3837) 2023-04-29 stuartmorgan@google.com [file_selector] Add getDirectoryPaths implementation for Windows (flutter/packages#3704) 2023-04-29 stuartmorgan@google.com [various] Update Android example min SDKs (flutter/packages#3847) 2023-04-28 49699333+dependabot[bot]@users.noreply.github.com Bump github/codeql-action from 2.2.12 to 2.3.2 (flutter/packages#3838) 2023-04-28 43054281+camsim99@users.noreply.github.com [camerax] Implement Image Streaming (flutter/packages#3454) 2023-04-28 reidbaker@google.com [various] update agp and gradle for all examples in packages (flutter/packages#3822) 2023-04-28 54558023+keyonghan@users.noreply.github.com Update xcode to 14c18 (flutter/packages#3774) 2023-04-28 andrewjohncoutts@gmail.com [camera_android] Add NV21 as an image stream format #3277 (flutter/packages#3639) 2023-04-28 gautier.bayzelon@gmail.com [go_router] Remove unused navigator keys (flutter/packages#3708) 2023-04-28 JeroenWeener@users.noreply.github.com [image_picker] Move I/O operations to a separate thread (flutter/packages#3506) 2023-04-28 49699333+dependabot[bot]@users.noreply.github.com [pigeon]: Bump org.jetbrains.kotlin:kotlin-gradle-plugin from 1.8.20 to 1.8.21 in /packages/pigeon/platform_tests/test_plugin/android (flutter/packages#3824) 2023-04-28 stuartmorgan@google.com [pigeon] Reland: Add an initial example app (flutter/packages#3832) 2023-04-28 engine-flutter-autoroll@skia.org Roll Flutter from c9004ff to 66fa4c5 (68 revisions) (flutter/packages#3830) 2023-04-28 stuartmorgan@google.com [various] Conditionalize the namespace in all Android plugins (flutter/packages#3836) 2023-04-27 49699333+dependabot[bot]@users.noreply.github.com [in_app_pur]: Bump com.android.billingclient:billing from 5.1.0 to 5.2.0 in /packages/in_app_purchase/in_app_purchase_android/android (flutter/packages#3672) 2023-04-27 ychris@google.com [auick_action_ios] Retries multiple times to not fail ci when there is a flake (flutter/packages#3823) 2023-04-26 magder@google.com Swap some iOS package CODEOWNERS (flutter/packages#3793) 2023-04-26 stuartmorgan@google.com [various] Add `targetCompatibility` to build.gradle (flutter/packages#3825) 2023-04-26 stuartmorgan@google.com [various] Set cmake_policy versions (flutter/packages#3828) 2023-04-26 stuartmorgan@google.com [path_provider] Allow `win32` up to version 4.x (flutter/packages#3820) 2023-04-26 stuartmorgan@google.com [tool] Move Android lint checks (flutter/packages#3816) 2023-04-25 evace93@gmail.com [google_maps_flutter_android] Fix Android lint warnings (flutter/packages#3751) 2023-04-25 49699333+dependabot[bot]@users.noreply.github.com [in_app_pur]: Bump androidx.annotation:annotation from 1.5.0 to 1.6.0 in /packages/in_app_purchase/in_app_purchase_android/android (flutter/packages#3381) 2023-04-25 jason-simmons@users.noreply.github.com Update test golden images for the latest Skia roll (flutter/packages#3787) 2023-04-25 stuartmorgan@google.com [various] Adds Android namespace (flutter/packages#3791) 2023-04-25 reidbaker@google.com [shared_preferences] Update gradle/agp in example apps (flutter/packages#3809) 2023-04-24 43640732+dancamdev@users.noreply.github.com [go_router] Adds name to TypedGoRoute (flutter/packages#3702) 2023-04-22 10687576+bparrishMines@users.noreply.github.com [webview_flutter] Adds support to receive permission requests (flutter/packages#3543) 2023-04-21 stuartmorgan@google.com [google_sign_in] Fix Android Java warnings (flutter/packages#3762) 2023-04-21 stuartmorgan@google.com [local_auth] Fix enum return on Android (flutter/packages#3780) 2023-04-21 stuartmorgan@google.com [pigeon] Warn when trying to use enums in collections (flutter/packages#3782) 2023-04-21 10687576+bparrishMines@users.noreply.github.com [webview_flutter_android] [webview_flutter_wkwebview] Platform implementations for supporting permission requests (flutter/packages#3792) 2023-04-21 scheglov@google.com [pigeon] Update for compatibility with a future change to the analyzer. (flutter/packages#3789) 2023-04-21 10687576+bparrishMines@users.noreply.github.com [camera_android] Fix Android lint warnings (flutter/packages#3716) 2023-04-21 10687576+bparrishMines@users.noreply.github.com [webview_flutter_platform_interface] Adds method to receive permission requests (flutter/packages#3767) 2023-04-21 magder@google.com [image_picker_ios] In unit test write and read kCGImagePropertyExifUserComment property (flutter/packages#3783) 2023-04-21 widrans@gmail.com [go_router_builder] Fixed the return value of the generated push method (flutter/packages#3650) 2023-04-21 JeroenWeener@users.noreply.github.com [image_picker] Mention `launchMode: singleInstance` in README (flutter/packages#3759) 2023-04-21 stuartmorgan@google.com Revert "[pigeon] Add an initial example app" (flutter/packages#3785) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC flutter-ecosystem@google.com,rmistry@google.com on the revert to ensure that a human ...

@reidbaker

Implements image streaming with all required image data, see [here](flutter/flutter#113792). Fixes flutter/flutter#120463. Fixes flutter/flutter#113792 along the way. Special shoutout to @reidbaker for pair programming on this!

camsim99 and others added 23 commits February 23, 2023 14:35

Recreating image capture impl from flutter/plugins

6125eca

Merge remote-tracking branch 'upstream/main' into camx_imgcap

5be1371

Fix integration tesT

b2fce3d

Merge branch 'main' into camx_imgcap

ed62a9a

Analyzer pt 2

2e3027b

Merge branch 'camx_imgcap' of github.com:camsim99/packages into camx_…

fbd3fe5

…imgcap

Personal cleanup

dfa477e

Add initialization of controller

a651b92

Merge remote-tracking branch 'upstream/main' into camx_imgcap

884fd6c

Modify some docs

d5f8b91

Address review

060b7e5

Merge remote-tracking branch 'upstream/main' into camx_imgcap

ff50c70

Fix analyze

fe6f1fc

Update packages/camera/camera_android_camerax/android/src/main/java/i…

e2643ae

…o/flutter/plugins/camerax/ImageCaptureHostApiImpl.java Co-authored-by: Maurice Parrish <10687576+bparrishMines@users.noreply.github.com>

Update packages/camera/camera_android_camerax/lib/src/image_capture.dart

b7194f7

Co-authored-by: Maurice Parrish <10687576+bparrishMines@users.noreply.github.com>

Start addressing review

dadc230

Finish addressing review

d15e1d4

Fix analyze

6dc2b77

Generate mocks on stable

f826b62

Fix/add tests

0f7f29a

Add isBound dart test

1e9739b

Start impl

83d6969

Add image analysis files

0f387e2

github-actions bot added p: camera platform-android labels Mar 13, 2023

camsim99 added 5 commits March 13, 2023 12:26

Start filling in image streaming info

0bf4f9a

More dev

97912fc

Dev

b889e98

Fix flutter api

e96c2ec

Binding refactor

cc2fcbe

reidbaker reviewed Apr 25, 2023

View reviewed changes

bparrishMines approved these changes Apr 25, 2023

View reviewed changes

.../android/app/src/androidTest/java/io/flutter/plugins/cameraxexample/InstanceManagerTest.java Outdated Show resolved Hide resolved

camsim99 added 8 commits April 26, 2023 14:58

Change getters to params, fix java tests

914e244

Fix analyze, dart tests

b704ff5

Add nonnull annotation for lint

6bae352

Bump test rerun

e4f522e

Merge remote-tracking branch 'upstream/main' into camx_imgstr

9d70b2e

Add annotations, try fix for test

f2a6167

Formatting

c42f7a1

Add annotation

402af7b

reidbaker reviewed Apr 27, 2023

View reviewed changes

camsim99 added 2 commits April 27, 2023 12:53

Revert instance manager integration test changes

c711743

Address review, mock test instance manager host api for failing test

5091aa3

camsim99 requested a review from reidbaker April 27, 2023 21:26

reidbaker reviewed Apr 27, 2023

View reviewed changes

reidbaker approved these changes Apr 27, 2023

View reviewed changes

camsim99 added 2 commits April 27, 2023 16:58

Remove asserts in plugin code

c6c5753

Fix comment

619a165

camsim99 added the autosubmit Merge PR when tree becomes green via auto submit App label Apr 28, 2023

auto-submit bot merged commit 1a90429 into flutter:main Apr 28, 2023

engine-flutter-autoroll mentioned this pull request May 1, 2023

Roll Packages from 7e3f5da42e81 to b01736711cc7 (40 revisions) flutter/flutter#125799

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request May 1, 2023

1a90429 [camerax] Implement Image Streaming (flutter/packages#3454)

efbb969

engine-flutter-autoroll mentioned this pull request May 1, 2023

Roll Packages from 7e3f5da42e81 to de6131dbe4f7 (41 revisions) flutter/flutter#125811

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request May 1, 2023

1a90429 [camerax] Implement Image Streaming (flutter/packages#3454)

e1658e0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[camerax] Implement Image Streaming #3454

[camerax] Implement Image Streaming #3454

camsim99 commented Mar 13, 2023 •

edited

Loading

reidbaker Apr 25, 2023

camsim99 Apr 26, 2023

reidbaker Apr 27, 2023

camsim99 Apr 28, 2023

reidbaker Apr 25, 2023

bparrishMines left a comment

reidbaker Apr 27, 2023

reidbaker Apr 27, 2023

reidbaker Apr 27, 2023

camsim99 Apr 27, 2023

bparrishMines Apr 28, 2023

reidbaker Apr 27, 2023

camsim99 Apr 27, 2023 •

edited

Loading

reidbaker left a comment

[camerax] Implement Image Streaming #3454

[camerax] Implement Image Streaming #3454

Conversation

camsim99 commented Mar 13, 2023 • edited Loading

Pre-launch Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bparrishMines left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

camsim99 Apr 27, 2023 • edited Loading

Choose a reason for hiding this comment

reidbaker left a comment

Choose a reason for hiding this comment

camsim99 commented Mar 13, 2023 •

edited

Loading

camsim99 Apr 27, 2023 •

edited

Loading