Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[camera_avfoundation] ignore audio samples until first video sample #4587

Merged
merged 21 commits into from
Sep 18, 2023
Merged

[camera_avfoundation] ignore audio samples until first video sample #4587

merged 21 commits into from
Sep 18, 2023

Conversation

misos1
Copy link
Contributor

@misos1 misos1 commented Jul 29, 2023 •
edited
Loading

Samples from ios video and audio input should be regarded as arriving from independent streams because in fact they are going from independent sources. Video is going from instance AVCaptureVideoDataOutput while audio from AVCaptureAudioDataOutput. Therefore it is possible that one of them will be initialised soon or later than the other and the first few samples will be only audio samples so the video file will start without image data and will be black until the first video sample arrives. This patch addresses this by ignoring audio samples until the first video sample arrives (alternatively it could wait until both video and audio samples are actively arriving).

Fix flutter/flutter#57831

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the relevant style guides and ran the auto-formatter. (Unlike the flutter/flutter repo, the flutter/packages repo does use dart format.)
  • I signed the CLA.
  • The title of the PR starts with the name of the package surrounded by square brackets, e.g. [shared_preferences]
  • I listed at least one issue that this PR fixes in the description above.
  • I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy, or this PR is exempt from version changes.
  • I updated CHANGELOG.md to add a description of the change, following repository CHANGELOG style.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@misos1 misos1 requested a review from hellohuanlin as a code owner July 29, 2023 17:13
@flutter-dashboard
Copy link

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

@google-cla
Copy link

google-cla bot commented Jul 29, 2023

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@hellohuanlin hellohuanlin marked this pull request as draft July 31, 2023 17:07
@hellohuanlin
Copy link
Contributor

Feel free to request my review after you add some tests. In the mean time I am converting this to draft.

@misos1 misos1 marked this pull request as ready for review August 3, 2023 18:12
@misos1
Copy link
Contributor Author

misos1 commented Aug 8, 2023

@hellohuanlin I added test, now it should be ready

@stuartmorgan
Copy link
Contributor

From triage: Ping @hellohuanlin on this review.

@misos1 misos1 requested a review from hellohuanlin August 16, 2023 20:37
@misos1 misos1 requested a review from hellohuanlin August 17, 2023 17:34
@toshifumimatsuno toshifumimatsuno mentioned this pull request Aug 28, 2023
Closed
@stuartmorgan
Copy link
Contributor

@bparrishMines for second review

stuartmorgan
stuartmorgan approved these changes Sep 9, 2023
View reviewed changes
Copy link
Contributor

@stuartmorgan stuartmorgan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM; once the version is re-bumped to resolve the conflicts with main we can land this. Thanks!

@stuartmorgan stuartmorgan added the autosubmit Merge PR when tree becomes green via auto submit App label Sep 18, 2023
@auto-submit auto-submit bot merged commit 09df102 into flutter:main Sep 18, 2023
@engine-flutter-autoroll engine-flutter-autoroll mentioned this pull request Sep 19, 2023
Closed
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Sep 19, 2023
@engine-flutter-autoroll
09df102 [camera_avfoundation] ignore audio samples until first video …
759cde2 
…sample (flutter/packages#4587)
@engine-flutter-autoroll engine-flutter-autoroll mentioned this pull request Sep 19, 2023
Closed
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Sep 19, 2023
@engine-flutter-autoroll
09df102 [camera_avfoundation] ignore audio samples until first video …
af74478 
…sample (flutter/packages#4587)
@engine-flutter-autoroll engine-flutter-autoroll mentioned this pull request Sep 19, 2023
Closed
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Sep 19, 2023
@engine-flutter-autoroll
09df102 [camera_avfoundation] ignore audio samples until first video …
27a0bf8 
…sample (flutter/packages#4587)
@engine-flutter-autoroll engine-flutter-autoroll mentioned this pull request Sep 20, 2023
Merged
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Sep 20, 2023
@engine-flutter-autoroll
09df102 [camera_avfoundation] ignore audio samples until first video …
00faa60 
…sample (flutter/packages#4587)
auto-submit bot pushed a commit to flutter/flutter that referenced this pull request Sep 20, 2023
@engine-flutter-autoroll
Roll Packages from d4e2454 to 51e74b9 (12 revisions) (#135145)
3a78e5c 
flutter/packages@d4e2454...51e74b9

2023-09-19 stuartmorgan@google.com [ios_platform_images] Convert to Pigeon (flutter/packages#4945)
2023-09-19 abdeluached@gmail.com [go_router_builder] Generate initialLocation with StatefulShellBranchConfig (flutter/packages#4880)
2023-09-19 engine-flutter-autoroll@skia.org Roll Flutter from b7d0e8c to 8936504 (15 revisions) (flutter/packages#4947)
2023-09-18 47866232+chunhtai@users.noreply.github.com [go_router] Adds on exit (flutter/packages#4699)
2023-09-18 47866232+chunhtai@users.noreply.github.com [go_router] Fixes RouteInformationParser that does not restore full RouteMatchList if the optionURLReflectsImperativeAPIs is set (flutter/packages#4713)
2023-09-18 stuartmorgan@google.com [file_selector] Fix unknown extensions on macOS (flutter/packages#4946)
2023-09-18 30872003+misos1@users.noreply.github.com [camera_avfoundation] ignore audio samples until first video sample (flutter/packages#4587)
2023-09-18 engine-flutter-autoroll@skia.org Roll Flutter from 1b18b13 to b7d0e8c (11 revisions) (flutter/packages#4944)
2023-09-18 tarrinneal@gmail.com [video_player] isCompleted event. (flutter/packages#4923)
2023-09-18 32538273+ValentinVignal@users.noreply.github.com [go_router] Fix an issue in the documentation that used `state.queryParameters` instead of `state.uri.queryParameters` (flutter/packages#4881)
2023-09-18 stuartmorgan@google.com [google_sign_in] Convert iOS to Pigeon (flutter/packages#4941)
2023-09-18 stuartmorgan@google.com Add dashboard link to README (flutter/packages#4902)

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-packages-flutter-autoroll
Please CC flutter-ecosystem@google.com,rmistry@google.com on the revert to ensure that a human
is aware of the problem.

To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
pro100andrey added a commit to pro100andrey/packages that referenced this pull request Sep 25, 2023
@pro100andrey
Merge branch 'main' into feature/fix
6787627 
* main: (590 commits)
  Bump github/codeql-action from 2.21.7 to 2.21.8 (flutter#4952)
  Manual roll Flutter from 8936504 to f92884c (48 revisions) (flutter#4985)
  [image_picker_android] check if data from result is empty when picking a single img or video (flutter#4836)
  [camera_android] Removes usage of `_ambiguate` method in tests (flutter#4948)
  [sign_in]: Bump com.google.android.gms:play-services-auth from 20.6.0 to 20.7.0 in /packages/google_sign_in/google_sign_in_android/android (flutter#4841)
  Manual roll Flutter (stable) from 367f9ea16bfa to 12fccda59847 (2 revisions) (flutter#4976)
  [go_router] Fixes the GoRouter.goBranch so that it doesn't reset extr… (flutter#4723)
  [webview_flutter_wkwebview] Adds WKWebView implementation to override console log (flutter#4703)
  [various] Update invalid NSURL tests (flutter#4959)
  [ios_platform_images] Convert to Pigeon (flutter#4945)
  [go_router_builder] Generate initialLocation with StatefulShellBranchConfig (flutter#4880)
  Roll Flutter from b7d0e8c to 8936504 (15 revisions) (flutter#4947)
  [go_router] Adds on exit (flutter#4699)
  [go_router] Fixes RouteInformationParser that does not restore full RouteMatchList if the optionURLReflectsImperativeAPIs is set (flutter#4713)
  [file_selector] Fix unknown extensions on macOS (flutter#4946)
  [camera_avfoundation] ignore audio samples until first video sample (flutter#4587)
  Roll Flutter from 1b18b13 to b7d0e8c (11 revisions) (flutter#4944)
  [video_player] isCompleted event. (flutter#4923)
  [go_router] Fix an issue in the documentation that used `state.queryParameters` instead of `state.uri.queryParameters` (flutter#4881)
  [google_sign_in] Convert iOS to Pigeon (flutter#4941)
  ...

# Conflicts:
#	packages/camera/camera/pubspec.yaml
Mairramer pushed a commit to Mairramer/flutter that referenced this pull request Oct 10, 2023
@engine-flutter-autoroll @Mairramer
Roll Packages from d4e2454 to 51e74b9 (12 revisions) (flutter#135145)
8a149ab 
flutter/packages@d4e2454...51e74b9

2023-09-19 stuartmorgan@google.com [ios_platform_images] Convert to Pigeon (flutter/packages#4945)
2023-09-19 abdeluached@gmail.com [go_router_builder] Generate initialLocation with StatefulShellBranchConfig (flutter/packages#4880)
2023-09-19 engine-flutter-autoroll@skia.org Roll Flutter from b7d0e8c to 8936504 (15 revisions) (flutter/packages#4947)
2023-09-18 47866232+chunhtai@users.noreply.github.com [go_router] Adds on exit (flutter/packages#4699)
2023-09-18 47866232+chunhtai@users.noreply.github.com [go_router] Fixes RouteInformationParser that does not restore full RouteMatchList if the optionURLReflectsImperativeAPIs is set (flutter/packages#4713)
2023-09-18 stuartmorgan@google.com [file_selector] Fix unknown extensions on macOS (flutter/packages#4946)
2023-09-18 30872003+misos1@users.noreply.github.com [camera_avfoundation] ignore audio samples until first video sample (flutter/packages#4587)
2023-09-18 engine-flutter-autoroll@skia.org Roll Flutter from 1b18b13 to b7d0e8c (11 revisions) (flutter/packages#4944)
2023-09-18 tarrinneal@gmail.com [video_player] isCompleted event. (flutter/packages#4923)
2023-09-18 32538273+ValentinVignal@users.noreply.github.com [go_router] Fix an issue in the documentation that used `state.queryParameters` instead of `state.uri.queryParameters` (flutter/packages#4881)
2023-09-18 stuartmorgan@google.com [google_sign_in] Convert iOS to Pigeon (flutter/packages#4941)
2023-09-18 stuartmorgan@google.com Add dashboard link to README (flutter/packages#4902)

If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-packages-flutter-autoroll
Please CC flutter-ecosystem@google.com,rmistry@google.com on the revert to ensure that a human
is aware of the problem.

To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
HugoOlthof pushed a commit to moneybird/packages that referenced this pull request Dec 13, 2023
@misos1 @HugoOlthof
[camera_avfoundation] ignore audio samples until first video sample (f…
ae68828 
…lutter#4587)

Samples from ios video and audio input should be regarded as arriving from independent streams because in fact they are going from independent sources. Video is going from instance `AVCaptureVideoDataOutput` while audio from `AVCaptureAudioDataOutput`. Therefore it is possible that one of them will be initialised soon or later than the other and the first few samples will be only audio samples so the video file will start without image data and will be black until the first video sample arrives. This patch addresses this by ignoring audio samples until the first video sample arrives (alternatively it could wait until both video and audio samples are actively arriving).

Fix flutter/flutter#57831
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hellohuanlin hellohuanlin hellohuanlin approved these changes

@stuartmorgan stuartmorgan stuartmorgan approved these changes

@bparrishMines bparrishMines Awaiting requested review from bparrishMines

Assignees
No one assigned
Labels
autosubmit Merge PR when tree becomes green via auto submit App p: camera platform-ios
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

when using camera plugin to record video in ios, the first frame may be black
3 participants
@misos1 @hellohuanlin @stuartmorgan