[Mac] feat: Support screen capture for macOS. #24

cloudwebrtc · 2022-05-08T17:00:43Z

Try adding screen sharing for macOS:
TODO: (1/5)

…creen-capture-for-mac

cloudwebrtc · 2022-06-13T09:56:18Z

The windows can now be enumerated.

But have some issues here, when I drag the captured window and change the size, the image will crack. There may be a bug in libwebrtc. @hiroshihorie @davidliu @davidzhao Have any ideas? xD

This problem has been bothering me for two days.

hiroshihorie

Wow great work. I'll think about the resize issue.

cloudwebrtc · 2022-06-13T12:25:46Z

Here is some code showing how to call it, maybe it can help you to test on swift sdk
https://github.com/flutter-webrtc/flutter-webrtc/pull/940/files#diff-dff3e74bce71fb4d131c17ba1af18ac75e9b370124e392da2518866db2a9386c

davidliu · 2022-06-13T16:51:36Z

Haven't digged into the code, but just from the looks of the image, it seems like the frame isn't updated with the correct dimensions. Since the video frame is a continguous array, if the width isn't updated, the pixels would spill into the next/previous row. That explains why there's a slant and the image is seemingly wrapping around. (I assume the slant gets worse as you expand/contract the window.)

davidzhao · 2022-06-13T20:53:10Z

@hiroshihorie your MacOS version didn't have this issue IIRC. Perhaps there are some insights there?

cloudwebrtc · 2022-06-16T12:56:51Z

okay, everything is working fine now, a story about the width not being divisible by 32.

hiroshihorie · 2022-06-16T15:36:51Z

wow nice find, needs to multiple of 32 ?

cloudwebrtc · 2022-06-16T15:45:56Z

Yes, When the width can be divisible by 32, the picture is fine. Other widths must be added with some numbers, which can be divisible by 32, But at this time, you need to remove the extra width, a black edge xD, It may be related to the capture mechanism of CGWindowListCreateImage. For now, it works perfectly. I still need to do some testing in intel mac, to make sure it works 100%.

https://github.com/webrtc-sdk/webrtc/pull/24/files#diff-0cd0a6ae24c482adb040d158c9b2e162f20c03741100c9036083360d7d1be694R81

davidzhao

great work tracking this one down!

…e timestamp.

…efptr. Updates all webrtc code, to have a small followup cl to just add the "explicit" keyword. Patchset #24 passed all webrtc tests, with explicit. Bug: webrtc:13464 Change-Id: I39863d3752f73209b531120f66916dc9177bf63a Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/242363 Reviewed-by: Tomas Gunnarsson <tommi@webrtc.org> Commit-Queue: Niels Moller <nisse@webrtc.org> Cr-Commit-Position: refs/heads/main@{#35718}

…frames with resolutions higher than 720p.

cloudwebrtc · 2022-07-28T13:27:11Z

hey @vickyleu, Thanks for reporting the issue, and are there any steps to reproduce？
BTW, macOS version and cpu (m1 or intel) is useful info.

* feat: Add screen capture support for macOS. * chore: rename files. * chore: update. * chore: Implement DesktopSource, DesktopMediaList ojbc interface. * chore: add license. * fix: capture_state. * update. * update. * Hide private delegate declaration. * Re-generate thumbnail for all sources when call UpdateSourceList. * fix: Fix window resize issue. * update. * update. * fix for intel mac. * cleanup. * update. * Add Window/Screen change listener. * Add force reload for MediaListUpdate and capture cursor. * fix: Fix the screen freeze caused by the desktop frame not setting the timestamp. * update. * fix issue for screen sharing. * Fixed H264 profile-level-id limitation that could not encode desktop frames with resolutions higher than 720p.

[Mac] feat: Support screen capture for macOS. (#24) (#36) fix: Get thumbnails asynchronously. (#37) fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) Fix the crash when setting the fps of the virtual camera. (#62)

Use M125 as the latest version and migrate historical patches to m125 Patches Group: ## 1. Update README.md b6c65fc * Add Apache-2.0 license and some note to README.md. (#9) * Updated readme detailing changes from original (#42) * Adding membrane framework (#51) * Updated readme (#83) ## 2. Audio Device Optimization 7454824 * allow listen-only mode in AudioUnit, adjust when category changes (#2) * release mic when category changes (#5) * Change defaults to iOS defaults (#7) * Sync audio session config (#8) * feat: support bypass voice processing for iOS. (#15) * Remove MacBookPro audio pan right code (#22) * fix: Fix can't open mic alone when built-in AEC is enabled. (#29) * feat: add audio device changes detect for windows. (#41) * fix Linux compile (#47) * AudioUnit: Don't rely on category switch for mic indicator to turn off (#52) * Stop recording on mute (turn off mic indicator) (#55) * Cherry pick audio selection from m97 release (#35) * [Mac] Allow audio device selection (#21) * RTCAudioDeviceModule.outputDevice / inputDevice getter and setter (#80) * Allow custom audio processing by exposing AudioProcessingModule (#85) * Expose audio sample buffers for Android (#89) * feat: add external audio processor for android. (#103) * android: make audio output attributes modifiable (#118) * Fix external audio processor sample rate calculation (#108) * Expose remote audio sample buffers on RTCAudioTrack (#84) * Fix memory leak when creating audio CMSampleBuffer #86 ## 3. Simulcast/SVC support for iOS/Android. b0b9fe9 - Simulcast support for iOS SDK (#4) - Support for simulcast in Android SDK (#3) - include simulcast headers for mac also (#10) - Fix simulcast using hardware encoder on Android (#48) - Add scalabilityMode support for AV1/VP9. (#90) ## 4. Android improvements. 9aaaab5 - Start/Stop receiving stream method for VideoTrack (#25) - Properly remove observer upon deconstruction (#26) - feat: Expose setCodecPreferences/getCapabilities for android. (#61) - fix: add WrappedVideoDecoderFactory.java. (#74) ## 5. Darwin improvements a13ea17 - [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28) - Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40) - rotationOverride should not be assign (#44) - [ObjC] Expose properties / methods required for AV1 codec support (#60) - Workaround: Render PixelBuffer in RTCMTLVideoView (#58) - Improve iOS/macOS H264 encoder (#70) - fix: fix video encoder not resuming correctly upon foregrounding (#75). - add PrivacyInfo.xcprivacy to darwin frameworks. (#112) - Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114) - Thread-safe `RTCInitFieldTrialDictionary` (#116) - Set RTCCameraVideoCapturer initial zoom factor (#121) - Unlock configuration before starting capture session (#122) ## 6. Desktop Capture for macOS. 841d78f - [Mac] feat: Support screen capture for macOS. (#24) (#36) - fix: Get thumbnails asynchronously. (#37) - fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (#63) - Fix the crash when setting the fps of the virtual camera. (#62) ## 7. Frame Cryptor Support. fc08745 - feat: Frame Cryptor (aes gcm/cbc). (#54) - feat: key ratchet/derive. (#66) - fix: skip invalid key when decryption failed. (#81) - Improve e2ee, add setSharedKey to KeyProvider. (#88) - add failure tolerance for framecryptor. (#91) - fix h264 freeze. (#93) - Fix/send frame cryptor events from signaling thread (#95) - more improvements for E2EE. (#96) - remove too verbose logs (#107) - Add key ring size to keyProviderOptions. (#109) ## 8. Other improvements. eed6c8a - Added yuv_helper (#57) - ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65) - more yuv wrappers (#87) - Fix naming for yuv helper (#113) - Fix missing `RTC_OBJC_TYPE` macros (#100) --------- Co-authored-by: Hiroshi Horie <548776+hiroshihorie@users.noreply.github.com> Co-authored-by: David Zhao <dz@livekit.io> Co-authored-by: davidliu <davidliu@deviange.net> Co-authored-by: Angelika Serwa <angelika.serwa@gmail.com> Co-authored-by: Théo Monnom <theo.monnom@outlook.com>

[Mac] feat: Support screen capture for macOS. (webrtc-sdk#24) (webrtc-sdk#36) fix: Get thumbnails asynchronously. (webrtc-sdk#37) fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be cropped. (webrtc-sdk#63) Fix the crash when setting the fps of the virtual camera. (webrtc-sdk#62)

feat: Add screen capture support for macOS.

287775b

cloudwebrtc changed the title ~~feat: Add screen capture support for macOS.~~ feat: Support screen capture for macOS. May 8, 2022

cloudwebrtc changed the title ~~feat: Support screen capture for macOS.~~ [macOS] feat: Support screen capture for macOS. May 8, 2022

cloudwebrtc changed the title ~~[macOS] feat: Support screen capture for macOS.~~ [Mac] feat: Support screen capture for macOS. May 8, 2022

cloudwebrtc added 5 commits May 9, 2022 21:20

chore: rename files.

cf8aa12

Merge remote-tracking branch 'origin/m97_release' into feat/support-s…

c5f2467

…creen-capture-for-mac

chore: update.

21bb596

chore: Implement DesktopSource, DesktopMediaList ojbc interface.

e220419

chore: add license.

fc84231

cloudwebrtc marked this pull request as ready for review June 10, 2022 14:19

cloudwebrtc added 2 commits June 11, 2022 01:00

fix: capture_state.

cd27d6a

update.

061bcd8

update.

5a9083d

hiroshihorie approved these changes Jun 13, 2022

View reviewed changes

cloudwebrtc added 3 commits June 14, 2022 08:51

Hide private delegate declaration.

0e3ee58

Re-generate thumbnail for all sources when call UpdateSourceList.

33e2dbc

fix: Fix window resize issue.

22b00ce

update.

0d02085

cloudwebrtc force-pushed the feat/support-screen-capture-for-mac branch from 2ac2d6e to 0d02085 Compare June 16, 2022 13:52

update.

2baa397

davidzhao approved these changes Jun 16, 2022

View reviewed changes

cloudwebrtc force-pushed the feat/support-screen-capture-for-mac branch from cffc13f to 2baa397 Compare June 17, 2022 04:00

fix for intel mac.

ddbf6ce

cloudwebrtc added 5 commits June 21, 2022 20:37

cleanup.

11b1cb7

update.

78cd48f

Add Window/Screen change listener.

2d22bfe

Add force reload for MediaListUpdate and capture cursor.

316ddb4

fix: Fix the screen freeze caused by the desktop frame not setting th…

ae4dd41

…e timestamp.

cloudwebrtc added 3 commits July 22, 2022 23:18

update.

703ae40

fix issue for screen sharing.

e958e66

Fixed H264 profile-level-id limitation that could not encode desktop …

3969a7b

…frames with resolutions higher than 720p.

cloudwebrtc mentioned this pull request Jul 26, 2022

screen sharing for desktop. livekit/client-sdk-flutter#135

Merged

9 tasks

cloudwebrtc merged commit 4315ec5 into m97_release Aug 1, 2022

cloudwebrtc deleted the feat/support-screen-capture-for-mac branch August 1, 2022 02:26

cloudwebrtc mentioned this pull request Jun 12, 2023

update to m114 #76

Merged

cloudwebrtc mentioned this pull request May 22, 2024

Update to m125. #119

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Mac] feat: Support screen capture for macOS. #24

[Mac] feat: Support screen capture for macOS. #24

cloudwebrtc commented May 8, 2022 •

edited

Loading

cloudwebrtc commented Jun 13, 2022 •

edited

Loading

hiroshihorie left a comment

cloudwebrtc commented Jun 13, 2022

davidliu commented Jun 13, 2022

davidzhao commented Jun 13, 2022

cloudwebrtc commented Jun 16, 2022

hiroshihorie commented Jun 16, 2022

cloudwebrtc commented Jun 16, 2022 •

edited

Loading

davidzhao left a comment

cloudwebrtc commented Jul 28, 2022 •

edited

Loading

[Mac] feat: Support screen capture for macOS. #24

[Mac] feat: Support screen capture for macOS. #24

Conversation

cloudwebrtc commented May 8, 2022 • edited Loading

cloudwebrtc commented Jun 13, 2022 • edited Loading

hiroshihorie left a comment

Choose a reason for hiding this comment

cloudwebrtc commented Jun 13, 2022

davidliu commented Jun 13, 2022

davidzhao commented Jun 13, 2022

cloudwebrtc commented Jun 16, 2022

hiroshihorie commented Jun 16, 2022

cloudwebrtc commented Jun 16, 2022 • edited Loading

davidzhao left a comment

Choose a reason for hiding this comment

cloudwebrtc commented Jul 28, 2022 • edited Loading

cloudwebrtc commented May 8, 2022 •

edited

Loading

cloudwebrtc commented Jun 13, 2022 •

edited

Loading

cloudwebrtc commented Jun 16, 2022 •

edited

Loading

cloudwebrtc commented Jul 28, 2022 •

edited

Loading