On the Windows app SDK using CameraHelper, FrameArrived time execution for a period of time will not be triggered.

[GreenShadeZhang]

Describe the bug

When I use wasdk 1.1.2, the processing of camera frame data, at first the FrameArrived event is normal, but after a period of time, the event does not trigger, I suspect it is was wasdk bug, but I will confirm it here first.
The uwp code is normal
like this
https://github.com/GreenShadeZhang/GreenShade.UWPDemo/tree/master/GreenShade.ML.EmoticonDetection

But when the code is ported to winui, it can only be normal for a while.

https://github.com/GreenShadeZhang/GreenShade.UWPDemo/tree/master/GreenShade.WinUI.EmoticonDetection

Regression

No response

Reproducible in sample app?

• This bug can be reproduced in the sample app.

Steps to reproduce

1. Download the code
2. Run the relevant code

Expected behavior

I want the FrameArrived event of the video stream to work fine on the Windows app SDK so I can do emoticon analysis.

Screenshots

No response

Windows Build Number

• Windows 10 1809 (Build 17763)
• Windows 10 1903 (Build 18362)
• Windows 10 1909 (Build 18363)
• Windows 10 2004 (Build 19041)
• Windows 10 20H2 (Build 19042)
• Windows 10 21H1 (Build 19043)
• Windows 11 21H2 (Build 22000)
• Other (specify)

Other Windows Build number

No response

App minimum and target SDK version

• Windows 10, version 1809 (Build 17763)
• Windows 10, version 1903 (Build 18362)
• Windows 10, version 1909 (Build 18363)
• Windows 10, version 2004 (Build 19041)
• Other (specify)

Other SDK version

No response

Visual Studio Version

2022

Visual Studio Build Number

No response

Device form factor

Desktop

Nuget packages

No response

Additional context

No response

Help us help you

No.

[ghost]

Hello GreenShadeZhang, thank you for opening an issue with us!

I have automatically added a "needs triage" label to help get things started. Our team will analyze and investigate the issue, and escalate it to the relevant team if possible. Other community members may also look into the issue and provide feedback 🙌

[michael-hawker]

@GreenShadeZhang I don't believe any of the Camera/Media stuff is enabled in the WindowsAppSDK yet (at least to the point it's exposed in the same way that we consume in the Toolkit). We've been waiting for them to enable support for these scenarios. I know that's definitely true for the CameraPreview which uses the MediaPlayerElement, but not sure what's supported or not 100% with the underlying MediaFrameReader api the CameraHelper uses.

I know in the docs here and here they're only calling out the CaptureCameraUI from the Windows.Media.Capture namespace, so not sure if this is a missing gap in documentation or their radar/planning.

I'd recommend opening a discussion on the Windows App SDK repo to be sure of what is and isn't supposed to be working at this time.

[GreenShadeZhang]

@GreenShadeZhang I don't believe any of the Camera/Media stuff is enabled in the WindowsAppSDK yet (at least to the point it's exposed in the same way that we consume in the Toolkit). We've been waiting for them to enable support for these scenarios. I know that's definitely true for the CameraPreview which uses the MediaPlayerElement, but not sure what's supported or not 100% with the underlying MediaFrameReader api the CameraHelper uses.

I know in the docs here and here they're only calling out the CaptureCameraUI from the Windows.Media.Capture namespace, so not sure if this is a missing gap in documentation or their radar/planning.

I'd recommend opening a discussion on the Windows App SDK repo to be sure of what is and isn't supposed to be working at this time.

Thank you very much for your answer. About the media playback control and video preview control is not porting too difficult to become a low priority?

[michael-hawker]

Thank you very much for your answer. About the media playback control and video preview control is not porting too difficult to become a low priority?

Hi @GreenShadeZhang, I'm not 100% sure I'm following what you're asking here. The Windows App SDK team handles the underlying APIs we're using, so filing an issue on their repo and asking for clarity on what should be supported is a first step. They may clarify if it's supposed to work now, not supposed to work, or something they're planning to implement in an upcoming release or not.

From there, once we know that, we can figure out on our side in the Toolkit for our next 8.0 release if this is something we can support, assuming it is supported in the platform at that time.

[AlexanderBlackman]

I've had a similar problem, I guess it can wait until the MediaPlayerElement is updated in the SDK.

[GreenShadeZhang]

I've had a similar problem, I guess it can wait until the MediaPlayerElement is updated in the SDK.

Recently I replaced the camera API of UWP with opencvsharp, and the effect is OK is that the CPU occupation is too high, I don't know if you have any good way.

[GreenShadeZhang]

Thank you very much for your answer. About the media playback control and video preview control is not porting too difficult to become a low priority?

Hi @GreenShadeZhang, I'm not 100% sure I'm following what you're asking here. The Windows App SDK team handles the underlying APIs we're using, so filing an issue on their repo and asking for clarity on what should be supported is a first step. They may clarify if it's supposed to work now, not supposed to work, or something they're planning to implement in an upcoming release or not.

From there, once we know that, we can figure out on our side in the Toolkit for our next 8.0 release if this is something we can support, assuming it is supported in the platform at that time.

Recently I found out that the Windows app SDK 1.2 preview version supports MediaPlayerElement, so I feel that 8.0 can test the integrity of this feature.
https://learn.microsoft.com/en-us/windows/apps/windows-app-sdk/preview-channel#winui-3

In addition, I also saw some precautions for camera data processing in the document, and I don't know if I need to pay attention to these things in the windows app SDK.
https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/process-media-frames-with-mediaframereader

[michael-hawker]

Thanks @GreenShadeZhang, our big hope with 8.0 is to maintain everything from a single codebase (vs. the forks/branches we have now). So, ideally we'll be able to make this easier for us to validate the behavior of and test across both UWP and the Windows App SDK.

[GreenShadeZhang]

Thanks @GreenShadeZhang, our big hope with 8.0 is to maintain everything from a single codebase (vs. the forks/branches we have now). So, ideally we'll be able to make this easier for us to validate the behavior of and test across both UWP and the Windows App SDK.

Sounds very good and very exciting.

[GreenShadeZhang]

The latest 1.2 already supports media playback controls, can the latest version of the Community Toolkit test this feature, or wait for the 8.0 release.

[michael-hawker]

@GreenShadeZhang I'm not sure, it's not something we've tested, but I don't think the code was commented out, so assuming the APIs map the same, then it may work?

We weren't able to validate and bring forward our existing winui branch, so we'll be starting to port features over to our new infrastructure that supports both UWP and the Windows App SDK in the near future.

If you do have a chance to try, let us know the outcome. Thanks!

[BreeceW]

I believe the issue here is that CameraHelper is calling VideoMediaFrame.GetVideoFrame without disposing the result. This code is where the issue is occurring.

Disposing the video frame after the event handler is invoked seems to work. I’m not sure if there would be an issue in doing that for API consumers, though.

Additionally, the Reader_FrameArrived should check if there are any subscribers to the CameraHelper.FrameArrived event before getting the video frame, so the common scenario of using CameraPreview, which does not rely on that event, can avoid the overhead.

[GreenShadeZhang]

Can this problem really be solved? I think this function can reduce a lot of code for people who use it for video processing.

@michael-hawker

[GreenShadeZhang]

I believe the issue here is that CameraHelper is calling VideoMediaFrame.GetVideoFrame without disposing the result. This code is where the issue is occurring.

Disposing the video frame after the event handler is invoked seems to work. I’m not sure if there would be an issue in doing that for API consumers, though.

Additionally, the Reader_FrameArrived should check if there are any subscribers to the CameraHelper.FrameArrived event before getting the video frame, so the common scenario of using CameraPreview, which does not rely on that event, can avoid the overhead.

Yesterday I located the cause of the problem. I found that it was still a problem with the release of unmanaged objects. I will test more today to see if I can solve this problem. If so, I will file a PR.

[ThreeSevenths]

Hi Everyone, I am still encountering an issue using the latest RC build, 8.1.240606-rc. My preview freezes after 400-500 frames. From what I can tell from Source Link, I cannot dispose the VideoFrame in my FrameArrived event handler because the getter on FrameEventArgs makes a copy of the original.

Can someone confirm:

1. fix:On the Windows app SDK using CameraHelper, FrameArrived time execution for a period of time will not be triggered. #376 is included in 8.1.240606-rc
2. If the fix is not included, when will it be?
3. If the fix is not included, that the only way for me to get the original VideoFrame from FrameEventArgs would be to reflect out the internal field _videoFrame and dispose of it in my handler
4. If there is another workaround for this issue or if I am doing something wrong?

My Window code looks like this below:

MainWindow.xaml.cs

public sealed partial class MainWindow : Window
{
    private readonly SemaphoreSlim _CameraSemaphore;

    public MainWindow()
    {
        _CameraSemaphore = new SemaphoreSlim(1);

        this.InitializeComponent();

        this.Activated += MainWindow_Activated;
        this.Closed += MainWindow_Closed;
    }

    private void MainWindow_Closed(object sender, WindowEventArgs args)
    {
        this.Closed -= MainWindow_Closed;
        UnloadCamera();
    }

    private void MainWindow_Activated(object sender, WindowActivatedEventArgs args)
    {
        this.Activated -= MainWindow_Activated;
        LoadCamera();
    }

    private async void UnloadCamera()
    {
        CameraPreview.CameraHelper.FrameArrived -= CameraHelper_FrameArrived;
        await CameraPreview.CameraHelper.CleanUpAsync();
        CameraPreview.Stop();
        CameraPreview.PreviewFailed -= CameraPreview_PreviewFailed;
    }

    private async void LoadCamera()
    {
        await _CameraSemaphore.WaitAsync();

        await CameraPreview.StartAsync();
        var result = await CameraPreview.CameraHelper.InitializeAndStartCaptureAsync();
        CameraPreview.PreviewFailed += CameraPreview_PreviewFailed;
        CameraPreview.CameraHelper.FrameArrived += CameraHelper_FrameArrived;

        _CameraSemaphore.Release();

    }

    static volatile int _Count = 0;

    private void CameraHelper_FrameArrived(object sender, CommunityToolkit.WinUI.Helpers.FrameEventArgs e)
    {

        try
        {
            var frame = e.VideoFrame;

            var framenumber = Interlocked.Increment(ref _Count);

            Debug.WriteLine("Frame {0}", framenumber);
        }
        catch(Exception ex)
        {
            Debugger.Break();
        }

    }

    private void myButton_Click(object sender, RoutedEventArgs e)
    {
        myButton.Content = "Clicked";
    }

    private void CameraPreview_PreviewFailed(object sender, CommunityToolkit.WinUI.Controls.PreviewFailedEventArgs e)
    {
        Debug.WriteLine("PreviewFailed");
    }
}

MainWindow.xaml

<Window
    x:Class="ZXIngTest1.MainWindow"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:local="using:ZXIngTest1"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    xmlns:controls="using:CommunityToolkit.WinUI.Controls"
    mc:Ignorable="d">

    <StackPanel Orientation="Vertical" HorizontalAlignment="Center" VerticalAlignment="Center">
        <controls:CameraPreview x:Name="CameraPreview" Margin="10" Height="320" />
        <Button x:Name="myButton" Click="myButton_Click" HorizontalAlignment="Center">Click Me</Button>
    </StackPanel>
</Window>