Long duration test not working

[yanj-github]

Long duration test playing fine at the beginning but it timed out at around 10 minutes.

[jpiesing]

@yanj-github Your comment in the other issue mentioned "The test is on default timeout at 10 minutes it is fine for most of test but not long duration.". Does that mean you think this is just a timeout issue and not a bug in the test?

[yanj-github]

The test timeout after 100 seconds (sorry I measured time wrong not 10 mins) as it set in creating test.
From following screen shot test created with 100 s timeout.

I guess test creation need to allow more timeout for this specific test. @FritzHeiden ?

[FritzHeiden]

This is indeed a problem. For now I will solve this problem by disabling the timeout for long duration test all together. Do you have any other suggestions?

[yanj-github]

Thanks @FritzHeiden I am not sure what is the impact on having no timeout for a test? I guess user need to stop it manually if something is wrong? I don't think I can think of any alternative solutions.
@jpiesing do you have any comments on this please?

[FritzHeiden]

Thanks @FritzHeiden I am not sure what is the impact on having no timeout for a test? I guess user need to stop it manually if something is wrong? I don't think I can think of any alternative solutions. @jpiesing do you have any comments on this please?

@yanj-github Removing the timeout would make this test semi-automatic. It runs fine when there is no issue, however, if something happens that prevents the video from playing all the way through you would have to manually resume the session. Alternatively we could adjust the timeout according to the test duration. If the video is 2.5h long we may have a timeout of 3h maybe. wdyt?

[yanj-github]

I think having timeout is better.

[FritzHeiden]

I merged some PRs to set the timeout for long duration tests to 3 hours using the test-config.json: #165 #166 cta-wave/dpctf-test-runner#50

This requires a rebuild of the image with --reload-tests and --reload-runner flags.

[yanj-github]

@FritzHeiden when the test is around 100s in time, the test status on result.html shows completed and API results reports timedout. Can it be improved please?

[FritzHeiden]

Thanks @yanj-github I can confirm this issue. To use a different timeout on the server the test would need to be tagged as manual test, so the manual timeout can be used. To tag the test as manual the file name needs to include manual, so this may require some changes in the OF aswell

[yanj-github]

Thanks @yanj-github I can confirm this issue. To use a different timeout on the server the test would need to be tagged as manual test, so the manual timeout can be used. To tag the test as manual the file name needs to include manual, so this may require some changes in the OF aswell

Thanks, OF gets file name from tests.json from test code to map to the file name so wont require the change.

[louaybassbouss]

Thanks @yanj-github I can confirm this issue. To use a different timeout on the server the test would need to be tagged as manual test, so the manual timeout can be used. To tag the test as manual the file name needs to include manual, so this may require some changes in the OF aswell

Thanks, OF gets file name from tests.json from test code to map to the file name so wont require the change.

Thanks @yanj-github we will then rename the long time playback test to include manual in the filename, and change the timeout of manual tests to a time above 2h (like 2h + 5min or so). wdyt?

[jpiesing]

2024:05-14: Issue can be closed when the test is renamed.

[yanj-github]

2024:05-14: Issue can be closed when the test is renamed.

Thanks Jon, can it stay open after the fix on TR until I run it with OF please?

[yanj-github]

@FritzHeiden and @louaybassbouss the audio content is missing for long duration test, OF terminating because required audio configuration is missing. I don't think we have right audio stream yet. If it is an easy update can we have a placeholder short audio until right audio stream is available please? Maybe we can just use same audio as 9.2 test.

[louaybassbouss]

@FritzHeiden and @louaybassbouss the audio content is missing for long duration test, OF terminating because required audio configuration is missing. I don't think we have right audio stream yet. If it is an easy update can we have a placeholder short audio until right audio stream is available please? Maybe we can just use same audio as 9.2 test.

What do you mean by short audio. The duration of the audio should fit with the duration of the video. In other words, we need long duration audio as well.

[jpiesing]

The 'tLD' content is at https://dash.akamaized.net/WAVE/vectors/cfhd_sets/12.5_25_50/tLD/2023-09-01/stream.mpd. I have downloaded the MPD and confirm it only contains video and no audio.

I think the relevant issue in the test content repo is cta-wave/Test-Content#39 (which was closed) which in turn refers to cta-wave/mezzanine#46.

That issue is closed with a comment from Nicholas Frame that the LD mezzanine has been updated to include the looped PN audio.

I think therefore that tLD needs re-encoding to have the audio & I've re-opened the issue.

[yanj-github]

@FritzHeiden and @louaybassbouss the audio content is missing for long duration test, OF terminating because required audio configuration is missing. I don't think we have right audio stream yet. If it is an easy update can we have a placeholder short audio until right audio stream is available please? Maybe we can just use same audio as 9.2 test.

What do you mean by short audio. The duration of the audio should fit with the duration of the video. In other words, we need long duration audio as well.

Hi @louaybassbouss I meant to have a temporally place holder audio until we have right audio stream available. So that OF can do the video testing. At the moment missing audio OF will terminating with error. If it is not an easy change to TR, or cause confusion, no worries I will comment out audio testing for long duration check so it can do video testing only. Thanks.

[FritzHeiden]

Renamed the test #181 and updated internal manual timeout to a high value cta-wave/dpctf-deploy#92

Pulling the dpctf-deploy repository before rebuilding is required for this to work

If it is not an easy change to TR, or cause confusion, no worries I will comment out audio testing for long duration check so it can do video testing only. Thanks.

I'd rather have this temporary fix in the OF as I don't know if having a shorter audio track in the player may cause unexpected behavior

[yanj-github]

Renamed the test #181 and updated internal manual timeout to a high value cta-wave/dpctf-deploy#92

Pulling the dpctf-deploy repository before rebuilding is required for this to work

If it is not an easy change to TR, or cause confusion, no worries I will comment out audio testing for long duration check so it can do video testing only. Thanks.

I'd rather have this temporary fix in the OF as I don't know if having a shorter audio track in the player may cause unexpected behavior

Thanks @FritzHeiden understand now. In this case it would be easier to turn off the audio observation for this test on OF end. I will make changes to OF temporally comment it out. So video only observation can be run.

[yanj-github]

I'd like to share video observation result here. The playback pauses a lot and it shows waiting when it does. This makes duration fail and current time match. This maybe a device specific issue waiting to get more test result on different devices.

• [OF] Every video frame S[k,s] shall be rendered and the video frames shall be rendered in increasing presentation time order.
  PASS: Total of missing frame count is 0.
• [OF] Video start-up delay: The start-up delay should be sufficiently low.
  PASS: Maximum permitted startup delay is 1000ms. The presentation start up delay is 2.5833ms.
• [OF] Video: The playback duration shall match the duration of the CMAF Track
  FAIL: Playback duration 15907504.87ms does not match expected duration 7200000.0ms +/- tolerance of 50ms. Detected duration is different by 8707504.87ms.
• [OF] Video: The presented sample shall match the one reported by the currentTime value within the tolerance.
  FAIL: Total failure count is 103.