Play audio in cutscenes #288

madebr · 2023-02-15T00:56:06Z

Update miniaudio to 0.11.11
Play sounds when playing the smacker cutscenes
Because I noticed a lag between video and audio, I created a mini player at tools/minismackerplay.c using libsmacker, miniaudio and SDL2. This shows the overhead/timing/sleeps/... of dethrace should be improved

dethrace-labs

LGTM

madebr

@dethrace-labs
Can you please review and/or take over? Thanks.

madebr · 2023-04-26T18:15:43Z

src/smackw32/smackw32.c

+    /* FIXME: get rid of bodge term */
+    if (now < user->smack_last_frame_time + smack->MSPerFrame - 5) {


I did this to avoid stuttering.
When sleeping too long, the next audio frame is not ready soon enough.

madebr · 2023-04-26T18:16:25Z

src/smackw32/smackw32.c

 int SmackSoundUseDirectSound(void* dd) {
-    // TODO: do some miniaudio init
+    // FIXME: is this a more appropriate place to init miniaudio? What is type of argument?


Because the arguent of SmackSoundUseDirectSound was NULL, I did the audio initialization in SmackOpen.

All good. The NULL argument is an optional pointer to a DIRECTSOUND instance, (NULL apparently means "create a new one")

https://github.com/TheNitesWhoSay/lawine/blob/master/lawine/misc/smack.c#L34

madebr · 2023-04-26T18:17:14Z

src/smackw32/include/smackw32/smackw32.h

@@ -43,7 +43,7 @@ typedef struct SmackTag {
    unsigned long addr32;

    // added by dethrace
-    void* smk_handle;
+    void* user;


I moved all user data to a private struct, that is also allocated.

madebr · 2023-04-26T18:17:54Z

src/smackw32/smackw32.c


-uint32_t smack_last_frame_time;
+extern ma_engine miniaudio_engine;


This re-uses miniaudio_engine, so the separation between smackw32 and S3 is not perfect.

dethrace-labs · 2023-04-26T23:12:53Z

@dethrace-labs Can you please review and/or take over? Thanks.

Sure, thanks for bringing it up to latest main!

madebr · 2023-04-26T23:17:28Z

tools/minismackerplay.c

@@ -0,0 +1,290 @@
+#define SDL_MAIN_HANDLED


In the long run, it would be nice to convert this tool to using smackw32 + brender.
Perhaps worth a fixme :)

dethrace-labs · 2023-05-01T20:13:00Z

I still experience stuttering with this. I started to experiment by trying to feed the audio 1 frame in front of the video so we always have the next audio page to consume before we run out, but its not quite working yet

PierreMarieBaty · 2023-12-16T18:29:54Z

Hello @dethrace-labs

Congratulations for this impressive work. I rewrote your smackw32 files to add sound playback support to videos using the supplied miniaudio backend. I started with @madebr's code, figured out the problem (sort of) with sound cracklings/stuttering, and moved it to smackw32 where I think it belongs.

There were three problems. One of them was that your frame duration measurement was incorrect. The time measurement shall be done exactly once per loop, not twice (begin / end). Because of this little mistake, the video playback speed wasn't exactly on par with the audio consumption rate, leading to frequent sound buffer starvation, causing arbitrary seek/restarts producing audible pops. The second problem, as you guessed, was that more than one audio page needs to be precached in miniaudio's linked list to avoid sound starvation at the end of the first page. The third problem, once the others were solved, appeared in so that even playing the videos with audio without any stuttering and both audio and video playing at their designated speeds, a time shift could be observed between audio and video that turned out to be simply proportional to the video length. It was very noticeable in the intro video (the longest), when the roadsign shortly spins at the end. Around 25 pages of audio needed to be precached (and the sound playback start delayed by this amount) for audio and video to become accurately synced. On the other hand, on shorter videos only 1 or 2 audio pages needed to be precached to achieve good sync. It's as if the player code was supposed to precache a certain amount of audio pages that's directly proportional to the length of the video. By trial and error I figured out that the value (total number of frames / 100) would give the exact number of pages to precache for each video. Don't ask me why it works this way... perhaps it's by design ?

As I have no write access on this repo, here are the files if you want to review them. The only files changed are smackw32.c and smackw32.h. There are a few comments and I tried to adopt your coding style. I hope you find this contribution useful: https://www.pmbaty.com/smackw32.zip

madebr · 2023-12-17T11:55:35Z

Feel free to create a new PR with your code. I will happily close this PR.

Or you can create a PR on my fork to the cutscenes-with-audio branch that I can merge into this pr.

dethrace-labs · 2023-12-18T20:09:05Z

Thanks @PierreMarieBaty for your analysis and code changes! As @madebr said, are you able to send the changes as a pull request?

PierreMarieBaty · 2023-12-19T09:41:51Z

I’m sorry, I don’t use git - except when I’m forced to. My SCM of choice is SVN (don’t feed the troll) :-)

Co-Authored-By: PierreMarie Baty <PierreMarieBaty@users.noreply.github.com>

madebr · 2023-12-19T17:40:56Z

I copied smackw32.c from the zip archive, but I still hear the audio too soon.

PierreMarieBaty · 2023-12-20T11:58:56Z

Interesting. Which SMK file did you test, what's your system? I tested on Win7/x64 and macOS with a French version of the Carmageddon CD. The results looked good to me. How noticeable is it? I'd like to be able to reproduce the problem. But as I said, this precache delay was mostly guessed out and discovered by trial and error. It is certainly possible that a better mechanism exists to trigger audio and video start at the right time, that's yet to be discovered.

madebr · 2023-12-21T18:46:01Z

I run a x64 Fedora 38 Linux system.

The video I'm seeing time synchronization issues with is GARAGE1.SMK.
That is the video that plays when starting a new game.
The video features a driver pressing some buttons, giving some synchronization references.

When I compare the video played through dethrace and through minismackerplay, the first one is not synchronized.

PierreMarieBaty · 2023-12-22T15:38:17Z

Okay. I compiled your minismackerplay (should have started with that), saw that it does no caching and optional conversion, and could see that all videos seem to play correctly with it. Which means the minismackerplay code must do it right. So I added that optional check for whether resampling is needed in smackw32.c, disabled soundpage caching, and that seemed to work just as well : no sound pops, no sound starve (except when e.g. I resize the game window and the SDL loop exceptionally exceeds the frame duration - an arbitrary sync is still needed in this case. BTW minismackerplay has the same problem). Anyway, this tends to indicate that sample conversion in miniaudio from and to the same sample rates (?) produce output that's shorter than the input. It didn't fix the sound delay in the intro video MIX_INTR.SMK though, so I checked further. I saw that dethrace uses SDL_GetTicks() and minismackerplay uses SDL_GetPerformanceCounter(). Having minismackerplay use SDL_GetTicks() too didn't help, so I made them both display the sound page sizes, and I could spot the discrepancy. minismackerplay (right): frame #0 sound converter framesIn 23154 framesOut 46308 frame #1 sound converter framesIn 1102 framesOut 2204 frame #2 sound converter framesIn 1102 framesOut 2204 frame #3 sound converter framesIn 1102 framesOut 2204 frame #4 sound converter framesIn 1102 framesOut 2204 ... dethrace (wrong): frame #0 sound converter framesIn 1102 framesOut 2204 frame #1 sound converter framesIn 1102 framesOut 2204 frame #2 sound converter framesIn 1102 framesOut 2204 frame #3 sound converter framesIn 1102 framesOut 2204 frame #4 sound converter framesIn 1102 framesOut 2204 ... Which seemed to prove 2 things: 1. When things are done right, the first audio page is much bigger than the other ones, and thus lasts much longer. 2. The sound "precaching" effect is simply necessary because dethrace skips that frame and directly proceeds to audio frame #1 as if it was frame #0. After some more investigation, the reason for this turned out because smk_enable_audio() was called *after* smk_first() in dethrace, whereas in libsmackerplay smk_get_audio() is called before it. By reordering things in SmackOpen() I could solve the problem. There was no need to precache anything after all. As a sidenote, having a big first audio page sounds like a sensible conscious choice by the RAD SMK encoder to ensure the sound won't starve at video init on low-end machines.

PierreMarieBaty · 2023-12-22T16:09:07Z

I updated my smackw32 files (.c and .h). Same URL: https://www.pmbaty.com/smackw32.zip Let me know if it sounds good this time.

dethrace-labs · 2024-01-15T23:11:19Z

Thank you! I love it when a bunch of complicated work goes in and the end result is a simple small change :). I'll make a PR with the changes.

dethrace-labs · 2024-01-16T21:04:31Z

@PierreMarieBaty I can't access the zip file - could you check your website, or share the zip somewhere else?

madebr · 2024-01-16T22:07:57Z

Weird, I can still download it.
This is the zip from #288 (comment):
https://dropmeafile.com/#e6f48d3f01

PierreMarieBaty · 2024-01-17T07:19:44Z

My server is located in Russia, that could be the reason. Your ISP might have a firewall or something blocking russian hosts maybe?

…

Message ID: ***@***.***>

dethrace-labs · 2024-05-28T03:58:47Z

Closing this and will reopen shortly

madebr force-pushed the cutscenes-with-audio branch from c84fb24 to de53b06 Compare February 15, 2023 01:01

madebr added 4 commits February 20, 2023 01:34

Update miniaudio to 0.11.11

6d58f1a

Add cutscene audio

71697ab

Add minismackerplay to play smack videos from cli

d11923f

Disable cutscene audio when sound is disabled

2289311

madebr force-pushed the cutscenes-with-audio branch from de53b06 to 2289311 Compare February 20, 2023 00:34

dethrace-labs approved these changes Feb 20, 2023

View reviewed changes

madebr added 6 commits February 22, 2023 14:54

Improve stdout logging of minismackerplay

6c76707

fix 'audioformat_ma may be used uninitialized in this function' error

1ca2f64

fix macosx warnings

98de83a

Remove unused includes

aaf6dda

Merge remote-tracking branch 'origin/main' into cutscenes-with-audio

a0e75d2

Fix ci macos warning

56179e6

madebr requested a review from dethrace-labs April 26, 2023 18:13

madebr commented Apr 26, 2023

View reviewed changes

madebr added 2 commits May 4, 2023 23:11

Update miniaudio to 0.11.15

72ebf15

Merge remote-tracking branch 'origin/main' into cutscenes-with-audio

cf6ac42

madebr and others added 2 commits December 19, 2023 18:28

Merge remote-tracking branch 'origin/main' into cutscenes-with-audio

e8842eb

Update smackw32

c1c2765

Co-Authored-By: PierreMarie Baty <PierreMarieBaty@users.noreply.github.com>

dethrace-labs closed this May 28, 2024

dethrace-labs mentioned this pull request Jul 9, 2024

Audio in cutscenes #390

Merged

madebr deleted the cutscenes-with-audio branch July 19, 2024 20:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Play audio in cutscenes #288

Play audio in cutscenes #288

madebr commented Feb 15, 2023

dethrace-labs left a comment

madebr left a comment

madebr Apr 26, 2023

madebr Apr 26, 2023

dethrace-labs May 1, 2023

madebr Apr 26, 2023

madebr Apr 26, 2023

dethrace-labs commented Apr 26, 2023

madebr Apr 26, 2023 •

edited

Loading

dethrace-labs commented May 1, 2023

PierreMarieBaty commented Dec 16, 2023

madebr commented Dec 17, 2023 •

edited

Loading

dethrace-labs commented Dec 18, 2023

PierreMarieBaty commented Dec 19, 2023 via email

madebr commented Dec 19, 2023

PierreMarieBaty commented Dec 20, 2023 via email

madebr commented Dec 21, 2023

PierreMarieBaty commented Dec 22, 2023 via email

PierreMarieBaty commented Dec 22, 2023 via email

dethrace-labs commented Jan 15, 2024

dethrace-labs commented Jan 16, 2024

madebr commented Jan 16, 2024 •

edited

Loading

PierreMarieBaty commented Jan 17, 2024 via email

dethrace-labs commented May 28, 2024

		/* FIXME: get rid of bodge term */
		if (now < user->smack_last_frame_time + smack->MSPerFrame - 5) {


		uint32_t smack_last_frame_time;
		extern ma_engine miniaudio_engine;

Play audio in cutscenes #288

Play audio in cutscenes #288

Conversation

madebr commented Feb 15, 2023

dethrace-labs left a comment

Choose a reason for hiding this comment

madebr left a comment

Choose a reason for hiding this comment

madebr Apr 26, 2023

Choose a reason for hiding this comment

madebr Apr 26, 2023

Choose a reason for hiding this comment

dethrace-labs May 1, 2023

Choose a reason for hiding this comment

madebr Apr 26, 2023

Choose a reason for hiding this comment

madebr Apr 26, 2023

Choose a reason for hiding this comment

dethrace-labs commented Apr 26, 2023

madebr Apr 26, 2023 • edited Loading

Choose a reason for hiding this comment

dethrace-labs commented May 1, 2023

PierreMarieBaty commented Dec 16, 2023

madebr commented Dec 17, 2023 • edited Loading

dethrace-labs commented Dec 18, 2023

PierreMarieBaty commented Dec 19, 2023 via email

madebr commented Dec 19, 2023

PierreMarieBaty commented Dec 20, 2023 via email

madebr commented Dec 21, 2023

PierreMarieBaty commented Dec 22, 2023 via email

PierreMarieBaty commented Dec 22, 2023 via email

dethrace-labs commented Jan 15, 2024

dethrace-labs commented Jan 16, 2024

madebr commented Jan 16, 2024 • edited Loading

PierreMarieBaty commented Jan 17, 2024 via email

dethrace-labs commented May 28, 2024

madebr Apr 26, 2023 •

edited

Loading

madebr commented Dec 17, 2023 •

edited

Loading

madebr commented Jan 16, 2024 •

edited

Loading