3.6.0: Sound queue plays with small gaps between clips

[ivan-mogilko]

Context

AGS has AudioClip.PlayQueued function that supposed to queue sounds if they cannot play right away. This may be used to sequence clips, provided there's a restricted number of channels for this audio type, preventing them from starting simultaneously. As such, this is also a sort of workaround for AGS's lack of a proper sound sequencer/mixer script API.

Problem

The queued sounds are regularly played with a small but noticeable gap between them. The reason is that the sound queue is updated along with the rest of the game logic, one per game frame (depends on fps). Therefore it cannot react to the previous playback's ending fast enough.

Before 3.6.0, and before the proper audio control thread was implemented, the audio was polled on the main thread, and maybe that somehow made it more "in sync" with the main game's update, so the queued sound had higher chance to start quick enough (it was still quite random). Now when the playback is processed on its thread it is "disconnected" from the game updates, and some things no longer work same way.

Expected behavior

Users expect the queued sounds to play seamlessly.

Potential workarounds

• Implement some kind of a transition to silence or out of silence at the ending/beginning of the queued tracks;
• Use crossfade setting, that makes the gap less ugly sounding, the transition becomes "softer" (could be a subjective impression though).
• Instead of using PlayQueued, one could instead schedule clips in script, and launch them from repeatedly_execute_always, but test not for the previous song completion, but for it being couple of game frames before completion, to account for the difference in timing.

Implementation problems

One big issue here is that the queue and audio core are not connected. Queue works within the game logic, and needs knowledge of the logical audio channels, where it will find a free one according to clip types and priorities. Audio core knows nothing about the game's channels and other things.
Audio core works on its own "audio thread", sound queue is updated along with a game on the game update thread (aka main thread).
Thus the solution may have to involve one of the following:

1. either queue should also be processed on a separate thread and react fast enough to audio core slots changing their states, or
2. audio core should have callback system to report its state change to the main engine code.

The logical audio channels are represented with SOUNDCLIP array controlled using some internal api.
One important nuance about these channels is, that in order to keep things consistent with script, those must not be modified at any random times, but only in between script callbacks (currently - only once per frame). There's a thorough explanation in the description to the commit ff05f69 that enforced such rule in 3.6.0.
This also means that we cannot modify this array as a reaction to the audio core slot stopping and queue being processed at any random time. OR we'll have to change how this array serves as a "frontend" to channels, by introducing more intermediate objects or something...

EDIT:
Another issue, the audio core is currently reporting sound position as (approximate) playback position of the openal source, not the position of the sound decoder (reader). This means that when the decoder is done reading, the openal may still have samples in buffer to be processed. May this add to the delay between previous and next sound, if the new sound is triggered by checking the position of the previous one?

[ivan-mogilko]

Pushed a somewhat "hacky" fix: e87e3a8

In short, did 2 things:

1. Sync logical channels with the audio subsystem also prior to updating queue and other things (crossfade etc). This fixes a 1 game frame queue delay. This actually has to be done regardless of this specific problem, I think.
2. Additionally, force queued clips to start 1 extra game frame earlier, by testing current playback position of active clips.

I will close this ticket, as the problem is formally resolved. "Cleaner" solution would require larger changes, likely a reimplementation of the sound queue, running on a different thread, and so forth.