Use local MP3 files for audio instead of soundfont/player

[Afra-r]

Hi,

I was wondering, if I wanted to replace each track's audio (as well as the full score audio) with my own MP3s, what would be the best way to implement this? I want to experiment with having my own recordings play instead of the player audio, but the codebase is so huge that I don't even know where to start. I couldn't find any other discussions pertaining to this.

[Danielku15]

Hey there.

We have this as a bigger feature on the roadmap but it was not the focus until now. It has been discussed some times in the past:

• Play score without soundfont #348
• How do you play the song at a specific time?  #372

This is likely not a small or easy task but certainly a very interesting and rewarding one. 😁

I was wondering, if I wanted to replace each track's audio (as well as the full score audio) with my own MP3s, what would be the best way to implement this?

The most challenging part that cannot be shipped out of the box (easily), is the synchronization between the external audio and the playback cursor relying on the actual audio information. The most promising approaches which alphaTab will try to follow in future is:

1. Allow the user (developer) to provide a list of markers in addition to the normal input file.
   • Each marker consists of 2 main parts: the time position of the external audio and the midi tick position of the song.
2. AlphaTab internally would differenciate between 3 player engines:
   • the current synthesizer
   • an external audio, likely supplied via <audio /> tag or maybe even a WebAudio context which can be hooked up.
   • an external video, likely supplied via <video /> tag.
3. Each engine would provide an interface to control the essentials of play/pause and the current position.(likely a subset of https://github.com/CoderLine/alphaTab/blob/develop/src/synth/IAlphaSynth.ts).

For the audio and video engines there is then a need to continuously keep track of the current time position and calculate according to the "markers" linearly the midi tick poision. With this the foundation is set for synchronizing audio and video correctly.

A advance service built with alphaTab might do preprocessing for each file on the backend. e.g. via an audio spectrum analysis. Then it could try to make a first best guess for the markers and store them along the file offered via your App/Website. From AI to comparing the spectrums of the alphaTab generated audio vs. video/audio or simply creating them manually everything is likely possible to generate those markers.

I want to experiment with having my own recordings play instead of the player audio, but the codebase is so huge that I don't even know where to start

For the implementation following aspects might be relevant:

1. AlphaTab is not only available as web component but also for .net and Android (about to be released). Any implementation we do should allow using any "external" technology. Not automatically of course, but the better the abstraction, the easier any external audio system could be plugged.
2. Building a marker based mapping system which can translate real audio back to the midi tick positions. (and vice versa)
3. Optional: Allow more intermediate cursor placement according to a given time/tick position (not only snapping to beats)
4. Build a separate audio engine taking care of the audio player synchronziation bits.
5. Unify the internal APIs between the different engines and expose an easy to use interface for devs to integrate their audio with alphaTab.

The key parts about the player and UI interaction can all be found in the AlphaTabApiBase. The current player interface is hooked up in the setupPlayer. The cursor update is basically just a repeating call to cursorUpdateTick.

The reverse parts are similar. When a user clicks on notes we tell the player to seek to a certain tick position. This is happening around the the setupClickHandling.

For a very early hands-on to fiddle around you would likely follow this development path:

1. Change the created IAlphaSynth instance https://github.com/CoderLine/alphaTab/blob/develop/src/platform/javascript/BrowserUiFacade.ts#L501
2. Try to create an own empty IAlphaSynth implementation with implementations for all methods and events. Many features might be no-ops at this point.
3. Step-by-step connect the player state (play/pause/stop/volume) parts to the underlying audio.
4. Build the positionChanged logic which needs to translate the times correctly. A simple version might again do this without complex markers.
5. Add the midi event functionality by plugging the midi sequencer to the new implementation.

I hope this gets you a bit kickstarted to fiddle around. You don't need to care about any rendering bits. Anything of interest should be within the AlphaTabApiBase and the synth folder.

You might also want to check out how Guitar Pro has solved this topic with their "audio track" feature.
https://www.youtube.com/watch?v=0VI3Nf2_MPY

[Afra-r]

Thank you so much for the incredibly detailed response! Excited to give this a shot, I really appreciate all the effort you've put into this.