Feature Request: Midi audio rendering

[kphillisjr]

This feature request is about adding one of the oldest known general purpose audio formats. This can be split into 3 main features needed to accomplish the whole spec.

• Load Standard Midi Files - The standard midi file ( .mid ) is fully standardized and has not seen many changes in over 20 years.
• standard SoundFont Instrument File loading- This is the file format that is essentially a bunch of wav files all in a single file.
• Rendering midi file using Soundfont - This is applying a few filters ( low pass, high pass, etc), and a frequency shift to the midi stream.
• Improved soundfont file loading - Some sound font files compress the audio using freely available audio codecs like ogg, mp3, flac and possibly others.

[acoto87]

Maybe this is what you are looking for: https://github.com/schellingb/TinySoundFont, it's a library just like this one but to render SoundFont files. It's really good. I'm using it in together with MiniAudio and they work well.

[kritzikratzi]

midi is a sequenced audio format, miniaudio is about sampled sound. you need a midisynth, and plug that into miniaudio.

[mackron]

Thanks for the request! Lets not prematurely write this idea off - this isn't the first time I've had a request for this. Just to be clear, you're only concerned about the playback of MIDI streams? You just want to load a MIDI file, attach a SoundFont and then hear something out of the speakers? There's already the notion of a decoder in miniaudio (ma_decoder) which achieves the same goal for WAV, FLAC, MP3 and Vorbis - loading a sound source and converting it to uncompressed PCM for playback.

That said, however, the ma_decoder API is actually just a wrapper around libraries I maintain in a separate repository (except Vorbis - that's not my project). If any kind of MIDI synthesis was implemented, it wouldn't be in miniaudio.h - it would follow the same paradigm currently used for WAV, FLAC, MP3 and Vorbis which is implement the decoding in a separate library, and then have miniaudio take advantage of it if it's available (if the header of the external library is included before the implementation of miniaudio.h). What I think could be reasonable is to maybe (no guarantees) add a backend to ma_decoder for MIDI streams which uses an appropriate external library (needs to be C, single file, public domain and simple API). If I were to do this, which I'm not guaranteeing, it would only be simple - complex MIDI stuff is better left to dedicated libraries.

The bad news: if this were to go ahead, it won't be happening any time soon as I personally don't really have much interest in MIDI and have a seemingly never ending list of other things to do... We're talking years before I'll even think about this, if at all. I can't speak for TinySoundFont - I've never used it, but maybe worth looking in to?

[acoto87]

I'm sorry, I'm not trying to be dismissive, the request is a valid and a nice-to-have one. I'm just trying to point out a solution I've for this.

You can see it here https://youtu.be/FJNfw00HUgM.

I'm using Miniaudio + TinySoundFont in all the music you are hearing in that video, which is in MIDI format. It works pretty well and I'm sure a better job can be done with more robust code. I like the idea of having a decoder (like you already have for WAV or MP3) that can load and play MIDI files.

Here is the code if you are interested: https://github.com/acoto87/war1/blob/05b261aac3e3a4ee4dd3d27ad99098cd07b2cd26/src/war_audio.c#L178

[mackron]

Yeah there's a definite market for MIDI, and having basic support for MIDI via ma_decoder with a supplied SoundFont I think would be useful, and I think would meet the requirements of @kphillisjr and many others. Still, it's important to be mindful of the goals of miniaudio which is simply delivering PCM audio to speakers, and receiving PCM audio from microphones. It'll never be a full-featured MIDI library.

What will really accelerate integration of basic MIDI playback will be a high quality library I can plug in as a backend, just like I do with dr_wav, dr_flac, etc.

[frink]

@mackron I like the idea of wrapping TinySoundFont and even OpenSFZ with ma_decoder... However, I'm not sure how you'd do this since you would obviously need to load possibly several SoundFonts - one for each channel - and other various settings. Plus, as soon as you do this there will be those asking if you can do MIDI filtering, processing, and routing. It may be better to provide this as an example rather than in full inclusion in the library proper. Nevertheless, I see a lot of value in streaming MIDI redering.

I guess the underlying question is: How deep should we go with MIDI if we start to support it?

Since @acoto87 has already done work with the two libraries of these it might be worth asking for a proof-of-concept PR to add a few files in the research/ folder putting the wrapper in ma_midiplay.h and example usage in ma_midiplay.c like other concepts that @mackron is working on there...

What do you think?

[mackron]

Nobody should put any work into any kind of PR right now as I haven't yet decided on whether or not I'll even support MIDI directly. That's a major project that, if I was to support it, would be years out from now.

[frink]

I've been pondering MIDI vs MiniAudio in the back of my mind for the past week and I have to agree that rendering probably should NOT be done directly inside of MiniAudio. There are too many backend possibilities (FluidSynth, TinySoundFont, OpenSFZ, etc.) and favoring one over the others hurts this library in a big way. There really needs to be more clarity on the best solution for MIDI...

I'm wonder if the "right way" to handle things is to do MIDI routing in the effects chain and then providing "effects wrappers" to render MIDI from various backends. (essentially like modern DAW plugins) That seems more flexible than being married to any backend and could provide an entire solution for software synthesis.... (Not sure that's where you want to go... But...)

MIDI needs a different ring buffer style because the MIDI clock and signal are very different from audio frames. Also MIDI is a vast specification and is getting more confusing! (Have you seen MIDI 2.0?) So that may be a good arguments for avoiding MIDI and letting another library handle things. Ultimately, I think a lot of this has to do with the final design of the effects chain system and how it handles side chains...

Thus, we really need to wait for what @mackron does with 1.0 before integrating MIDI begins to make sense...

[meshula]

My feeling is that you're right ~

I'm wonder if the "right way" to handle things is to do MIDI routing in the effects chain and then providing "effects wrappers" to render MIDI from various backends.

I'm exploring that right now in my LabSound/LabMidi libraries, and am finding that a design for a flexible solution involves a lot of side work in voice allocations, priorities, channel interpretations, and huge inconsistencies between authored Midi files; it's not straight forward. I took a little side trip down Tracker emulation, just to clear the palette and get some nice sounding results without melting my brain :)

[mackron]

I've considered this further and have decided that I'm not going to be adding built-in support for MIDI. It's just that it can be done as a layer on top of miniaudio, and an external implementation can implement it as a ma_data_source if they want integration with the main library. But not only that, miniaudio is already getting too big so I need to be selective with what I add to the main file (integration of the high level stuff is going to take it to ~75K lines, maybe more, and then by the time dr_vorbis and dr_opus are integrated it'll be over 100K).