Add multi channels support for audio driver.

[jjceresa]

This PR addresses #665

1. Add new functions for multi channels support: fluid_synth_write_float_channels(), fluid_synth_write_s16_channels()
2. dsound and waveout driver make use of this support. tested on 2 audio devices:
   • creative SB Live! (6 channels).
   • Realtek: ALC889A (8 channels).

[derselbst]

Before going down into a detailed review, I keep thinking about whether fluid_synth_write_float_channels and fluid_synth_write_s16_channels is the right way to go. Perhaps we should better use fluid_synth_process() and manually translate its planar buffers into interleaved PCM. The things I have in mind:

1. fluid_synth_process is ready for surround audio.
2. It should be possible to remove the extra mixing currently done in fluid_synth_process by passing the user-provided buffers directly into rvoice_mixer.
3. The effects are always written to the first stereo buffer.
4. 200 new lines of code in fluid_synth.c vs. a functionality that (I think) we already have.
5. Having a possible "planar to interleaved PCM translation with optional dithering" might turn out as a functionality that we could add to our public API if requested; whereas I'd be cautious adding fluid_synth_write_float_channels or fluid_synth_write_s16_channels adding to the public API. (More rendering functions make it more confusing, IMO.)

For point 2. it would be helpful if we don't have more "generic" audio rendering interfaces that we have to care about.

For point 3. I've seen that you #668 would address this. However, I have always thought of this fluid_rvoice_mixer_set_mix_fx() more as a hack that we better should get rid of in the future. And IMO fluid_synth_process with its flexible buffer setup allows us to do this.

So, I'm not strongly against these two functions, I'm just thinking of alternatives. And my proposal would be to use fluid_synth_process and while performing the planar to interleaved PCM conversion, also perform the optional dithering. If performance turns out to be a bottleneck, we can still fall back to your provided solution.

What do you think?

[jjceresa]

Before going down into a detailed review, I keep thinking about whether fluid_synth_write_float_channels and fluid_synth_write_s16_channels is the right way to go. Perhaps we should better use fluid_synth_process() and manually translate its planar buffers into interleaved PCM.

Actually fluid_synth_write_float(), fluid_synth_write_float() or similar fluid_synth_write_xxxx_channels() can be called by all audio drivers. These function accept a flexible buffer organisation (interleaved/non interleaved). This is required by the host audio driver that dictates how the buffers must be written (interleaved/not interleaved) by these functions. (for example alsa driver require interleaved for PCM and non interleaved for float sample). So, I see these functions as the simpler appropriate direct interface between the mixer and any host device driver to final load speakers.

point 1,2

I see fluid_synth_process() as an appropriate generic interface between the mixer and any application that intends to do more processing. This interface exposes the main buffer and fx buffer. So the application can do what it wants with these buffers (map, process, mix).
fluid_synth_process() is probably the only one rendering API that an application should keep in mind.

whereas I'd be cautious adding fluid_synth_write_float_channels or fluid_synth_write_s16_channels adding to the public API. (More rendering functions make it more confusing, IMO.)

Yes, and for now I don't think that fluid_synth_write_xxxx_channels() should be public API (IMO they are only useful for driver and should stay private).

Point 3 and PR 668

The intends was to propose an API that allows to

• map any MIDI channel to any dry buffer.
• map any MIDI channel to any fx unit input.
• map any fx output to any dry buffer.

The motivation is to get (for example):

• distinct MIDI instrument played on distinct stereo speakers.
• distinct MIDI instrument having its own fx unit.
• instrument fx output not necessarily mixed on the same speakers that the one used for this instrument dry audio.

All these feature are possible if the mixer keeps its fx mix mode capability.
Please see attached figure below.

200 new lines of code in fluid_synth.c vs. a functionality that (I think) we already have.

fluid_synth_write_xxxx_channels() each have 20 lines of required new code.
fluid_synth_write_s16(), fluid_synth_write_float() each require 4 lines of new code because
they call respective fluid_synth_write_xxxx_channels() .
Total: 50 lines of new code. 150 lines of comments !. ;)

[jjceresa]

I have added the missing document in branch #668

[jjceresa]

Thinking about points 2 and 5

2 It should be possible to remove the extra mixing currently done in fluid_synth_process by passing the user-provided buffers directly into rvoice_mixer

Ok, that means that:

• the mixer (fluid_rvoice_buffers_mix()) should be able to put the dry audio directly in the caller-provided dry buffers.
• similarly fluid_rvoice_mixer_process_fx() should be able to put fx audio directly in the caller-provided fx buffers.
  That means that before calling fluid_synth_render_blocks(), the mixer should have knowledge of these caller-provided buffers.
  2.1) So, from inside fluid_synth_process() an "internal API" is required to set the mixer with theses buffers. Up to the caller of 'fluid_synth_process()' to decide if the fx buffers should be mixed in the same buffer than dry buffer.

5-Having a possible "planar to interleaved PCM translation with optional dithering" might turn out as a functionality that we could add to our public API if requested.

5.1) assuming fluid_synth_process() is called by an audio driver, it seems that this API functionality is required because before calling fluid_synth_process(), the mixer must know how the buffer (possibly provided by the host device driver) should be written:

• (interleaved/not interleaved),
• float or PCM (with/without dithering).
• another option could be to request the mixer to force the mixing of fx audio into the dry buffer. This is useful for audio driver that intends fx mixed with dry audio. This avoid to provide and prepare fx buffer each time fluid_synth_process() is called.

That sounds as a mixer enhancement as skipping actual internal mixer buffers should be a bit faster.
Also this should simplify adding support for audio driver with multiple output.

All this means also that fluid_synth_write_float() and fluid_synth_write_s16() should be updated and may be
part of functionalities inside should be moved into the mixer.

Both these "mixer API" functionalities (2.1) and (5.1) need to be designed carefully.
Now I have talked to much, I need to drink something.

[derselbst]

Ok, I understand your points.

It means that internally, we are not using our existing rendering interfaces, we rather make old-existing ones more generic. At the same time, there seems to be a need for an audio rendering interface that supports interleaved rendering. Actually, I do have the same need in my app, currently with manual planar to interleaved translation though.

On the other hand and without major changes to rvoice_mixer, my point 2. is only possible with an interface like fluid_synth_process ...It's a dilemma...

I think that for now, I'll take your change. And if I ever get to implement my point 2. I'll also consolidate the rendering interfaces.

I'll give you a more detailed review soon. And we need to find out why the unit tests are failing.

[jjceresa]

At the same time, there seems to be a need for an audio rendering interface that supports interleaved rendering.
Yes this audio rendering interface should support both interleaved (or non-interleaved) rendering.

And we need to find out why the unit tests are failing.

Hope that commit e9a9d35 will fix that ?

[derselbst]

Ok, seems to work now. Thanks!