Best way to use FluidSynth with an external MIDI router

[jimhen3ry]

The Miditzer, which is a pipe organ simulator (www.miditzer.org), has included a copy of FluidSynth as a default soft synthesizer for over 15 years. During that time FluidSynth has undergone major enhancements, the Miditzer not so much. We are now in the process of doing a major overhaul to the Miditzer trying to bring it up to date, or at least not too far behind the times.

The Miditzer is essentially an elaborate MIDI router with controls that allow the performer to adjust the routing in real time. Up to this point, the Miditzer has collected real time MIDI inputs, mouse input, computer keyboard input, and FluidSynth MIDI player input and merged them. It then does the routing functions and emits MIDI output and/or sends FluidSynth MIDI Channel Messages.

15 years ago, when computers were almost all single core and parallelism was minimal to non-existent, this worked fine. As we update the Miditzer to a modern 64 bit programs, Windows only for the time being, my ears are telling me that the timing is no longer being maintained with sufficient accuracy. As I study the FluidSynth API, my understanding is that we should be using FluidSynth MIDI Events, which I understand to be timestamped, to keep the relative timing of MIDI events more consistent. Am I on the right track so far?

The first question is does this mean that we should be sending Miditzer output to the FluidSynth MIDI Sequencer? Would the high level data flow for the MIDI Player be:
[FS MIDI Player] --> [Miditzer Router] --> [FS Sequencer]

The next question is can we use FluidSynth to receive real time MIDI input and provide timestamped MIDI events so that the MIDI Player and real time MIDI input will have consistent time stamps?

The final question for now is whether there is a way of applying a synchronous time stamp to mouse input and computer key input to generate time stamped MIDI events to send to the FluidSynth MIDI Sequencer?

Thanks in advance for any help you can provide to make the Miditzer developers better FluidSynth users.

Jim Henry

[derselbst]

Up to this point, the Miditzer has collected real time MIDI inputs, mouse input, computer keyboard input, and FluidSynth MIDI player input and merged them. [...]
The next question is can we use FluidSynth to receive real time MIDI input

Ok, so you would use fluid_player_set_playback_callback() to intercept MIDI events coming from the player. And optionally, you could use new_fluid_midi_driver() to intercept MIDI events collected by fluidsynth coming from real-time devices. Miditzer would then receive a bunch of fluid_midi_event_t. Note that those are real-time event, i.e. they don't have a timestamp, because they are valid for the time you're receiving the callback, and you should also dispatch them to the synth while handling the callback. So, this first part would look like this:

[FS MIDI Player] --
                    \
[FS MIDI Driver] --> + --> [Miditzer Router] --> merged real-time event(s)

Those "merged real-time events" could be of type fluid_midi_event_t or fluid_event_t. In the first case, you would use fluid_sequencer_add_midi_event_to_buffer() to send them to the sequencer for immediate synth dispatching. In the later case, you would use fluid_sequencer_send_now(). Of course you would have the ability to schedule those events for later dispatching with fluid_sequencer_send_at(). Not sure if this answers your time-stamp question.

While you do all this event processing, you would have another thread continuously rendering audio from the synth. This could be one of fluidsynth's audio drivers or Miditzer itself by calling fluid_synth_write*() and friends.

[jimhen3ry]

Thank you! That helped me to gain a much better understanding of the data flow through FluidSynth.

Is there a way to add a later time for sequencer dispatch when using fluid_sequencer_add_midi_event_to_buffer()? If not, should there be? I peeked at fluid_seqbind.c and it seems like it would be a straightforward addition. I'll volunteer to make the addition if you think it is needed.

[derselbst]

Yes, the entire conversion logic performed by fluid_sequencer_add_midi_event_to_buffer() should be out-sourced to another function e.g. fluid_event_from_midi_event(fluid_event_t*, const fluid_midi_event_t*). Then you could do with that fluid_event_t whatever you want, either send_now() or send_at(). You're welcome to propose such a change. If you do, pls. base your work against the 2.2.x branch.

[jjceresa]

The next question is can we use FluidSynth to receive real time MIDI input and provide timestamped MIDI events so that the MIDI Player and real time MIDI input will have consistent time stamps?

The final question for now is whether there is a way of applying a synchronous time stamp to mouse input and computer key input to generate time stamped MIDI events to send to the FluidSynth MIDI Sequencer?

In the case you intend to use intensive sequencing and flexible MIDI event routing without any problem about timing accuracy and time stamp MIDI event (1 ms resolution), perhaps may you consider using the excellent MidiShare Library (http://midishare.sourceforge.net/).