Need More Complex Examples

[frink]

Hey @mackron!!

First, so sorry for disappearing on you a few years ago. My life took a turn for crazy and I'm just getting back on my feet. Very sorry to leave you hanging. I finally had time to thoroughly review all the work on node graphs and everything else from #196. Generally, it looks great! I'm very impressed with how everything works. After looking through everything I've just got a few bits of feedback for more complex scenarios that I thought was worth passing along:

1. How can I do a complex node graph with an indeterminate amount of "wires" connecting from a single out to many node ins? Right now it seems like you'll need to do a ma_splitter_node every time you want to split out a new node. I'm wondering if this can be simplified by simply calling ma_node_attach_output_bus() and letting MA deal with the splitting behind the scenes. In any case, it would be good to show how the node graph can be rebuilt dynamically on the fly as things are connected and disconnected.

2. How does a display thread connect to the processing thread to get input to do things like show a waveform in process? This is common place when building DSP to give visual feedback. Sharing data between threads is always tricky. This might be a good thing for an example to show how to trigger other things off of audio input.

3. How do we synchronize two looped sources to stay sample accurate? Game soundtracks often stack loops and let them morph to create a soundscape but the music needs to stay in sync. Looper guitar pedal records a stream of audio and either stores a file for each loop or does a sum in place and replaces the PCM in a ring buffer in memory. It seems like there should be a complex looping example that shows how multiple loops can stay in sync when they are multiples of one another.

4. How does a DJ do scratching with MA? Since MA is so small it really presents itself well for building embedded systems doing DJ type stuff. (In fact I've seen some threads where people are doing just that...) Seems like an example on moving the play head could be quite helpful as scratching is not trivial.

5. How would control voltage work? In analog synthesis CV is used to modulate parameters constantly. (That's how you build cool synthesizers...) I think this should probably be treated just like another audio source but I'm not sure how it could/should work. At any rate, it would be good to show how to modulate a filter from an LFO since this is a common use-case in synthesizer design.

I can't think of any other wacky ways to break the system or what I might do with MA that it doesn't do already baked in. I'm enjoying getting back into audio programming and looking forward to doing some crazy stuff with MA now that it is mature for most of my use cases. Like I said above, I'm very impressed by how far it's come in the last two years! Many many kudos and thanks!!!

[mackron]

Welcome back!

1. At the moment, the best way would probably be to just do a custom node that's based on the stock splitter node and just support an arbitrary output count. You would set the output bus count to 0 in the vtable which would require you to set the output bus count on a per-node basis via ma_node_config. However, looking at the code, there's little reason for the stock splitter node to be restricted to only two outputs, so I might investigate supporting that. For dynamically connecting/disconnecting wires, you would have to make it so the number of outputs is large enough to account for your maximum scenario, and then just leave the unused outputs disconnected which miniaudio should handle just fine.

2. This is program-specific. There's just so many different ways to do it and different UI toolkits to deal with that there's just no real answer. However, you could have a passthrough node which is used only to capture the raw PCM data and then pass that to some buffer somewhere that the UI thread has access to. You could connect that node to the end of the graph, but just before the endpoint, and have everything connect to that. In that case you would set the MA_NODE_FLAG_PASSTHROUGH flag in the node's vtable (ma_node_vtable), and then in the processing callback just read from the input buffer and leave the output buffer untouched.

3. I'm not entirely sure what you're asking with this one. Might need a few more details on this. I mean, data sources (ma_data_source) support loop points which means looping can "just work" at the data source level. There is a global timer that can be retrieved with ma_node_graph_get_time() which is just a running sample count of the number of frames that have been read from the graph. It's just a number and can be set at any time with ma_node_graph_set_time() if you need to do some kind of re-sync with some other clock you've got running. Setting this timer won't do any kind of seeking of data sources or anything - it's mainly used for setting the state of a node (started or stopped) at a specific PCM frame: ma_node_set_state_time(). If you're wanting some kind of system where you resync the graph's timer which then seeks to the appropriate frame in all of the connected data sources, that's not something that miniaudio does and is something that would involve you having to manually seek each of your data sources to the correct location. You'd need to make sure you aren't in the middle of processing anything on the audio thread so you don't get any threading issues.

4. No idea!

5. I think what you mean here is a system where an effect parameter changes over time at the resolution of a PCM frame? I've had requests for this, and I support the idea, but to be honest I have no idea how to design a good API to support it. Suggestions welcome!

[frink]

Thanks! Glad to be back!

1. Sounds like for my generalized DAW I need to make a Splitter with a max connection count. (I'm already doing this to preallocate connection memory on the visual side. I can just mirror it on the audio side. This is a simple enough change I might try to send a PR next week on the Splitter just like you've described. Although, as fast as you are I bet you'll beat me to it! 😉

2. For noobs getting into audio programming, they don't understand the dangers of mixing audio and video contexts. When they learn the need for separation there is always a question of how to communicate between the audio and video threads. While I don't think we need to show every permutation of different UIs something like an ncurses volume meter would be wonderfully informative as it explains not only the communication between the two threads but also who is responsible for memory allocation and provides a teaching ground for most of the gotchas that are common with those just coming into audio programming.

3. This is a bit more complex of an example. But I was thinking about the many ways we use loops synchronously. While I know the node graph keeps things synchronized, (I can even listen for the graph time to find starting points) most noobs will find that part troublesome and really need a bit more hand-holding to understand how this works. My idea is something like a basic live looper where you start the program and it records PCM to a buffer from line in until you press ENTER. Then it plays back in a loop while asynchronously saving to disk. When you press ENTER again where it starts recording a second loop loop twice as long as the first and overdubs that buffer until you hit ENTER. A final press of ENTER exits the program. This shows several things that are gotchas including asynchronously saving to disk, calculating loop sync length, and overdubbing a buffer in place using a ring buffer to sum old and new audio.

4. I've seen mention of DJ-like applications which come with the requirement of scratching sooner or later. Basically you're moving the play head around a buffer in some creative ways. I've never done anything like that personally so I'm not sure exactly how it works from a technical perspective. I'll have to investigate and see if I can find some better implementation specifics to bring back to you. It would be great if we could figure that out and include it as another example though. Would be really cool if games did a live record scratch transition between audio scenes.

5. For passing CV signals around I'd suggest "wires" that regular PCM so that your LFOs can just be normal oscillators and sound sources. Then you create a passthru node (I'd call it a control node) that is initialized with a pointer to the float you are assigning and two static values for min and max. Then the control node can translate the value from PCM to a ranger between the min and max values relative to PCM data. You can then use other nodes to translate PCM data to CV in various ways. (Like amplitude, RMS or frequency driving the controls...)

Hopefully that makes more sense.

[mackron]

I've pushed a change to the dev branch to support an arbitrary output count for the splitter node. I already had some infrastructure in place the support this - I didn't need to make any change to the processing callback at all. I think I might have had this idea earlier and then just forgot about it. Just keep in mind that it's limited to MA_MAX_NODE_BUS_COUNT which is 254 (this is a global limitation, not just the splitter). Like I said in my earlier comment, just leave the relevant output buses disconnected (or disconnect an already connected one) and miniaudio should deal with it (if not, it's a bug). Just don't willy nilly set it to MA_MAX_NODE_BUS_COUNT if it far exceeds what you'll actually need because there are caches allocated internally for each output bus for data processing reasons so you'll just we wasting memory. Also, there will be a slight processing overhead because a node's processing callback is not aware of disconnected outputs (they assume all outputs are connected). This is just a tradeoff for implementation simplicity (disconnected outputs are fairly uncommon so I'm happy to make this trade). Even though a node will still do processing for disconnected outputs, nothing will break, and it won't contribute to the final mix in any way.

Regarding examples, I don't know. I agree with your general sentiment that there needs to be more examples (especially for the high level API - I don't think I've even got an example for basic spatialization!), but I'm just not sure how I want to approach it yet because it's a fairly significant time sink to write and maintain. I've been wanting to do a functional music player for some time, and I think that would make a really good example because it'd be practical and would cover a lot of topics. It would cover the UI thing you mentioned because it could do visualisations, and it could also do effects like an equalizer which could be used to show off the node graph system. You could even have a plugin system to support additional encoding formats to make use of miniaudio's custom decoder system. Heaps of things you could show off.

[frink]

Thanks for pushing the splitter expansion!!! 🕺💃 🎉 That will be VERY helpful for me.

I like your idea of the forefront example being a basic media player with EQ. I can see a lot of value in that one example. But I also agree that "could do" is always a big time sink and we need to be careful about where we put our time. The right way to balance this is probably how LibWebSockets or Dear ImGUI manage examples. The provide a lot of examples but always expect the community to maintain them and they get removed if they sit broken for too long.