Best way to implement audio streaming?

[Azorlogh]

Hey there, I'm making a program that needs to audio streams from the internet, and I've hit some challenges using Symphonia with HTTP readers, because I cannot block on the audio thread but the reader might not have any data available.

My general strategy until now was to pre-fill a buffer with some data on a separate thread, and hook Symphonia to that buffer.
If the data is not available, I return a WouldBlock, which gets passed through by Symphonia and gets interpreted by my audio routine to return silence until some time has passed to try again.

This works for normal audio streaming, but requires additional logic when first creating the source (ensuring enough data is available for the FormatReader to initialize), and when seeking (making sure the data at the next location is available).

For some formats this is easy: Mp3Reader only needs a few bytes prepared in the buffer to begin working, and for seeking, I can just pre-fetch a bunch of bytes at an arbitrary location before I tell Symphonia to seek there. (Well, Mp3Reader will need SeekMode::Coarse for that but until then I'm just restarting Symphonia there)

However for more complex formats, like IsoMp4Reader, this strategy gets tricky. Upon creation, it tries to fetch data from all around the stream and there's no way for me to know what to load in advance to avoid blocking. That is not too bad, I can always make the source creation blocking since it doesn't happen on the audio thread.
But for seeking, things are more subtle. Again I don't know which bytes will need to be read, so I can't pre-fill the buffer and need to make seeking blocking. But that means I have to move my source to a separate thread to do this operation then move it back to the audio thread which seems clunky.

Another strategy I thought of is to just put Symphonia in a separate thread from the audio buffer so that it can block. But I'm not sure how wasteful it would be to transfer the audio samples across threads, instead of only compressed data, so I'm hesitant.

Does any of these approaches make sense or is there an easier possibility that I'm missing?

I think if Symphonia supported resuming after WouldBlock errors, it would make things easier for me (was also mentioned here: #27 (comment)) because then I wouldn't need complex pre-fetching and could just return silence when I see a WouldBlock. But I assume that would require quite a bit more logic on Symphonia's side to handle this.

[jpochyla]

Hi @Azorlogh, I also encountered this problem when integrating Symphonia into Psst. I think it works quite well, but I have zero experience with audio processing, so beware (@pdeljanov would be the authoritative voice here). I also might have over-engineered things a little.

Related sources:

• Audio output, generic audio source: https://github.com/jpochyla/psst/tree/master/psst-core/src/audio. Output reads from audio source in a dedicated thread, checks a channel in the audio callback for atomic switches to a new source.
• Worker: https://github.com/jpochyla/psst/blob/master/psst-core/src/player/worker.rs. Worker is an actor (just a thread with some communication boilerplate removed), reads the decoded data from Symphonia and pushes them onto a ring-buffer). Supports seeking and reporting the play-head position.
• Storage: https://github.com/jpochyla/psst/blob/master/psst-core/src/player/storage.rs Keeps track of what is already downloaded and what isn't, basically.

[erikas-taroza]

The failure with the second curl run indicates to me that the MP4 you're using is not optimized for streaming.

That's weird. The files I am using for testing are scraped from YouTube so I would assume they are optimized for streaming.

You'll probably just notice that when you play this file there will be some initial seeks before playback as all the tables are loaded into memory.

Yes, this is what I noticed.

I will test as you have suggested and see if anything comes up.

Thank you.

[pdeljanov]

The failure with the second curl run indicates to me that the MP4 you're using is not optimized for streaming.

That's weird. The files I am using for testing are scraped from YouTube so I would assume they are optimized for streaming.

I believe by default youtube-dl and its forks "fixup" (remux) the file as part of the process. If you don't do a fixup, then you actually get a fragmented MP4 which is even more different. Symphonia does support fragmented MP4s though.

[erikas-taroza]

Hello @pdeljanov

I have tested my streaming file vs a normal Rust file. Initially, my file had some issues with requesting chunks beyond the bounds (which the http server did not like). Another thing I noticed was that the normal file allowed seeks to beyond the file size, where my implementation just returned the read position as the seek position. The tests that had normal/in bound numbers did pass as expected though. Here is my test project if you need to take a look.

I started to test again with Symphonia and I did find one issue was with how I was buffering. I would do an early return with Ok(buf.len) which was very incorrect. This caused a seek with a very big offset (>2 billion). I have fixed this issue.

Now when loading the file, it keeps on seeking and reading until the end of the file is reached, essentially downloading the whole file. Only then is playback started. I did some debugging and found this line as the initiator of these seek calls. Is this the result of the fragmented MP4 that YouTube serves? If it is, then how do I go around fixing it, if it's even possible?

[erikas-taroza]

I found a link to a normal MP4 file and I was able to stream it, with seeking, without any issues.

I spent some time with https://www.onlinemp4parser.com/ and ChatGPT to learn how the MP4 format works. I downloaded a fragmented MP4 and also made a copy with ffmpeg that is not fragmented. I found that the STBL atom in the fragmented MP4 is pretty much empty when compared to the normal file. I also noticed that in the TRUN atom of every MOOF atom, the sample size entries corresponded to those in the STSZ atom of the normal file. This maybe explains why I was previously only able to freely seek in downloaded parts.

I am not really sure where to go from here. Maybe course seeking for MP4 would work?

[pdeljanov]

If I understand correctly, it sounds like the MediaSource works properly now and the only problem is that loading the MP4 is slow?

What you're seeing sounds expected to me. The reader scans the top-level atoms in try_new to make sure it's playable. With a fragmented MP4, there are a lot of top-level atoms (mostly MOOF + MDAT pairs). While scanning, the function ignore_bytes is used to skip the body of the atoms. Now, ignore_bytes will perform a seek if the count is large enough, but if not, it just reads until it has skipped enough. I looked at a YouTube MP4 and the fragments are pretty short (10 seconds) and may fall below the threshold required for a seek. Therefore, it's likely that the file is just being read entirely start to finish.

I think there are few avenues for optimization (from easiest to hardest):

1. I think ignore_bytes can be replaced by an explicit seek if is_seekable is true. However, you will still be doing a lot of seek + small read operations on the first load as the atoms are skipped over.
2. If the MOOV and FTYP atoms were already encountered, it may be reasonable to terminate scanning of the top-level atoms at the first MOOF or MDAT atom. I'd need some documentation (i.e., standards, etc.) or proof that this is a good general optimization that won't break anything. However, this would still defer loading the fragments to the first seek. I think that may be more reasonable though.
3. Use the SIDX atom (segment index) to avoid the scanning that would happen on a seek. This is likely a very tricky thing to implement.

Let's move further discussion over to #175 since this likely isn't the best place for such a discussion.

[aschey]

If anyone else stumbles on this discussion looking for a way to add streaming to their project, I made a crate for this that should hopefully be flexible enough for most use cases: https://github.com/aschey/stream-download-rs.