Waveform run more and more out of sync to audio

[markusweigelt]

When i play my audio file it works correctly but after some time the curve doesn't quite fit the audio file.

I'm using a podcast audiofile that is only spoken, so there are some pause between the words and sentences.
My audio file for testing https://media.bibliothek.kit.edu/world/2020/DIVA-2020-76_mp3.mp3

Besides that, I use the zoom sample implementation to zoom into the wave.
https://wavesurfer-js.org/examples/#zoom.js

At the beginning the wave is still synchronous. At minute 20 for example there is a pause in the wave, but the speaker speaks.

Wavesurfer.js version(s):

7.0.0-beta.8

Browser and operating system version(s):

Chrome

[katspaugh]

Thanks Markus! This issue exists indeed.
My working hypothesis is that the data decoded by Web Audio has a slight mismatch with the audio decoded by the native audio element, and it accumulates over time. They even report a slightly different duration sometimes.
I don’t know yet how to fix this. Need to research what options we have.

[markusweigelt]

I think it is a problem with the calculation of cursor position maybe in connection with the zoom level. Because if i disable the autostart and jump directly to a later position in the audio, for example the last wavy lines, it is synchronous.

[katspaugh]

Hmm, maybe. The cursor position is set as a percentage of the width of the waveform. Basically currentTime / duration * 100.

https://github.com/katspaugh/wavesurfer.js/blob/beta/src/wavesurfer.ts#L243
https://github.com/katspaugh/wavesurfer.js/blob/beta/src/renderer.ts#L511

Maybe some precision is lost due to that math, or even CSS. However, in my tests, the desynchronization was sort of random. Sometimes it would be accurate, sometimes not on the same region of audio.

[markusweigelt]

Thanks, I think the calculation of the cursor position is correct. Currently I have the assumption that the event handler of the timeupdate event takes too long and more and more timeupdate events come on the event stack and are processed later and later.

In the event documentation is the following hint:

assuming the event handlers don't take longer than 250ms to run

https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/timeupdate_event

Maybe there is a need for a timeout or the performance of the handler maybe optimized where it is possible, e.g. cache the getDuration result instead of get them from the media element. Maybe the frequenz of the smooth function / tick event is to high.

[katspaugh]

Interesting thought, definitely worth checking. Can be tested by removing both tick and timeupdate listeners, then setting the time programmatically to a position with a known desync (using wavesufer.setTime) and playing from there. If it starts playing from the right position, it must be the event frequency.

[katspaugh]

I've experimented with the following settings:

import WaveSurfer from 'https://unpkg.com/wavesurfer.js@beta'

const wavesurfer = WaveSurfer.create({
  container: document.body,
  url: 'https://media.bibliothek.kit.edu/world/2020/DIVA-2020-76_mp3.mp3',
  minPxPerSec: 100,
})

const button = document.createElement('button')
button.textContent = 'Play'
button.style.margin = '1em 0'
document.body.appendChild(button)

wavesurfer.on('play', () => {
  button.textContent = 'Pause'
})
wavesurfer.on('pause', () => {
  button.textContent = 'Play'
})

wavesurfer.on('ready', () => {
  wavesurfer.setTime(3 * 60 + 40)

  button.addEventListener('click', () => {
    if (wavesurfer.isPlaying()) {
      wavesurfer.pause()
    } else {
      wavesurfer.play()
    }
  })
})

Even with the tick event commented out, when it jumps to the 3:40 mark on ready, there's already a mismatch between the audio and the waveform.

[alexbegey]

I was also having some playback timing issues. After converting my mp3 file to CBR (constant bit rate), the issue seems to be gone and everything is working correctly now. It seems like the seek functionality is struggling with VBR (variable bit rate) audio files.

Source: https://stackoverflow.com/questions/32144092/inconsistent-seeking-in-html5-audio-player

[katspaugh]

@alexbegey that makes a lot of sense! Thanks for solving this mystery.
I wonder if there’s anything we can do apart from not using HTML5 audio.

[tracy-burnett]

Instead of converting my mp3 files to CBR, I have been running them through the program called VBRFix, which tweaks their tags/headers but afaik does not convert them to CBR. After running them through the program, the files work just fine in Wavesurfer.

Where in the code does Wavesurfer read the LAME or Xing (or VBRI) tag of the audio file? Is it possible that it is currently set to read only one of those kinds of headers, and that it cannot recognize the other two?

I ask because even mp3 and mp4/m4a (I can't remember which) files made by modern iPhones and software programs are tripping up Wavesurfer, but VBRFix somehow makes them work just by tweaking their headers to be supported by Wavesurfer.

TIA

[katspaugh]

Interesting finding, @skysnolimit08, thank you!
Wavesurfer doesn’t decode audio files in JS, it’s done by the browser (Web Audio).
I suppose we could try and fix those headers in JS before decoding.

[tracy-burnett]

Interesting finding, @skysnolimit08, thank you! Wavesurfer doesn’t decode audio files in JS, it’s done by the browser (Web Audio). I suppose we could try and fix those headers in JS before decoding.

It would be great if somebody else could verify what I've found, since I'm out of the country with no access to my materials right now and it's been a long time since I have thought about this. (It is still a common problem for my users, though)

In terms of tweaking the headers in JS, I wouldn't know where to start, but it sounds like an ideal solution. I tried to find the actual source code of VBRFix to see what it does, but couldn't locate it. Maybe one of the standard audio diagnostics tools used (mp3tag? or even VBRFix itself) can be used to inspect the headers both before and after a problem file is run through VBRFix. We could see what type VBRFix changes the header to and what it says, then figure out how to replicate that in JS.

[NegiBaba]

Don't know if this is related to CBR or VBR but I am recording and audio with a library in react and when paused I pass the audio/mpeg blob to wavesurfer. But usually most of time when wavesurfer loads the audio the duration is different that what it actually is e.g. if the audio was paused at 30s then wavesurfer would return a number between 27-30s and when the audio is being played with wavesurfer the seeker slows down or just jitters for some time as the audio and visualisation are out of sync thus slowing down helps it to sync but looks really weird. And when I download the audio and play it using mac audio player it displays correct duration and runs smoothly.
I have looked at various StackOverflow threads to find any issue with it, some are saying to use ffmpeg to convert the audio or something like that.
Maybe the issue is with code not properly passing the blob to wavesurfer.

[katspaugh]

@NegiBaba it's something with how the audio is decoded by Web Audio. See #3049.

[NegiBaba]

Hey @katspaugh , So I tried something similar to what @skysnolimit08 said above.
My use case is that I am recording something and then passing the recorded blob to wavesurfer for a waveform audio playback, after checking the audio information for the same I found the audio was recorded as VBR.
After that I used the ffmpeg library to convert the recorded audio blob's encoding to CBR and passed it wavesurfer and voila! everything seems to be working soundly.
I can see the duration issue has been resolved by this as well as the attached issue #3049 audio plays till the end without missing any part or the cursor lagging somewhere. I am yet to test it thoroughly with large audio recording and see if it works as expected.

[katspaugh]

@NegiBaba looks like specifying a fixed bit rate solves that issue for the Recorder plugin:

E.g.

RecordPlugin.create({
  audioBitsPerSecond: 128000,
})

[NegiBaba]

@katspaugh Yeah but I am actually using a different library to record audio, so now need to make changes to that library. I think we should have some note in the documentation regarding the same that wavesurfer works best if you provide a CBR audio file/blob to it and then link to more info about bit rate in audio. That can really help developers like me who would have to go through many issues in the library, what you say?

[katspaugh]

Yes, it's in the FAQ in the readme. We could also add it as a docstring of the url parameter.

[markusweigelt]

Thx @katspaugh and @alexbegey! It works for me with the conversion to CBR too.

[corysimmons]

Context for anyone else stumbling upon this issue, on the manpage to VBRfix it says:

In an average song there are points that require high quality and points that require low
quality (i.e., silence). Instead of having the whole file at 160kbps (CBR - Constant Bit
Rate), we can use VBR - Variable Bit rate. This allows us to have say 96Kbps at points not
requiring high quality and 192kbps when we need it: resulting in an overall smaller but
higher quality MP3.

However, the problem is that many MP3 programs estimate the time of a MP3 based on the
first bitrate they find and the filesize. With VBR you can get fairly random times; as
most songs start with silence you usually get the song length being shown as much longer
than it should be. Also when you jump through the file in VBR: 50% through the file is not
50% through the song.

A VBR null frame is placed at the beginning of the file to tell the MP3 player information
about the song length and indexing through the song. Some poor encoders don't produce
this null frame or do so incorrectly - this is what vbrfix attempts to fix. Vbrfix can
also fix other problems with MP3s as it deletes all non-MP3 content (other than tags you
state you want to keep).

From what I can tell from reading about this, it seems like if you just set everything to be a CBR, the file size is going to be significantly larger, so the ideal solution would be to always insert a null frame at the beginning of every mp3 somehow... but I doubt realtime it'd be performant, so I don't think Wavesurfer.js can fix this (you'd just have to use VBRfix to all your files before you deploy them).

I'm unsure if somehow the WebAudio.js shim already solves this like it says in the FAQ. Or if you can just specify backend: 'WebAudio'.

I'd definitely like to set up all my Wavesurfer projects to stay in sync even with large files. 😆