Audio synch issue

[egorapolonov]

Hi,
There is one issue according to audio/video sync.
I tried to extract some part of video, but the sound comes earlier than video. You can reproduce it using this code:

import java.io.File;

import org.bytedeco.javacv.FFmpegFrameGrabber;
import org.bytedeco.javacv.FFmpegFrameRecorder;
import org.bytedeco.javacv.Frame;
import org.bytedeco.javacv.FrameGrabber.Exception;

public class VideoExtractor {
    
    public static void main(String[] args) throws Exception, org.bytedeco.javacv.FrameRecorder.Exception {
        cutVideo(1631500000, 1661500000);
    }
    
    public static void cutVideo(long startTimeMicros, long endTimeMicros) throws Exception, org.bytedeco.javacv.FrameRecorder.Exception {
        File outputFile = new File("fragment_" + System.currentTimeMillis() + ".mp4");
        try (FFmpegFrameGrabber video = new FFmpegFrameGrabber("https://www.dropbox.com/s/255gqsm1wjn77ev/h264high_sound_issue.mp4?dl=1")) {
            video.start();
            video.setTimestamp(startTimeMicros);
            try (FFmpegFrameRecorder rec = new FFmpegFrameRecorder(outputFile, video.getImageWidth(),
                    video.getImageHeight(), video.getAudioChannels())) {
                initializeEncoder(video, rec);
                rec.start();
                Frame frame;
                while (video.getTimestamp() <= endTimeMicros && (frame = video.grab()) != null) {
                    rec.record(frame);
                }
            }
        }
    }
    
    public static void initializeEncoder(FFmpegFrameGrabber grabber, FFmpegFrameRecorder recorder) {
        //video
        recorder.setVideoCodecName("libx264");
        recorder.setFrameRate(grabber.getFrameRate());
        recorder.setVideoBitrate(grabber.getVideoBitrate());
        recorder.setVideoOption("threads", "4");
        recorder.setVideoOption("profile", "main");
        recorder.setVideoOption("level","4");
        //audio
        recorder.setSampleRate(grabber.getSampleRate());
        recorder.setAudioBitrate(grabber.getAudioBitrate());
        recorder.setAudioCodec(grabber.getAudioCodec());
    }
    
}

Playing with startTimeMicros you can find positions when audio and video are synchronized.
I tried to set timestamps on recorder using FFmpegFrameRecorder.setTimestamp() but it didn't help. Then, what I saw that there is an invocation of the method grabFrame(false, true, false, false) in the grabberFFmpegFrameGrabber.setTimestamp(). It means doAudio = false. I used inheritance and overrided the method

@Override
public Frame grabFrame(boolean doAudio, boolean doVideo, boolean processImage, boolean keyFrames) throws Exception {
    return super.grabFrame(true, doVideo, processImage, keyFrames);
}

Now it fixed the problem, I tested it on more than 100 time ranges.
Could you explain is it a bug or I forgot to configure something?

[saudet]

Sounds like a good fix. Could you send a pull request?

[egorapolonov]

Sure. Seems like ready
#871

[saudet]

Fix included in newly released version 1.4. Thanks again for the contribution!

[egorapolonov]

Thank you, too.

[egorapolonov]

Hi @saudet ,

I wanted to upgrade javacv version from 1.4.4 to 1.5.2. I made some regression tests including this one.
And I see this bug is reproducible in versions 1.5, 1.5.1 and 1.5.2.
I tried both setTimestamp flags but it didn't help
video.setTimestamp(startTimeMicros, true); // doesn't matter in this example, produces same file

Could you, please, advice how to solve it?
Should I create one more ticket?

[saudet]

The "old" behavior should be the same, yes. If you have a test case that fails, it would help if you could add it as unit test and send a pull request, but at least please open a new issue and provide more details. Thanks!