Why FFMPEGFrameGrabber doesn't correctly process the previous frame on some videos?

[DeNRuDi]

Hello! Sorry for my bad English, i am from Ukraine🙂
I am facing a problem on android when i want to get the previous frame. On some videos, the previous frame is set correctly, but on some it happens that instead of rewinding 1 frame, the grabber rewinds ~ 10-30 frames (most often equal to the number of fps). This only happens when rewinding of back, when I get the next frame - I always get the correct frame. My guess is that this problem is due to the closest frame capture (like as in MediaMetadataRetriever OPTION_CLOSEST_SYNC). Here is the most simplified code that can cause this problem:

grabber.getFrameNumber(); // for example, 300
grabber.setFrameNumber(grabber.getFrameNumber() - 1);
grabber.grabImage(); // 270, although 299 is expected

How can it be solved? Maybe I missed something in the documentation? Noticed this on mp4 video format.
Thank you.

[saudet]

For typical formats, seeking is a not a simple process. If the only frame you need to access is the previous frame, I would recommend to decode as usual, but copy each frame into another a "previousFrame" object or something, and use that one when you need to access the previous frame.

[anotherche]

@DeNRuDi , this is most likely because you are using setFrameNumber to select the frame number. It does not distinguish between video and audio frames. In this case, the underlying setTimestamp function looks for any frame (video or audio) with needed timestamp. If the video file encounters an audio frame with a suitable timestamp, which appears before the video frame in the file, then the position will be set to this audio frame. Next, when you grab a video frame with grabImage, then its timestamp will always be less than required. As a rule, the difference in timestamps between audio and video frames in files does not exceed 1 second. Therefore, you get a video frame with a timestamp that is 1 second earlier.
To fix this, just use setVideoFrameNumber instead of setFrameNumber.

[DeNRuDi]

@anotherche Didn't realize there was such a method, thanks. I tried to call the setVideoFrameNumber() method, and bassically, with rewinding frames by the number of fps was solved, but on problem videos rewinding occurs not by 1 frame, but for some reason by 3 frames - I checked this for 2 videos. It's still better than 10 or 30.

[anotherche]

@DeNRuDi could you share a problem video?
I have found examples myself. They are almost everywhere).
My guess is that it might be related to b-frames. Their order in the video does not correspond to a direct sequence - frames later in time of presentation may turn out to be earlier in time of decoding. I'm afraid this behavior is irresistible. Anyway, no ideas yet. However, everything is solved absolutely error-free when it is necessary to get neighboring frames. You use @saudet approach proposed above - you just need to save the previous (or several previous) frames as they are grabbed. This is not only an error-free way, but also much faster. Since each call setVideoFrameNumber() method implicitly leads to grabbing tens or hundreds of frames.

[anotherche]

Well, after a careful study of the situation, I see - these are not b-frames. This is again the "good old" rounding errors. Since the timestamps do not strictly follow the FPS increment, the error in determining the frame number lies within + -1. Therefore, the error in determining the position of two adjacent frames can theoretically be in the range from -1 to +3. Perhaps this can be corrected. But the problem associated with the inconsistency of the time stamp increment in the general case seems insurmountable. All the same, to select adjacent frames, it is best to read them sequentially.

[DeNRuDi]

@anotherche okay, I understand, thanks. In general, in my implementation, it is not known in advance when the previous frame should be received. Sometimes it needs to be received 1 time, sometimes 10, so allocating a frame buffer for my implementation is so-so. I have an idea for my implementation - I want to try, in case of receiving the previous frame, first check which frame was received, and if it is 2-3 less than I need, then simply use grabImage to get extra frames in the loop and pick required, i.e.: current frame -1. I'll check out this idea.

[saudet]

@DeNRuDi Improvements from @anotherche have been merged!
Please give it a try with the snapshots: http://bytedeco.org/builds/

[DeNRuDi]

@saudet perfectly. I'll try this. Thanks.

[DeNRuDi]

@saudet finally got around to trying the snapshot build. Works great. Checked on all problematic videos - no more problems. I think the question can be closed. Thanks🙂

[saudet]

The fix for this has been released with JavaCV 1.5.8. Thanks for the contribution!