FFMpegFrameGrabber fails to identify proper resolution with multiple SPS/PPS entries in H264 encoded stream

[robertoandrade]

I have a video that changes orientation during playback which causes the insertion of additional SPS/PPS NALUs specifying the new stream dimension but even though the underlying library detects the change and logs it accordingly (like ffmpeg and ffplay do) the grabber instance I have will still spit me frames with the previous dimensions. Is there a way to reset it upon detection of the change?

I've trying setting setImageWidth/Height before starting the stream but couldn't find a way to do it mid way through playback.

Initial log feedback:

Input #0, h264, from 'src/test/resources/videos/orientation-change.h264':
  Duration: N/A, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuvj420p(pc), 364x648, 25 fps, 25 tbr, 1200k tbn, 50 tbc

When the extra SPS/PPS entries are found:

[h264 @ 0x7f97b4a6c000] Reinit context to 1152x656, pix_fmt: yuvj420p

[robertoandrade]

I ended up extending it and "reinitializing" some of the buffer variables setup during the startUnsafe() call based on the new dimension provided by the video_c AVCodecContext as follows:

import static org.bytedeco.javacpp.avcodec.avpicture_fill;
import static org.bytedeco.javacpp.avcodec.avpicture_get_size;
import static org.bytedeco.javacpp.avutil.av_malloc;
import static org.bytedeco.javacpp.opencv_core.IPL_DEPTH_8U;

import java.awt.Dimension;
import java.io.File;
import java.lang.reflect.Field;

import org.bytedeco.javacpp.BytePointer;
import org.bytedeco.javacpp.avcodec.AVCodecContext;
import org.bytedeco.javacpp.avcodec.AVPicture;
import org.bytedeco.javacpp.avutil.AVFrame;
import org.bytedeco.javacpp.opencv_core.IplImage;
import org.bytedeco.javacv.FFmpegFrameGrabber;

public final class OrientationAwareFFmpegFrameGrabber extends
        FFmpegFrameGrabber {

    public OrientationAwareFFmpegFrameGrabber(String filename) {
        super(filename);
    }

    public OrientationAwareFFmpegFrameGrabber(File file) {
        super(file);
    }

    @SuppressWarnings("unchecked")
    public <T> T getField(String fieldName) throws NoSuchFieldException,
            SecurityException, IllegalArgumentException, IllegalAccessException {
        return (T) field(fieldName).get(this);
    }

    public <T> void setField(String fieldName, T value)
            throws NoSuchFieldException, SecurityException,
            IllegalArgumentException, IllegalAccessException {
        field(fieldName).set(this, value);
    }

    public Field field(String fieldName) throws NoSuchFieldException {
        Field field = getClass().getSuperclass().getDeclaredField(fieldName);
        field.setAccessible(true);
        return field;
    }

    public AVCodecContext video_c() {
        try {
            return getField("video_c");
        } catch (NoSuchFieldException | SecurityException
                | IllegalArgumentException | IllegalAccessException e) {
            e.printStackTrace();
            return null;
        }
    }

    @Override
    public IplImage grab() throws Exception {
        checkDimensionChange();

        return super.grab();
    }

    Dimension lastDimension = null;

    public void checkDimensionChange() {
        try {
            int width = video_c().width();
            int height = video_c().height();

            Dimension dimension = new Dimension(width, height);

            if (lastDimension != null && lastDimension.equals(dimension)) {
                return;
            }

            lastDimension = dimension;

            int fmt = getPixelFormat();

            // Determine required buffer size and allocate buffer
            int size = avpicture_get_size(fmt, width, height);

            BytePointer buffer_rgb = new BytePointer(av_malloc(size));
            setField("buffer_rgb", buffer_rgb);

            AVFrame picture_rgb = getField("picture_rgb");

            // Assign appropriate parts of buffer to image planes in picture_rgb
            // Note that picture_rgb is an AVFrame, but AVFrame is a superset of
            // AVPicture
            avpicture_fill(new AVPicture(picture_rgb), buffer_rgb, fmt, width,
                    height);
            picture_rgb.format(fmt);
            picture_rgb.width(width);
            picture_rgb.height(height);

            setField("return_image", IplImage.createHeader(dimension.width,
                    dimension.height, IPL_DEPTH_8U, 1));

        } catch (Throwable e) {
            e.printStackTrace();
        }
    }

    @Override
    public String toString() {
        AVCodecContext video_c = video_c();

        return super.toString() + "(" + video_c.width() + "x"
                + video_c.height() + ")";
    }
}

[saudet]

Looks good! If you could make a patch out of that to modify FFmpegFrameGrabber, after removing the field hacks as well as java.awt.Dimension because that doesn't work on Android, send over a pull request so we can integrate that! Thanks

[robertoandrade]

From what I just checked when upgrading to the latest javacv and ffmpeg preset I noticed the grabber source changed a bit so this solution doesn't work on the latest and from my tests it's still an issue on the latest. Is there any other recommended way to deal with the orientation change and reinitializing the internal state?

[saudet]

We can still do exactly the same thing. I'm just waiting for someone to produce a proper patch...

[robertoandrade]

Ok, will try to do that today... updating my logic and testing it right now.

[saudet]

Any updates?

[saudet]

I'm pretty sure pull #769 fixes this. Please let me know if you still have issues with this! Thanks

[saudet]

Thanks to @mifritscher, support is now included in version 1.4! Enjoy