Calling FFmpegFrameRecorder.record() with pixel format AV_PIX_FMT_NV21 results in wrong output

[michaeldietz]

I am using the FFmpegFrameRecorder to record the camera output on Android. When I use the record(frame) method the output looks normal. However, when I use record(frame, avutil.AV_PIX_FMT_NV21) then the output is wrong.

I think the reason for that is because step = width; in FFmpegFrameRecorder.java#L892 is only executed if the pixel format is set to AV_PIX_FMT_NONE which is the case when using record(frame). However when setting the pixel format directly with record(frame, avutil.AV_PIX_FMT_NV21) then this is missing which might cause the wrong output.

[saudet]

That's how it should be working. What are you trying to do?

[michaeldietz]

My code looks something like that:

switch (inputFormat)
{
	case ImageFormat.NV21:
		imageFrame = new Frame(width, height, Frame.DEPTH_UBYTE, 2);

		// This results in a wrong output, it only works if I set it to avutil.AV_PIX_FMT_NONE
		pixelFormat = avutil.AV_PIX_FMT_NV21;
		break;
	case ImageFormat.FLEX_RGB_888:
		imageFrame = new Frame(width, height, Frame.DEPTH_UBYTE, 3);
		pixelFormat = avutil.AV_PIX_FMT_RGB24;
		break;
	case ImageFormat.FLEX_RGBA_8888:
		imageFrame = new Frame(width, height, Frame.DEPTH_UBYTE, 4);
		pixelFormat = avutil.AV_PIX_FMT_RGBA;
		break;
}

[...]

ffmpegRecorder.record(imageFrame, pixelFormat);

When the inputFormat is NV21 then setting the pixelFormat to AV_PIX_FMT_NV21 results in a wrong output as you can see here:

However, when I change the pixelFormat to AV_PIX_FMT_NONE in this case then everything looks normal. Therefore I think this block in FFmpegFrameRecorder...

if (pixelFormat == AV_PIX_FMT_NONE) {
	if ((depth == Frame.DEPTH_UBYTE || depth == Frame.DEPTH_BYTE) && channels == 3) {
		pixelFormat = AV_PIX_FMT_BGR24;
	} else if ((depth == Frame.DEPTH_UBYTE || depth == Frame.DEPTH_BYTE) && channels == 1) {
		pixelFormat = AV_PIX_FMT_GRAY8;
	} else if ((depth == Frame.DEPTH_USHORT || depth == Frame.DEPTH_SHORT) && channels == 1) {
		pixelFormat = ByteOrder.nativeOrder().equals(ByteOrder.BIG_ENDIAN) ?
				AV_PIX_FMT_GRAY16BE : AV_PIX_FMT_GRAY16LE;
	} else if ((depth == Frame.DEPTH_UBYTE || depth == Frame.DEPTH_BYTE) && channels == 4) {
		pixelFormat = AV_PIX_FMT_RGBA;
	} else if ((depth == Frame.DEPTH_UBYTE || depth == Frame.DEPTH_BYTE) && channels == 2) {
		pixelFormat = AV_PIX_FMT_NV21; // Android's camera capture format
		step = width;
	} else {
		throw new Exception("Could not guess pixel format of image: depth=" + depth + ", channels=" + channels);
	}
}

...might needs to be changed to:

if (pixelFormat == AV_PIX_FMT_NONE) {
	if ((depth == Frame.DEPTH_UBYTE || depth == Frame.DEPTH_BYTE) && channels == 3) {
		pixelFormat = AV_PIX_FMT_BGR24;
	} else if ((depth == Frame.DEPTH_UBYTE || depth == Frame.DEPTH_BYTE) && channels == 1) {
		pixelFormat = AV_PIX_FMT_GRAY8;
	} else if ((depth == Frame.DEPTH_USHORT || depth == Frame.DEPTH_SHORT) && channels == 1) {
		pixelFormat = ByteOrder.nativeOrder().equals(ByteOrder.BIG_ENDIAN) ?
				AV_PIX_FMT_GRAY16BE : AV_PIX_FMT_GRAY16LE;
	} else if ((depth == Frame.DEPTH_UBYTE || depth == Frame.DEPTH_BYTE) && channels == 4) {
		pixelFormat = AV_PIX_FMT_RGBA;
	} else if ((depth == Frame.DEPTH_UBYTE || depth == Frame.DEPTH_BYTE) && channels == 2) {
		pixelFormat = AV_PIX_FMT_NV21; // Android's camera capture format
	} else {
		throw new Exception("Could not guess pixel format of image: depth=" + depth + ", channels=" + channels);
	}
}

if (pixelFormat == AV_PIX_FMT_NV21) {
	step = width;
}

[saudet]

Ah, I see. Please send a pull request!

[saudet]

Fix included in newly released version 1.4. Thanks again for the contribution!