Using FFmpegFrameRecorder to merge multiple MP4 videos with different frame rates!

[DsrMedia]

I'm using FFmpegFrameRecorder and FFmpegFrameGrabber to merge multiple videos and it works perfectly for videos with same frame rate. But If I use videos with different FPS (30fps, 24fps and 17fps) it looks like some are in slow motion and others are speed up!

So, what do I have to do to make sure that audio and video are in sync throughout videos with different frame rates?

[saudet]

Try to call FFmpegFrameRecorder.setTimestamp() to use variable frame rate (VFR).

[saudet]

FYI, there's an example of this here:
https://github.com/bytedeco/javacv/blob/master/samples/RecordActivity.java#L297

[DsrMedia]

Thanks @saudet, setTimestamp() solved the problem!

[holmes1491]

I have the same problem merge output video is slow down, have same frame rate with each video file
can you elaborate bit more on your suggestion FFmpegFrameRecorder.setTimestamp()
where to apply this
@DsrMedia @saudet please help with this

`@SuppressWarnings("resource")
private static void mergeVideo1() throws Exception, org.bytedeco.javacv.FrameRecorder.Exception {
LocalTime localTime = LocalTime.now();
String mfmt = localTime.getHour()+""+localTime.getMinute()+""+localTime.getSecond();

	avutil.av_log_set_level(avutil.AV_LOG_ERROR);
	FFmpegFrameGrabber grabber1 = new FFmpegFrameGrabber(mPath+"/tmp/5-57_output.mp4");
    grabber1.start();           
    FFmpegFrameGrabber grabber2 = new FFmpegFrameGrabber(mPath+"/tmp/6-32_output.mp4");
    grabber2.start();
    FFmpegFrameGrabber grabber3 = new FFmpegFrameGrabber(mPath+"/tmp/7-33_output.mp4");
    grabber3.start();
    FFmpegFrameGrabber grabber4 = new FFmpegFrameGrabber(mPath+"/tmp/8-46_output.mp4");
    grabber4.start();           


    FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(mPath+"/tmp/merge/"+mfmt+"_output.mp4", grabber1.getImageWidth(),grabber1.getImageHeight(),grabber1.getAudioChannels());
	recorder.setVideoCodec(avcodec.AV_CODEC_ID_MPEG4);
	recorder.setFormat("mp4");
	//recorder.setFrameRate(grabber1.getFrameRate());
	recorder.setSampleRate(grabber1.getSampleRate());
    recorder.setVideoBitrate(grabber1.getVideoBitrate());
	recorder.setAudioChannels(grabber1.getAudioChannels());

    recorder.start();
    
    Frame frame1 = null, frame2 = null, frame3 = null, frame4 = null;

    while ((frame1 = grabber1.grabFrame()) != null || (frame2 = grabber2.grabFrame()) != null || (frame3 = grabber3.grabFrame()) != null || (frame4 = grabber4.grabFrame()) != null) {

        recorder.record(frame1);
        recorder.record(frame2);
        recorder.record(frame3);
        recorder.record(frame4);
        System.out.println("After - "+recorder.getTimestamp());
    }

    recorder.stop();
    grabber4.stop();
    grabber3.stop();
    grabber2.stop();
    grabber1.stop();
}`

[holmes1491]

@DsrMedia how to solve your problem
please help me with this
i am stuck at this...

[saudet]

@holmes1491 Since all your files have the same frame rate, try to use packets instead, as shown here:
https://github.com/bytedeco/javacv/blob/master/samples/PacketRecorderTest.java

[holmes1491]

thanks for the reply i will try this one and let you know...

[holmes1491]

below my code

`

    private static final int RECORD_LENGTH = 5000;
public static void packetRecord() throws FrameGrabber.Exception, FrameRecorder.Exception {

    int audioChannel = AUDIO_ENABLED ? 1 : 0;
    
	LocalTime localTime = LocalTime.now();
	String mfmt = localTime.getHour()+"_"+localTime.getMinute()+"_"+localTime.getSecond();        
	avutil.av_log_set_level(avutil.AV_LOG_WARNING);//    AV_LOG_ERROR
    FFmpegFrameGrabber grabber1 = new FFmpegFrameGrabber(mPath+"/tmp/5-57_output.mp4");
    FFmpegFrameGrabber grabber2 = new FFmpegFrameGrabber(mPath+"/tmp/6-32_output.mp4");
    FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(mPath+"/tmp/merge/"+mfmt+"_output.mp4", 1280, 720, audioChannel);

    grabber1.start();
    grabber2.start();
    recorder.start(grabber1.getFormatContext());

    avcodec.AVPacket packet1 = null, packet2 = null;
    long t1 = System.currentTimeMillis();
    while ((packet1 = grabber1.grabPacket()) != null || (packet2 = grabber2.grabPacket()) != null) {
        recorder.recordPacket(packet1);
        recorder.recordPacket(packet2);
        if ((System.currentTimeMillis() - t1) > RECORD_LENGTH) {
            break;
        }
    }

    recorder.stop();
    grabber1.stop();
    grabber2.stop();

}

`
and i got below debug

`
[mp4 @ 000000001d1c84c0] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[mp4 @ 000000001d1c84c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
[mp4 @ 000000001d1c84c0] pts has no value
.
.
.
.
.

also explain what is significant of RECORD_LENGTH and how it gonna work with merge videos
please help me with this @saudet

[holmes1491]

for first message
[mp4 @ 000000001d1c84c0] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
in class FFmpegFrameRecorder -> recordPacket(AVPacket pkt) method use some Deprecated api

rest of two i have no idea...

[saudet]

The code for grabbing and recording packets uses deprecated functions, and it needs to be updated, see issue #818. If the current code doesn't work for you, feel free to fix it and send a pull request! If you have any questions regarding that, please post on that thread.

RECORD_LENGTH is whatever you want. Remove it if you don't need it.

[holmes1491]

sure!!! I have to bcoz I have a project related to video processing

But
can you help me with the above code which I posted earlier
I am very close just Seek little help from you
something wrong with recorder.setFrameRate()

as you suggested above FFmpegFrameRecorder.setTimestamp() gonna help

and @DsrMedia also solve his problem
but i can't figure it out

please help me with this...

Here is my output video (slow motion) link

i have four video length of 0:19 so merged video length around 1:20
but you found output video length is 4:15 which is bcoz Frame-Rate

[holmes1491]

thanks for your support

I succeed in merging video

what I do is pick a file dynamically which I do hard-coded earlier

but still i got slow issue [4 file = duration 0.19 || output file = duration 1.19 which must be 1.16]
i am not sure its frame rate issue or not..

just want to you look into it once

Below code work perfectaly fine for me

public class MergeVideoService {

	private static final String mPath = "E:/eclipse/eclipse-workspace/sportanalysis/src/main/resources/static/tmp/";
	private static final String OUTPUT_FILE = mPath + "/merge/";
	
	private static File splitFiles = new File(mPath);// get all files which are to be join
	private static File fileJoinPath = new File(OUTPUT_FILE);// merge video files saved in this location

	@SuppressWarnings("resource")
	public static boolean mergeVideo() throws IOException {
		LocalTime localTime = LocalTime.now();
		String mfmt = localTime.getHour() + "_" + localTime.getMinute() + "_" + localTime.getSecond();

		// DateTimeFormatter timeFormatter = DateTimeFormatter.ofPattern("HH:mm:ss");
		// System.out.println(localTime.format(timeFormatter));
		
		if (fileJoinPath.exists()) {
			FileUtils.deleteDirectory(fileJoinPath);
			System.out.println("Delete Directory -> " + fileJoinPath.getAbsolutePath());
		}
		
		if (splitFiles.exists()) {
			File[] files = splitFiles.getAbsoluteFile().listFiles();
			if (files.length != 0) {
				System.out.println("Total files to be join: " + files.length);

				if (files.length <= 1) return false;

				avutil.av_log_set_level(avutil.AV_LOG_WARNING);//  AV_LOG_ERROR
				List<FFmpegFrameGrabber> fgs = new ArrayList<>();
				for (File f : files) {
					System.out.println("Reading the file -> "+ f.getName());
					FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(f);
					fgs.add(grabber);
					grabber.start();
				}

				if (!fileJoinPath.exists()) {
					fileJoinPath.mkdirs();
					System.out.println("Created Directory -> " + fileJoinPath.getAbsolutePath());
				}

				FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(OUTPUT_FILE+ mfmt + "_output.mp4", fgs.get(0).getImageWidth(), fgs.get(0).getImageHeight());
				recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);
				recorder.setFrameRate(fgs.get(0).getFrameRate());
				recorder.setSampleRate(fgs.get(0).getSampleRate());
				recorder.setVideoBitrate(fgs.get(0).getVideoBitrate());
				recorder.setAudioChannels(fgs.get(0).getAudioChannels());
				
				recorder.start();
				
				Frame frame;
				for (FFmpegFrameGrabber fg : fgs) {
					while ((frame = fg.grabFrame(false, true, true, false)) != null) {
						recorder.record(frame);
					}
					fg.stop();
				}

				recorder.stop();
				return true;
			}

		}
		return false;

	}

}

can you sugegest me if i miss something...

[saudet]

@holmes1491 It looks like the problem you are facing is described in issue #1096 along with a fix. If you have any questions regarding that, please post on that thread, not here.

[keerthanmahesh]

Try to call FFmpegFrameRecorder.setTimestamp() to use variable frame rate (VFR).

@saudet could you please elaborate on how to use FFmpegFrameRecorder.setTimestamp() in order to combine two video with different frame rates. I am facing the exact same issue. Your help is appreciated

[saudet]

Something like this:

    while ((frame = grabber.grabFrame()) != null) {
        recorder.setTimestamp(grabber.getTimestamp());            		
        recorder.record(frame);
    }

[keerthanmahesh]

Something like this:

    while ((frame = grabber.grabFrame()) != null) {
        recorder.setTimestamp(grabber.getTimestamp());            		
        recorder.record(frame);
    }

@saudet thank you for your reply. I have attached the code below for your reference:

`
FFmpegFrameGrabber video1Grabber = new FFmpegFrameGrabber("video-test1.mp4");
video1Grabber.start();

        FFmpegFrameGrabber video2Grabber = new FFmpegFrameGrabber("video-test2.mp4");
        video2Grabber.start();

        URL url = new URL(S3Urls.AUDIO_DREAMY);
        BufferedInputStream inputStream = new BufferedInputStream(url.openStream());
        FileOutputStream fileOS = new FileOutputStream("dreamy-audioFile.mp3");
        byte data[] = new byte[1024];
        int byteContent;
        while ((byteContent = inputStream.read(data, 0, 1024)) != -1) {
            fileOS.write(data, 0, byteContent);
        }

        FFmpegFrameGrabber audioGrabber = new FFmpegFrameGrabber("dreamy-audioFile.mp3");
        audioGrabber.start();

        FFmpegFrameRecorder audioRecorder = new FFmpegFrameRecorder("audio-output.mp3", audioGrabber.getAudioChannels());
        audioRecorder.setSampleRate(audioGrabber.getSampleRate());
        audioRecorder.start();

        Frame audioFrame;
        while ((audioFrame = audioGrabber.grab()) != null) {
            audioRecorder.record(audioFrame);
        }
        audioRecorder.stop();

        FFmpegFrameGrabber finalAudio = new FFmpegFrameGrabber("audio-output.mp3");
        finalAudio.start();

        FFmpegFrameRecorder recorder = new FFmpegFrameRecorder("video-output.mp4", 1280, 720);
        recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);
        recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);
        recorder.setSampleRate(finalAudio.getSampleRate());
        recorder.setAudioChannels(finalAudio.getAudioChannels());
        recorder.setFormat("mp4");
        recorder.start();

        Frame video1Frame;
        recorder.setFrameRate(video1Grabber.getFrameRate());
        while((video1Frame = video1Grabber.grabFrame())!= null) {
            recorder.record(video1Frame);
        }
        video1Grabber.stop();

        Frame video2Frame;
        recorder.setFrameRate(video2Grabber.getFrameRate());
        while((video2Frame = video2Grabber.grabFrame())!= null) {
            recorder.setTimestamp(video2Grabber.getTimestamp());
            recorder.record(video2Frame);
        }
        video2Grabber.stop();

        finalAudio.stop();
        recorder.stop();
        fileOS.close();
        fileOS.flush();`

I am getting this error:
Error: [mp4 @ 0x7fea7f4b8200] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 460800 >= 0

org.bytedeco.javacv.FFmpegFrameRecorder$Exception: av_interleaved_write_frame() error -22 while writing interleaved video packet.

Thank you for your assistance!

[saudet]

Don't use InputStream, not many formats of FFmpeg can support that.

[saudet]

You might also need to increase the frame rate.

[keerthanmahesh]

Video 1 has a frame rate of 30fps. Video 2 has a frame rate of 0.25 fps. If I increase the frame rate of the second video which is of 20 seconds long, the video gets over very quickly - within a few seconds, and nothing in the second part of the video will be seen. Please let me know if there is any solution to this

[saudet]

As long as you set the timestamp manually and that you use a format that supports variable frame rates, that will not happen.

[DearTan]

recorder

@saudet which format that supports variable frame rates？