Fix: B-adapt 1 should use B-frames 16

[Anton-Latukha]

Preface

There is huge number of options in x264.
I had some experience, hobby of encoding videos.
And I looked at your settings, and most of that settings don't need tweaking.

I know that most of that x264 options are provided from FFmpeg defaults.

B-adapt 1 and B-frames number is the old trope that I find on and on, and I fight against it for many-many years. Over years I saw that error even in many big video-streaming services, because system around the question has a predisposition to overlook the proper answer.

It is ideal PeerTube is Open Source and you are open for contributions. So as soon I saw that we can make the most important and effective fix for PeerTube encoding - I came to you.

Short history of B-adapt

In the invention of x264, B-frames was introduced.
x264 picked B-frames by the number in the B-frames option, and the pattern on I BBBB P BBBB ... frames.
Despite being primitive it was giving a pretty good results, comparing to MPEG2.
That is how x264 worked for a number of years.

For a long time no one could invent adaptive algorithm to pick B-frames.
In the end it was invented.

commit 68c13530b5ffc28325aee408f4cd19ab7da06715 [revision 137]
Author: Loren Merritt <pengvado@videolan.org>
Date: Wed Feb 23 19:58:02 2005 +0000

Preliminary adaptive B-frame decision (not yet tuned).
Fix flushing of delayed frames when the encode finishes.

https://www.videohelp.com/software/x264-Encoder/version-history

And for long time was in experimental stage.

commit f89e0d06700620d4e2f1467e80995f8192182496 [revision 967]
Author: Fiona Glaser <fiona@x264.com>
Date: Tue Sep 9 23:51:17 2008 -0700

Fix crash when using b-adapt at resolutions 32x32 or below.
Original patch by BugMaster, but was mostly rewritten in order to make b-adapt actually *work* at such resolutions, not merely stop crashing.

During that time, b-adapt 2 was invented.
Original thread: https://forum.doom9.org/showthread.php?t=139827

26th July 2008, 01:00 Dark Shikari x264 developer
The new B-frame decision (--bframes 16 and you)

---

One of the GSOC projects for the summer was to improve B-frame  decision. 
This was the most important project, IMO... but the student  on the task just disappeared after the first week.

So I decided to do his job for him, and with akupenguin's assistance on  the algorithm side of things, spent this week writing an optimal  trellis-like function for B-frame decision.

This algorithm provides perfect B-frame decision--or more precisely, as perfect as one can get with the current cost approximation function used.
The downside is that its a bit slower, and moreover, its speed scales linearly with --bframes.

While with the previous algorithm, --bframes 16 was only minimally slower than --bframes 3, here it will be a lot  slower.

And this new algorithm is going to be default, since it is  considerably better.
The old algorithm will still be available for  those who need a fast but inaccurate B-frame decision method.
So, this is a warning to all those who use --bframes 16, MeGUI and otherwise; the new B-frame decision is coming soon, and when it does, in the interests of speed you may want to lower the --bframes value to 3-5 or something otherwise reasonable.

commit 95ed2720b7772199f04cc9a657632107bb1c548c [revision 969]
Author: Fiona Glaser <fiona@x264.com>
Date: Sun Sep 14 18:18:15 2008 -0700

Add optional more optimal B-frame decision method
This method (--b-adapt 2) uses a Viterbi algorithm somewhat similar to that used in trellis quantization.
Note that it is not fully optimized and is very slow with large --bframes values.
It also takes into account weightb, which should improve fade detection.
Additionally, changes were made to cache lowres intra results for each frame to avoid recalculating them. This should improve performance in both B-frame decision methods.
This can also be done for motion vectors, which will dramatically improve b-adapt 2 performance when it is complete.
This patch also reads b_adapt and scenecut settings from the first pass so that the x264 header information in the output file will have correct information (since frametype decision is only done on the first pass).

Conclusion

You probably already see the pattern in messages.
The story is:
B-adapt 1 was created in heuristic way with skip-hacks and with a lot of polish on top. B-frames option does affect algorithm but not in the speed/complexity way. In fact in many cases B-frames 16 runs a tiny bit faster while producing a better results. It was also my experience, but it was years ago.

B-adapt 2 does the proper by logic the path of analysis, so the bigger the chain of B-frames allowed - the more it's tree of analysis grows.

From then-on the improvements in B-adapt 2 and B-adapt 1 go toe to toe.
But heuristic/hacked B-adapt 1 turned-out very productive.

Date: Thu Jul 28 19:34:04 2016 +0200

Adjust --preset slow

* Swap --me umh for --trellis 2. They have a similar effect on performance
but the latter gives slightly better results in most cases.
* Change --b-adapt from 2 to 1. Negligible difference in quality since the
b-adapt 1 improvements, but it's significantly faster.
Also remove a redundant assignment from veryfast (--me hex is set by default).

Maybe I need to make new tests, but I am sure as dev-logs show the old parity.

For reference:

1. Thread of creation of B-adapt 2: https://forum.doom9.org/showthread.php?t=139827
2. MEGUI Wiki:
   2.a B-adapt: https://en.wikibooks.org/wiki/MeGUI/x264_Settings#b-adapt
   2.b B-frames: https://en.wikibooks.org/wiki/MeGUI/x264_Settings#b-frames
3. Old x264 wiki page: http://www.chaneru.com/Roku/HLS/X264_Settings.htm#b-adapt
4. I submitted errata to: http://www.videorip.info/x264/78-polnoe-opisanie-vseh-kljuchej-kodirovanija-x264libx264 they had information on b-adapt/b-frames, in two places, one of them was right.
5. x264 commit logs: https://www.videohelp.com/software/x264-Encoder/version-history

[Anton-Latukha]

Waiting input.

[Anton-Latukha]

Since you have B-frames set to 3, and average video-content with hands-off approach sometimes goes to 3-7 chains in x264, I think that clear gain is ~1.5-2%.

Since x264 would use B-frames in place of forced by options P-frames previously.

Especially gain would be seen for videos with a static video content. Like: https://getjupiter.com/.

So podcasts, live cams where camera not moving, non-gaming live streams, conferences, presentations, discussions, debates and so on, where the is less need for frequent P frames. There gain can be 2-4%.

[Anton-Latukha]

Also lets from the beginning distinguish the difference in ref frames option and B-frames.

Ref frames increase the complexity of interlinks in the frames (between frames) and so in the chain. Primarily it impacts complexity of P-frames, and in lesser extend B-frames. And so ref option impacts encoding/decoding complexity significantly.

Raising ref frames steeply increase the encoding computation requirements,

MPEG4 Levels refer to ref frames option. Because option raises a requirements to the decoding complexity (somewhat) and primarily frame buffer requirements - DPB (Decoded Picture Buffer). And H.264 decoding hardware has frame buffers implemented in hardware. And hardware buffer has it's physical size, so that is why they have strict requirement/support of Levels. If P-frame refers to the frames that does not got in the buffer - P-frame gets dropped/partly decoded, and so it impacts B-frames that rely to it.

That is about ref frames.

====

B-frames are not ref frames option. There is no limit on length of chain of B-frames in the standards, the length of B-frame chains does not affect decoding much.

Permitting longer B-frames chains offsets it's tiny increase in decoding complexity by eliminating P-frames complexity (P-frames have the peak major complexity during decoding), so usage of B-frames sometimes even can decrease decoding complexity overall.

The longer B-frames chain x264 detects - the better bit-compression we get. If in the shot - picture is still or moves gradually, naturally - we don't need frequent P-frames, B-frames would flow referring to their neighbors.

[rigelk]

Thanks @Anton-Latukha! That was very informative, to say the least.

Could you edit you PR so that is passes the linting test?

Also, it would be nice if you could come to our IRC channel. We have some interesting discussions about encoding there 🙂

[Anton-Latukha]

Ok, I am new to TypeScript. I would pass lint test.

Then if there is instructions - I can deduct and write some asserts, if - else. But if there is more - you can run with it.

[Anton-Latukha]

Because I did not found your IRC channel - I got into Jupiter Broadcasting telegram channel, there guys contributing Ops/DevOps to the studio infrastructure (https://getjupiter.com), and also created a community instance (https://peertube.linuxrocks.online). I ran some tests there and would report back, as soon as would ask and receive logs.

[Anton-Latukha]

So at first we must solve this issue: #779

[Anton-Latukha]

And then the issue of that I can not find IRC of PeerTube 8]

[tcitworld]

IRC chan is #peertube on Freenode, and #peertube:matrix.org on Matrix.

[Anton-Latukha]

Matrix! Nice.

_{Somehow I searched and not found peertube on matrix first time, still they have strange interface}

Also I had dramatic entrance on Matrix, and was accused of dumping a CV, while I just talked about my beneficial strengths, and at first glance to chatroom shared all info I thought useful to post in first message.

So now first IRC impression - I am sad 8[

But you encouraged me there, which normalized my experience there.

[Anton-Latukha]

I want to wait on when Jupiter Broadcasting shares me their source footage, so I can test both on in and on: https://media.xiph.org/ footage - to get and show the real facts in real numbers.

I would go into compression testing in a bulk, so I would test tweaks on different content.

I switch to other tasks, I would resume, when I would receive sample of production content from JB, and would go into gathering bulk video encoding stats/tests. Or if someone wants - can run away with tests himself, obviously.

[Anton-Latukha]

It may be cool to create a thread on doom9 forum asking for help to tune codecs for Open Source federated p2p streaming platform. That maniacs going to tear any encoding topic apart, and would be glad to contribute their old-school ways to optimize codecs for content types.

[rezonant]

So the remaining linting issues:

ERROR: /home/travis/build/Chocobozzz/PeerTube/server/helpers/ffmpeg-utils.ts[92, 1]: jsdoc is not formatted correctly on this line
ERROR: /home/travis/build/Chocobozzz/PeerTube/server/helpers/ffmpeg-utils.ts[92, 1]: asterisks in jsdoc must be aligned
ERROR: /home/travis/build/Chocobozzz/PeerTube/server/helpers/ffmpeg-utils.ts[93, 1]: jsdoc is not formatted correctly on this line
ERROR: /home/travis/build/Chocobozzz/PeerTube/server/helpers/ffmpeg-utils.ts[93, 1]: asterisks in jsdoc must be aligned
ERROR: /home/travis/build/Chocobozzz/PeerTube/server/helpers/ffmpeg-utils.ts[96, 1]: jsdoc is not formatted correctly on this line
ERROR: /home/travis/build/Chocobozzz/PeerTube/server/helpers/ffmpeg-utils.ts[96, 1]: asterisks in jsdoc must be aligned

[rigelk]

@rezonant @Anton-Latukha otherwise I can just integrate the changes myself. I already have a PR related to encoding, so I can add your options with mine: e0e2bed.

[Anton-Latukha]

Ok. Real results:

Input

I ran fast-assembled encoding test on classic test - Big Buck Bunny raw source footage. It is an animation test.

ffmpeg -i ./big_buck_bunny_1080p24.y4m -c:v libx264 -crf 23 -movflags faststart output-ref3-badapt1-bf16.mkv

ffmpeg -i ./big_buck_bunny_1080p24.y4m -c:v libx264 -crf 23 -bf 16 -movflags faststart output-ref3-badapt1-bf16.mkv

Results

The difference is so massive and obvious, that I do not need other tests.

-bf 3 (default setting):

[libx264 @ 0x762920] frame I:151   Avg QP:18.05  size:209171
[libx264 @ 0x762920] frame P:4176  Avg QP:21.82  size: 30327
[libx264 @ 0x762920] frame B:9988  Avg QP:25.31  size:  6169
[libx264 @ 0x762920] consecutive B-frames:  3.9%  5.8%  9.8% 80.4%
[libx264 @ 0x762920] kb/s:2948.67
1734.58user 20.99system 7:52.95elapsed 371%CPU (0avgtext+0avgdata 689144maxresident)

We already can see, that B chain (3.9% 5.8% 9.8% 80.4%) floores in ~80% of cases, it wants to go further, but default setting does not allow it. So I already expected massive change. Interesting why this still is not fixed in FFMpeg. I probably would raise a issue for them. Or ask libav folks, they are also HandBrake devs, and they have other defaults.

-bf 16:

[libx264 @ 0x22cb9e0] frame I:151   Avg QP:17.53  size:220066
[libx264 @ 0x22cb9e0] frame P:2624  Avg QP:21.98  size: 37530
[libx264 @ 0x22cb9e0] frame B:11540 Avg QP:25.66  size:  6766
[libx264 @ 0x22cb9e0] consecutive B-frames:  3.7%  5.3%  9.1%  7.3%  5.9% 11.4%  4.9%  8.4%  4.1%  4.7%  3.3%  4.5%  3.3%  3.0%  2.5%  2.6% 15.9%
[libx264 @ 0x22cb9e0] kb/s:2813.70
1888.44user 22.45system 7:17.72elapsed 436%CPU (0avgtext+0avgdata 1204828maxresident)k
84877520inputs+422456outputs (0major+23962minor)pagefaults 0swaps

B-frame chains

We can see -bf 16 permits longer B-chains.

[libx264 @ 0x22cb9e0] consecutive B-frames:  3.7%  5.3%  9.1%  7.3%  5.9% 11.4%  4.9%  8.4%  4.1%  4.7%  3.3%  4.5%  3.3%  3.0%  2.5%  2.6% 15.9%

Also we see that P-frames usage changed to B-frames usage change:

-bf 3
[libx264 @ 0x762920] frame P:4176  Avg QP:21.82  size: 30327
[libx264 @ 0x762920] frame B:9988  Avg QP:25.31  size:  6169
vs
-bf 16
[libx264 @ 0x22cb9e0] frame P:2624  Avg QP:21.98  size: 37530
[libx264 @ 0x22cb9e0] frame B:11540 Avg QP:25.66  size:  6766

1 - (2624/4176) = 0.3716475095785441 -> ~37.2% reduction of P-frames usage.
(11540/9988) - 1 = 0.1553864637565079 -> ~15.6% increase of B-frames usage (we have bigger pool of B-frames then P-frames, that is why 1500 P-frames reduction of ~37.2% increased B pool on ~15.6%)

Encoding time

So we see the time of encoding reduced mainly because it encodes much less P frames (it used to floor B-frames to 3, so x264 was forced to encode P, and they have more encoding/decoding complexity). I-frames number was the same, because algorithm of their detection does not depend on B-frames or P-frames, but on input from the source.

The speed-up of encoding we got: 0.09x, or 9%, which is massive gain number in encoding.

Compression

In stream metric

We got 2950.8kbits/s - 2815.8kbits/s = 134.97 kbits/s stream reduction! That is the whole supplementary audio stream right there.

In filesize metric

1-(204975kB/214802kB) = 4.58% decrease of file size.
Or in MBytes, we got 1 MByte/minute savings.

Decoding benchmark

-bf 3

frame=14315 fps=615 q=-0.0 Lsize=N/A time=00:09:56.45 bitrate=N/A speed=25.6x
bench: utime=138.506s
bench: maxrss=110292kB

-bf 16

frame=14315 fps=592 q=-0.0 Lsize=N/A time=00:09:56.45 bitrate=N/A speed=24.7x
video:7381kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
bench: utime=142.228s
bench: maxrss=135544kB

-bf 16 decoding took (142.228/138.506) - 1 = 2.7% more CPU time.

So, lets sum-up:

B-frames 16 give:

* +9% to encoding speed
* +2.7% decoding complexity 
* +4.58% better compression

As you know - most times users have playback problems due to networking. And +4.58% better compression gives much better user expirience bump, then +2.7% of decoding complexity. Like mobile phone users - most of the time users have that 3% of CPU while playing video around, but they often rely on mobile network coverage. And 3%-5% reduction means a smoother/faster loading playback to them.

And +9% to encoding speed.

So we got what I said in previous messages.

Encoding logs

B-frames 3

Output #0, matroska, to 'output-ref3-badapt1-bf3.mkv':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: h264 (libx264) (H264 / 0x34363248), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 24 fps, 1k tbn, 24 tbc
    Metadata:
      encoder         : Lavc57.107.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame=14315 fps= 30 q=-1.0 Lsize=  214802kB time=00:09:56.33 bitrate=2950.8kbits/s speed=1.27x      
video:214693kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.050843%
[libx264 @ 0x762920] frame I:151   Avg QP:18.05  size:209171
[libx264 @ 0x762920] frame P:4176  Avg QP:21.82  size: 30327
[libx264 @ 0x762920] frame B:9988  Avg QP:25.31  size:  6169
[libx264 @ 0x762920] consecutive B-frames:  3.9%  5.8%  9.8% 80.4%
[libx264 @ 0x762920] mb I  I16..4: 19.0% 62.7% 18.4%
[libx264 @ 0x762920] mb P  I16..4:  1.9%  7.3%  0.8%  P16..4: 27.2%  8.1%  5.6%  0.0%  0.0%    skip:49.1%
[libx264 @ 0x762920] mb B  I16..4:  0.3%  0.9%  0.1%  B16..8: 23.1%  1.7%  0.4%  direct: 1.1%  skip:72.3%  L0:41.4% L1:53.3% BI: 5.2%
[libx264 @ 0x762920] 8x8 transform intra:69.8% inter:74.9%
[libx264 @ 0x762920] coded y,uvDC,uvAC intra: 55.9% 60.8% 28.5% inter: 7.9% 8.0% 1.1%
[libx264 @ 0x762920] i16 v,h,dc,p: 26% 19%  4% 51%
[libx264 @ 0x762920] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 14% 18%  5%  8% 13%  7%  8%  7%
[libx264 @ 0x762920] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 14% 12%  6% 11% 12%  8%  7%  6%
[libx264 @ 0x762920] i8c dc,h,v,p: 52% 19% 17% 11%
[libx264 @ 0x762920] Weighted P-Frames: Y:4.9% UV:3.3%
[libx264 @ 0x762920] ref P L0: 63.9% 18.1% 12.0%  6.0%  0.1%
[libx264 @ 0x762920] ref B L0: 84.8% 12.4%  2.8%
[libx264 @ 0x762920] ref B L1: 96.3%  3.7%
[libx264 @ 0x762920] kb/s:2948.67
1734.58user 20.99system 7:52.95elapsed 371%CPU (0avgtext+0avgdata 689144maxresident)k
84964792inputs+444136outputs (0major+23772minor)pagefaults 0swaps

B-frames 16

Output #0, matroska, to 'output-ref3-badapt1-bf16.mkv':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: h264 (libx264) (H264 / 0x34363248), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 24 fps, 1k tbn, 24 tbc
    Metadata:
      encoder         : Lavc57.107.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame=14315 fps= 33 q=-1.0 Lsize=  204975kB time=00:09:56.33 bitrate=2815.8kbits/s speed=1.36x       
video:204866kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.053076%
[libx264 @ 0x22cb9e0] frame I:151   Avg QP:17.53  size:220066
[libx264 @ 0x22cb9e0] frame P:2624  Avg QP:21.98  size: 37530
[libx264 @ 0x22cb9e0] frame B:11540 Avg QP:25.66  size:  6766
[libx264 @ 0x22cb9e0] consecutive B-frames:  3.7%  5.3%  9.1%  7.3%  5.9% 11.4%  4.9%  8.4%  4.1%  4.7%  3.3%  4.5%  3.3%  3.0%  2.5%  2.6% 15.9%
[libx264 @ 0x22cb9e0] mb I  I16..4: 18.4% 62.3% 19.3%
[libx264 @ 0x22cb9e0] mb P  I16..4:  2.8% 10.5%  1.2%  P16..4: 29.4%  8.9%  5.9%  0.0%  0.0%    skip:41.4%
[libx264 @ 0x22cb9e0] mb B  I16..4:  0.3%  1.0%  0.1%  B16..8: 21.1%  2.0%  0.4%  direct: 1.3%  skip:73.7%  L0:43.4% L1:50.9% BI: 5.6%
[libx264 @ 0x22cb9e0] 8x8 transform intra:69.6% inter:75.6%
[libx264 @ 0x22cb9e0] coded y,uvDC,uvAC intra: 56.6% 61.5% 29.4% inter: 7.3% 6.8% 0.7%
[libx264 @ 0x22cb9e0] i16 v,h,dc,p: 26% 18%  4% 52%
[libx264 @ 0x22cb9e0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 14% 17%  5%  8% 13%  8%  8%  7%
[libx264 @ 0x22cb9e0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 14% 12%  6% 11% 12%  8%  7%  6%
[libx264 @ 0x22cb9e0] i8c dc,h,v,p: 52% 19% 18% 11%
[libx264 @ 0x22cb9e0] Weighted P-Frames: Y:7.4% UV:5.3%
[libx264 @ 0x22cb9e0] ref P L0: 65.2% 18.0% 11.7%  4.9%  0.1%
[libx264 @ 0x22cb9e0] ref B L0: 86.7% 11.8%  1.5%
[libx264 @ 0x22cb9e0] ref B L1: 96.0%  4.0%
[libx264 @ 0x22cb9e0] kb/s:2813.70
1888.44user 22.45system 7:17.72elapsed 436%CPU (0avgtext+0avgdata 1204828maxresident)k
84877520inputs+422456outputs (0major+23962minor)pagefaults 0swaps

Now if you would allow me, please I would finish my pull request. At least to learn and understand TypeScript better, maybe I would start committing some simple additions in it, or understand TypeScript processing more.

Sorry for some time spent waiting, as I was getting the facts straight, so we all can be sure that this global change is in right direction.

[Anton-Latukha]

I forgot to do SSIM/PSNR check. But as we use default -crf 23 in both cases - SSIM/PSNR would be very close, because -crf 23 defines that level of data-noise.

[rigelk]

@Anton-Latukha thanks for the extensive debug! 😮

Of course we can let you correct your PR. If you want to use the linter locally on your machine to have feedback, you can use yarn run travis -- lint. It will show you the errors of style Typescript detected.

Basically this is just your way of formatting the comments, à la JSDoc, which is not correct JSDoc formatting. Simple comments without asterisks would not make the linter show errors 🙂

[rigelk]

LGTM! Thanks for the hard work @Anton-Latukha

[Anton-Latukha]

Thank you for PeerTube.

[Anton-Latukha]

This change in the case of streaming

Often, changing video-encoding settings leads to handicapping of streaming.

I decided also to mention thoughts relating to this, as this is important factor here.

Also I see no problems with big B-frame chains for rewinding/streaming.

In the middle of stream playback recovers info from I-frame anyway. As I-frame it is the only solid starting point to calculate from. And first x264 determines places of I-frames, and then P&B-frames placing depends on I-frame place. We saw that I-frames does not depend on change we do.

Also, if player positions only on I frames - again, they are not touched by this change.
If player permits placing on I&P - P frames are maximally 16 frames apart, which is less then a second. Which seems sufficient precision.

And if player permits placing on I&P&B - there is no difference for that player what frame is under the timestamp.

And so with data downloaded during rewind/coming in the middle of stream - player always does the same. Searches for I-frame, and calculates video data from it, to position it needs.
And since we made stream leaner - needed video data now downloads a tiny bit faster.

[emansom]

I'm in the progress of introducing hardware acceleration support for PeerTube and came across this.

I have a feeling this very commit has caused videos on PeerTube to come across the viewer very blocky (example here). Was any VMAF testing done before and after?

It may also be bitrate related? I'm a novice when it comes to ffmpeg, do enlighten me if my contemplation is incorrect.