When using automatic bitrate selection, if --max-bitrate goes over threshold video will be encoded on lowest possible settings.

[DimkaTsv]

Issue description:
I tried to write some batch script so i could choose rate control and bitrates dynamically and not locked to one script per configuration. [And it actually works, surprisingly for me]
To avoid errors when i use no input for any value that assumes one, i default it to 0 (if possible, if not i default it to one that is default by spec). Bitrate and --max-bitrate is one such thing.
But when i tried to run script i found some discrepancy between simple one i had to test some values and "frankenstein" on.
When i used bitrate=0 and max_bitrate=0 i got basically copressed to nothingness file (up to 51 avg QP and 500 kbps)

After some investigation i came to conclusion that this bug triggers only if automatic! selection of --max-bitrate resulted in 480000 or 800000 kbps.

Reproduction:
This happens only if you do these combinations.
--codec hevc --profile main --tier high --level 6.1/6.2 --vbr 0 --max-bitrate 0
Preset doesn't matter. Value of 0 for max-bitrate ismandatory for reproduction (but for bitrate it is not). Only level 6.1/6.2 with combination of --tier high results in --max-bitrate limit of 480000+ which causes problems
It is a bug, because if you set max-bitrate manually, everything works fine. Even if you set it at 800000

Logs:

Bugged condition:
--avsw --codec hevc --profile main --tier high --level 6.2 --preset slow --vbr 0 --max-bitrate 0 --audio-copy --chapter-copy --log vceencc_log.txt --log-level info
vceencc_log.txt

C:\Users\-----\Videos\test sample>C:\Users\-----\Desktop\Utilites\VCE_Encoder\VCEEncC_x64\\VCEEncC64.exe -i "C:\Users\-----\Videos\test sample\F003C013_190925_MN99.mp4" -o "F003C013_190925_MN99_processed.mp4" --avsw --codec hevc --profile main --tier high --level 6.2 --preset slow --vbr 0 --max-bitrate 0 --audio-copy --chapter-copy --log vceencc_log.txt --log-level info
--------------------------------------------------------------------------------
F003C013_190925_MN99_processed.mp4
--------------------------------------------------------------------------------
VCEEnc (x64) 8.21 (r1577) by rigaya, Dec 10 2023 04:01:37 (VC 1937/Win)
OS:            Windows 11 x64 (22631) [UTF-8]
CPU:           AMD Ryzen 7 5800X3D 8-Core Processor [4.48GHz] (8C/16T)
GPU:           AMD Radeon RX 7800 XT [31.0.24019.1006]
AMF:           Runtime 1.4.33 / SDK 1.4.30
Input Info:    avsw: prores(yuv444(12bit))->nv12 [AVX2], 4480x1856, 24/1 fps
Output:        H.265/HEVC  main @ Level 6.2 (high tier)
               4480x1856p 0:0 24.000fps (24/1fps)
               avwriter: hevc, pcm_s24le, pcm_s24le, pcm_s24le, pcm_s24le,
                pcm_s24le => mp4
Quality:       slow
VBR:           20000 kbps
Max bitrate:   800000 kbps
QP:            Min: 0:0, Max: 51:51
VBV Bufsize:   500000 kb
Bframes:       0 frames
Pre Analysis:  off
Ref frames:    2 frames
LTR frames:    0 frames
Motion Est:    Q-pel
Slices:        1
GOP Len:       240 frames
Others:        skipframe:on deblock

encoded 271 frames, 50.31 fps, 500.74 kbps, 0.67 MB
encode time 0:00:05, CPU: 68.1%, GPU: 0.8%, VE: 59.8%
frame type IDR   2
frame type I     2,  avgQP  51.00,  total size  0.01 MB
frame type P   269,  avgQP  51.00,  total size  0.66 MB

Normal condition:
--avsw --codec hevc --profile main --tier high --level 6.2 --preset slow --vbr 0 --max-bitrate 800000 --audio-copy --chapter-copy --log vceencc_log.txt --log-level info
vceencc_log.txt

C:\Users\-----\Videos\test sample>C:\Users\-----\Desktop\Utilites\VCE_Encoder\VCEEncC_x64\\VCEEncC64.exe -i "C:\Users\-----\Videos\test sample\F003C013_190925_MN99.mp4" -o "F003C013_190925_MN99_processed.mp4" --avsw --codec hevc --profile main --tier high --level 6.2 --preset slow --vbr 0 --max-bitrate 800000 --audio-copy --chapter-copy --log vceencc_log.txt --log-level info
--------------------------------------------------------------------------------
F003C013_190925_MN99_processed.mp4
--------------------------------------------------------------------------------
VCEEnc (x64) 8.21 (r1577) by rigaya, Dec 10 2023 04:01:37 (VC 1937/Win)
OS:            Windows 11 x64 (22631) [UTF-8]
CPU:           AMD Ryzen 7 5800X3D 8-Core Processor [4.48GHz] (8C/16T)
GPU:           AMD Radeon RX 7800 XT [31.0.24019.1006]
AMF:           Runtime 1.4.33 / SDK 1.4.30
Input Info:    avsw: prores(yuv444(12bit))->nv12 [AVX2], 4480x1856, 24/1 fps
Output:        H.265/HEVC  main @ Level 6.1 (high tier)
               4480x1856p 0:0 24.000fps (24/1fps)
               avwriter: hevc, pcm_s24le, pcm_s24le, pcm_s24le, pcm_s24le,
                pcm_s24le => mp4
Quality:       slow
VBR:           20000 kbps
Max bitrate:   800000 kbps
QP:            Min: 0:0, Max: 51:51
VBV Bufsize:   20000 kb
Bframes:       0 frames
Pre Analysis:  off
Ref frames:    2 frames
LTR frames:    0 frames
Motion Est:    Q-pel
Slices:        1
GOP Len:       240 frames
Others:        skipframe:on deblock

encoded 271 frames, 51.03 fps, 20434.83 kbps, 27.51 MB
encode time 0:00:05, CPU: 67.2%, GPU: 0.8%, VE: 65.0%
frame type IDR   2
frame type I     2,  avgQP  20.00,  total size   0.51 MB
frame type P   269,  avgQP  18.68,  total size  26.99 MB

---

Also... Few notes for documentation to @rigaya
AMD GPU's (at least currently) do not support main444 profile for h.264 and hevc [VCN does not support YUV444 encoding (or decoding as well)]

AV1 encoding level currently (based on 7800XT) is capped at 6 [aka 6.0 specifically... On 6.1 or 6.2 it reports to me big fat "You shall not pass"]. Maybe there is higher level capability for 7900XTX?, but not sure. Maybe some note that not every encoder will be able to do full range would be good? Moreover iirc levels of 7+ are not specified in parameters even.

AMF now supports all 7 rate control modes for all three of AVC HEVC and AV1 based on documentation. CQP, CBR, VBR, VBR_LAT, QVBR, HQVBR, HQCBR

For QVBR setting bitrate is basically useless. It uses --qvbr-quality as only guideline basically. Should it still be used?

Ah yeah... And --vpp-resize advanced also throws this error:

Error: Invalid value "advanced" for "--vpp-resize"
  Option value should be one of below...
    auto, bilinear, bicubic, spline16, spline36, spline64, lanczos2, lanczos3,
    lanczos4, amf_bilinear, amf_bicubic, amf_fsr

[rigaya]

Thank you for the detailed report, I've found ways to avoid this issue, and will have it fixed in the next release.

[rigaya]

The issue shall be fixed with VCEEnc 8.22.

[DimkaTsv]

Fix confirmed, now VBV buffersize is equal to set target bitrate, and bug doesn't trigger with specified in issue condition.
(i also just noted that true reason for this was VBV buffer size, heh... I am kinda blind sometimes)

Closing issue.