Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug][DG2][VPP]: tonemap_vaapi extremely slow on certain input size 1280*(722 vs 720) #1628

Closed
nyanmisaka opened this issue Mar 13, 2023 · 19 comments · Fixed by #1646
Closed
Assignees
Labels
VP Video Processing

Comments

@nyanmisaka
Copy link
Contributor

Which component impacted?

Video Processing

Is it regression? Good in old configuration?

No, this issue exist a long time

What happened?

  1. Linux 6.2.5, FFmpeg 5/6/cartwheel, Arc A380, ReBAR On
  2. libva 2.17, gmmlib 22.3.4, media-driver 23.1.3
  3. Run FFmpeg cli and watch the fps:

Two bad resolutions: 1280x722 and 1280x692 - Results in very slow FPS - < 5 FPS.

frame=   22 fps=3.8 q=-0.0 Lsize=N/A time=00:00:00.35 bitrate=N/A speed=0.0606x

But 1280x720 and 1280x694 works just fine - hundreds FPS.

Other resolutions may also affected but I haven't found the pattern yet.

ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i ANY_10Bit_HDR_CLIP.mp4 -vf scale_vaapi=1280:722,tonemap_vaapi=format=nv12:t=bt709:m=bt709:p=bt709 -f null -
ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i ANY_10Bit_HDR_CLIP.mp4 -vf scale_vaapi=1280:692,tonemap_vaapi=format=nv12:t=bt709:m=bt709:p=bt709 -f null -

What's the usage scenario when you are seeing the problem?

Transcode for media delivery

What impacted?

HDR10 to SDR tone-mapping in media delivery.

Debug Information

  1. libva 2.17, gmmlib 22.3.4, media-driver 23.1.3
  2. yes renderD128
  3. vainfo.log
  4. libva_trace.log.141251.zip
  5. no i915 hang

Guess it's due to the incorrect image size alignment on DG2 platform?
Older platforms are not affected such as KBLx and TGLx.

Other common VPP filters are not affected too.

 ... deinterlace_vaapi V->V       Deinterlacing of VAAPI surfaces
 ... denoise_vaapi     V->V       VAAPI VPP for de-noise
 ... overlay_vaapi     VV->V      Overlay one video on top of another
 ... procamp_vaapi     V->V       ProcAmp (color balance) adjustments for hue, saturation, brightness, contrast
 ... scale_vaapi       V->V       Scale to/from VAAPI surfaces.
 ... sharpness_vaapi   V->V       VAAPI VPP for sharpness
 ... transpose_vaapi   V->V       VAAPI VPP for transpose

Do you want to contribute a patch to fix the issue?

None

@nyanmisaka
Copy link
Contributor Author

@FurongZhang May need to trouble you to look into it. Thanks in advance.

@Jexu Jexu added the VP Video Processing label Mar 21, 2023
@intel-mediadev
Copy link
Contributor

Auto Created VSMGWL-62709 for further analysis.

@eero-t
Copy link

eero-t commented Mar 22, 2023

Does not look like an alignment issue, at least not a straight-forward one. While fast 720 height is divisible by 16, 694 is only by 2, whereas slow 692 height is divisible by 4...

@nyanmisaka
Copy link
Contributor Author

Indeed. So I haven't found the pattern of this issue.
Common resolutions works just fine such as 1280x720, 1920x1080 and 3840x2160.

@FurongZhang
Copy link
Contributor

@nyanmisaka , well received the information, will update the status ww13.

@nyanmisaka
Copy link
Contributor Author

@nyanmisaka , well received the information, will update the status ww13.

Any news on this? I see that the LUT media kernel has been updated recently but it doesn't seem to help this issue.

@FurongZhang
Copy link
Contributor

The fix code #1646 has been merged and upstreamed.

@nyanmisaka
Copy link
Contributor Author

@FurongZhang Tested and it works just fine now. Thanks for your help!

@FurongZhang
Copy link
Contributor

@nyanmisaka , glad to know that. You are very welcome.

@almightiest
Copy link

almightiest commented May 23, 2023

@nyanmisaka I built media-driver 23.2.2 (which contains this fix), installed it in the OS, then replaced /usr/lib/jellyfin-ffmpeg/lib/dri/iHD_drv_video.so with the newly built version (build_media/media_driver/iHD_drv_video.so), restarted the service, played a cropped 4k HDR (height divisible by 2) needing tone mapping, and I am still running into this issue.

I am transcoding on an i7-13700k cpu/igpu.

I am on jellyfin-ffmpeg-5.1.3-1-jammy.

The input resolution of the file I'm having trouble with is 3840x1604 and the output resolution is 1920x802.

sudo -u jellyfin /usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -init_hw_device vaapi=va:,driver=iHD,kernel_driver=i915 -init_hw_device qsv=qs@va -init_hw_device opencl=ocl@va -filter_hw_device qs -hwaccel vaapi -hwaccel_output_format vaapi -autorotate 0 -canvas_size 1920x802 -i file:"/path/to/video.mkv" -autoscale 0 -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:0 -codec:v:0 h264_qsv -low_power 1 -preset veryfast -look_ahead 0 -b:v 5616000 -maxrate 5616000 -bufsize 11232000 -g:v:0 72 -keyint_min:v:0 72 -filter_complex "[0:3]scale=flags=fast_bilinear,format=bgra,hwupload=derive_device=qsv:extra_hw_frames=64[sub];[0:0]setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,scale_vaapi=w=1920:h=802:extra_hw_frames=24,procamp_vaapi=b=0:c=1.2,tonemap_vaapi=format=nv12:p=bt709:t=bt709:m=bt709:extra_hw_frames=32,hwmap=derive_device=qsv,format=qsv[main];[main][sub]overlay_qsv=eof_action=endall:shortest=1:repeatlast=0:w=1920:h=802" -start_at_zero -codec:a:0 libfdk_aac -ac 2 -ab 384000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/mnt/ramdisk/jellyfin/transcodes/25ecaf0b52d0e4389921627d2959e5c2%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/mnt/ramdisk/jellyfin/transcodes/25ecaf0b52d0e4389921627d2959e5c2.m3u8"

I'm sure it's me, just wondering if I missed a way to deploy this change.

If there's a better place to discuss this, apologies - couldn't find a related issue in the jellyfin-ffmpeg or jellyfin github projects.

@nyanmisaka
Copy link
Contributor Author

If there's a better place to discuss this, apologies - couldn't find a related issue in the jellyfin-ffmpeg or jellyfin github projects.

We can discuss here in jellyfin repo jellyfin/jellyfin-ffmpeg#245

@FurongZhang
Copy link
Contributor

?? Need to update driver. We fixed this issue in driver.

@nyanmisaka
Copy link
Contributor Author

nyanmisaka commented May 25, 2023

@FurongZhang It's a different issue on RPL-S / 13th Gen. Using tonemap_vaapi hangs and resets the i915 kernel driver.

[    2.700921] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adls_dmc_ver2_01.bin (v2.1)
[    2.704972] i915 0000:00:02.0: [drm] GuC firmware i915/tgl_guc_70.6.4.bin version 70.6.4
[    2.704974] i915 0000:00:02.0: [drm] HuC firmware i915/tgl_huc_7.9.3.bin version 7.9.3
[    2.719629] i915 0000:00:02.0: [drm] HuC authenticated
[    2.719633] i915 0000:00:02.0: [drm] GuC submission disabled
[    2.719636] i915 0000:00:02.0: [drm] GuC SLPC disabled
[    2.722123] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[    2.722747] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 1
[    2.780112] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    2.844668] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[   16.962506] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[   16.967171] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[   16.969761] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
[65426.599038] i915 0000:00:02.0: [drm] Resetting vecs0 for preemption time out
[65426.599107] i915 0000:00:02.0: [drm] ffmpeg[1963716] context reset due to GPU hang
[65426.751730] i915 0000:00:02.0: [drm] GPU HANG: ecode 12:8:2cffbffd, in ffmpeg [1963716]
[65433.158859] i915 0000:00:02.0: [drm] Resetting vecs0 for preemption time out
[65433.158928] i915 0000:00:02.0: [drm] ffmpeg[1963716] context reset due to GPU hang
[65433.311510] i915 0000:00:02.0: [drm] GPU HANG: ecode 12:8:2cffbffd, in ffmpeg [1963716]

Edit:
tonemap_vaapi won't hang the GPU driver but it will fail shortly.

frame= 0 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 0 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 0 fps=0.0 q=11.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 2 fps=0.6 q=25.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 4 fps=1.0 q=25.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 6 fps=1.2 q=25.0 size=N/A time=00:00:00.25 bitrate=N/A speed=0.0504x
frame= 8 fps=1.3 q=25.0 size=N/A time=00:00:00.25 bitrate=N/A speed=0.0423x
frame= 10 fps=1.4 q=25.0 size=N/A time=00:00:00.25 bitrate=N/A speed=0.0365x
frame= 11 fps=1.5 q=25.0 size=N/A time=00:00:00.64 bitrate=N/A speed=0.085x
[Parsed_tonemap_vaapi_6 @ 0x557580bb0440] Failed to start picture processing: 1 (operation failed).
Error while filtering: Input/output error
Failed to inject frame into filter network: Input/output error

@almightiest
Copy link

almightiest commented May 25, 2023

@FurongZhang It's a different issue on ADL-S / 13th Gen. Using tonemap_vaapi hangs and resets the i915 kernel driver.

[    2.700921] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adls_dmc_ver2_01.bin (v2.1)
[    2.704972] i915 0000:00:02.0: [drm] GuC firmware i915/tgl_guc_70.6.4.bin version 70.6.4
[    2.704974] i915 0000:00:02.0: [drm] HuC firmware i915/tgl_huc_7.9.3.bin version 7.9.3
[    2.719629] i915 0000:00:02.0: [drm] HuC authenticated
[    2.719633] i915 0000:00:02.0: [drm] GuC submission disabled
[    2.719636] i915 0000:00:02.0: [drm] GuC SLPC disabled
[    2.722123] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[    2.722747] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 1
[    2.780112] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    2.844668] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[   16.962506] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[   16.967171] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[   16.969761] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
[65426.599038] i915 0000:00:02.0: [drm] Resetting vecs0 for preemption time out
[65426.599107] i915 0000:00:02.0: [drm] ffmpeg[1963716] context reset due to GPU hang
[65426.751730] i915 0000:00:02.0: [drm] GPU HANG: ecode 12:8:2cffbffd, in ffmpeg [1963716]
[65433.158859] i915 0000:00:02.0: [drm] Resetting vecs0 for preemption time out
[65433.158928] i915 0000:00:02.0: [drm] ffmpeg[1963716] context reset due to GPU hang
[65433.311510] i915 0000:00:02.0: [drm] GPU HANG: ecode 12:8:2cffbffd, in ffmpeg [1963716]

enable_guc=2 was the culprit to the GPU HANG issues, but there's still an issue even with the rocket lake default enable_guc=3 to enable GuC submissions.

With the latest media-driver from master, I still only get <1 fps when transcoding a 3840x1608 movie with VPP tonemapping. Then it fails with an Input/Output error.

$ ffmpeg -analyzeduration 200M -init_hw_device vaapi=va:,driver=iHD,kernel_driver=i915 -init_hw_device qsv=qs@va -init_hw_device opencl=ocl@va -filter_hw_device qs -hwaccel vaapi -hwaccel_output_format vaapi -autorotate 0 -canvas_size 1920x802 -i file:"/path/to/file.mkv" -autoscale 0 -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:0 -codec:v:0 h264_qsv -low_power 1 -preset veryfast -look_ahead 0 -b:v 5616000 -maxrate 5616000 -bufsize 11232000 -g:v:0 72 -keyint_min:v:0 72 -filter_complex "[0:3]scale=flags=fast_bilinear,format=bgra,hwupload=derive_device=qsv:extra_hw_frames=64[sub];[0:0]setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,scale_vaapi=w=1920:h=802:extra_hw_frames=24,procamp_vaapi=b=0:c=1.2,tonemap_vaapi=format=nv12:p=bt709:t=bt709:m=bt709:extra_hw_frames=32,hwmap=derive_device=qsv,format=qsv[main];[main][sub]overlay_qsv=eof_action=endall:shortest=1:repeatlast=0:w=1920:h=802" -start_at_zero -codec:a:0 libfdk_aac -ac 2 -ab 384000 -af "volume=2" -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/mnt/ramdisk/jellyfin/transcodes/b30a53cbab7ab145862668b3fc0d0941%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/mnt/ramdisk/jellyfin/transcodes/b30a53cbab7ab145862668b3fc0d0941.m3u8"

...
frame= 0 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 0 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 0 fps=0.0 q=11.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 2 fps=0.6 q=25.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 4 fps=1.0 q=25.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 6 fps=1.2 q=25.0 size=N/A time=00:00:00.25 bitrate=N/A speed=0.0504x
frame= 8 fps=1.3 q=25.0 size=N/A time=00:00:00.25 bitrate=N/A speed=0.0423x
frame= 10 fps=1.4 q=25.0 size=N/A time=00:00:00.25 bitrate=N/A speed=0.0365x
frame= 11 fps=1.5 q=25.0 size=N/A time=00:00:00.64 bitrate=N/A speed=0.085x
[Parsed_tonemap_vaapi_6 @ 0x557580bb0440] Failed to start picture processing: 1 (operation failed).
Error while filtering: Input/output error
Failed to inject frame into filter network: Input/output error
Error while processing the decoded data for stream #0:0
[libfdk_aac @ 0x557580b48000] 2 frames left in the queue on closing

happy to take this specific issue to another Issue thread as well, just let me know :)

@almightiest
Copy link

@FurongZhang

I can also reproduce this using a sample file from here - click on " Download the UHD HDR10 - Sony Bravia OLED File " link on that page to download/save.

Then run the following 2 commands on i7-13700k or similar iGPU on ubuntu lunar with kernel 6.2 and the latest master media-driver:

  1. 1920x802 scale_vaapi - fails input/output error on 23.2.3 (master) - slow fps but no failures on 23.1.6
    ffmpeg -analyzeduration 200M -init_hw_device vaapi=va:,driver=iHD,kernel_driver=i915 -init_hw_device qsv=qs@va -filter_hw_device qs -hwaccel vaapi -hwaccel_output_format vaapi -autorotate 0 -i file:"/path/to/sample.mkv" -autoscale 0 -vf "setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,scale_vaapi=w=1920:h=802:extra_hw_frames=24,tonemap_vaapi=format=nv12:p=bt709:t=bt709:m=bt709" -f null -

  2. 1920x800 scale_vaapi - succeeds >200fps on 23.2.3 (master) and 23.1.6
    ffmpeg -analyzeduration 200M -init_hw_device vaapi=va:,driver=iHD,kernel_driver=i915 -init_hw_device qsv=qs@va -filter_hw_device qs -hwaccel vaapi -hwaccel_output_format vaapi -autorotate 0 -i file:"/path/to/sample.mkv" -autoscale 0 -vf "setparams=color_primaries=bt2020:color_trc=smpte2084:colorspace=bt2020nc,scale_vaapi=w=1920:h=800:extra_hw_frames=24,tonemap_vaapi=format=nv12:p=bt709:t=bt709:m=bt709" -f null -

@nyanmisaka
Copy link
Contributor Author

@almightiest Can you apply this patch to master branch and try again?

https://github.com/intel/media-driver/commit/d00ada51.patch

@almightiest
Copy link

almightiest commented May 25, 2023

@almightiest Can you apply this patch to master branch and try again?

https://github.com/intel/media-driver/commit/d00ada51.patch

Sorry I thought that was applied to master already. Just applied it, here's showing I have master plus this patch only:

git diff
diff --git a/media_driver/linux/gen12/ddi/media_sku_wa_g12.cpp b/media_driver/linux/gen12/ddi/media_sku_wa_g12.cpp
index 5843a77d3..1d31de532 100644
--- a/media_driver/linux/gen12/ddi/media_sku_wa_g12.cpp
+++ b/media_driver/linux/gen12/ddi/media_sku_wa_g12.cpp
@@ -520,6 +520,8 @@ static bool InitAdlsMediaSku(struct GfxDeviceInfo *devInfo,
         MEDIA_WR_SKU(skuTable, FtrGT0_5, 1);
     }

+    MEDIA_WR_SKU(skuTable, FtrHeight8AlignVE3DLUTDualPipe, 1);
+
     MEDIA_WR_SKU(skuTable, FtrAV1VLDLSTDecoding, 1);

     //Disable VP8 for ADLS

Same issue, though. Input/Output error for the 1920x802 or 1280x692 resolutions. 1920x1080, 1920x800, 1280x694 worked fine.

@Sherry-Lin
Copy link
Contributor

@FurongZhang It's a different issue on RPL-S / 13th Gen. Using tonemap_vaapi hangs and resets the i915 kernel driver.

[    2.700921] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adls_dmc_ver2_01.bin (v2.1)
[    2.704972] i915 0000:00:02.0: [drm] GuC firmware i915/tgl_guc_70.6.4.bin version 70.6.4
[    2.704974] i915 0000:00:02.0: [drm] HuC firmware i915/tgl_huc_7.9.3.bin version 7.9.3
[    2.719629] i915 0000:00:02.0: [drm] HuC authenticated
[    2.719633] i915 0000:00:02.0: [drm] GuC submission disabled
[    2.719636] i915 0000:00:02.0: [drm] GuC SLPC disabled
[    2.722123] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[    2.722747] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 1
[    2.780112] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    2.844668] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[   16.962506] snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[   16.967171] mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_component_ops [i915])
[   16.969761] mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
[65426.599038] i915 0000:00:02.0: [drm] Resetting vecs0 for preemption time out
[65426.599107] i915 0000:00:02.0: [drm] ffmpeg[1963716] context reset due to GPU hang
[65426.751730] i915 0000:00:02.0: [drm] GPU HANG: ecode 12:8:2cffbffd, in ffmpeg [1963716]
[65433.158859] i915 0000:00:02.0: [drm] Resetting vecs0 for preemption time out
[65433.158928] i915 0000:00:02.0: [drm] ffmpeg[1963716] context reset due to GPU hang
[65433.311510] i915 0000:00:02.0: [drm] GPU HANG: ecode 12:8:2cffbffd, in ffmpeg [1963716]

Edit: tonemap_vaapi won't hang the GPU driver but it will fail shortly.

frame= 0 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 0 fps=0.0 q=0.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 0 fps=0.0 q=11.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 2 fps=0.6 q=25.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 4 fps=1.0 q=25.0 size=N/A time=00:00:00.00 bitrate=N/A speed= 0x
frame= 6 fps=1.2 q=25.0 size=N/A time=00:00:00.25 bitrate=N/A speed=0.0504x
frame= 8 fps=1.3 q=25.0 size=N/A time=00:00:00.25 bitrate=N/A speed=0.0423x
frame= 10 fps=1.4 q=25.0 size=N/A time=00:00:00.25 bitrate=N/A speed=0.0365x
frame= 11 fps=1.5 q=25.0 size=N/A time=00:00:00.64 bitrate=N/A speed=0.085x
[Parsed_tonemap_vaapi_6 @ 0x557580bb0440] Failed to start picture processing: 1 (operation failed).
Error while filtering: Input/output error
Failed to inject frame into filter network: Input/output error

@nyanmisaka Considering it's different issue, could you create a new one? Thanks.

@almightiest
Copy link

@Sherry-Lin logged #1672

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
VP Video Processing
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants