Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding support for vulkan decode extensions #3427

Open
FelixLamontagne2 opened this issue Sep 8, 2024 · 3 comments
Open

Adding support for vulkan decode extensions #3427

FelixLamontagne2 opened this issue Sep 8, 2024 · 3 comments
Labels
Feature An improvement or feature Unresolved Waiting for a fix or implementation

Comments

@FelixLamontagne2
Copy link

FelixLamontagne2 commented Sep 8, 2024

Description

Hi,

I have a Vulkan application that currently "works", but when trying to launch it with RenderDoc I'm missing the decode extensions in the physical device. When launching without RenderDoc the extensions are there and the decoding "works". I say "works" because I'm getting a green image and would love to debug this with RenderDoc 😅

It would be amazing if RenderDoc was able to actually debug the decoding itself, but this is probably a major undertaking so perhaps as a first step just allowing these extensions so that I can inspect the resulting Image would be enough?

I'd love to provide more details, but I'm unsure about the internals of how RenderDoc works so feel free to ask me any additional questions.

Environment

  • RenderDoc version: 1.34
  • Operating System: Windows
  • Graphics API: Vulkan 1.3.290

Extensions:

  1. VK_KHR_video_decode_h264
  2. VK_KHR_video_decode_h265
  3. VK_KHR_video_decode_av1
  4. VK_KHR_video_queue
  5. VK_KHR_video_decode_queue
@FelixLamontagne2 FelixLamontagne2 changed the title [Vulkan] Adding support for vulkan decode extensions Adding support for vulkan decode extensions Sep 8, 2024
@baldurk baldurk added Feature An improvement or feature Unresolved Waiting for a fix or implementation labels Sep 9, 2024
@baldurk
Copy link
Owner

baldurk commented Sep 9, 2024

I'll leave this open for now but this may not ever happens so the issue may be closed.

The vulkan video extensions were not designed with support for capture/replay tools in mind, so I don't think this will be possible. Aside from that these extensions are a huge API surface that is for the most part unrelated and orthogonal to actual rendering - RenderDoc will never implement any meaningful debugging for these extensions as it is a graphics debugger not a video decoding debugger, so a video player program using this API would not get any use out of RenderDoc and you would not be able to use it to solve your problem.

@FelixLamontagne2
Copy link
Author

FelixLamontagne2 commented Sep 9, 2024

Thanks for the quick (and honest) response!

I think it would help here that I clarify my proposition and maybe we can make a list of v1 features that are much more digestible for RenderDoc as a product.

I definitely do not expect RenderDoc to allow debuging the actual decoding processing that's going on in the gpu. As you said, the api wasn't made with that in mind and a lot of what's happening is just the driver doing it's thing (and also the video demuxing on the cpu side which has nothing to do with Vulkan or RenderDoc).

What I do find unfortunate though is that just because I want to decode in my application then I can't inspect every other Vulkan commands and resources that RenderDoc currently has great support for.

Would it be possible as a first step to support a handful of the most important decode calls? For example, we can see in the nsight capture the vkCmdDecodeVideoKHR which allows me to inspect the parameters of the function and, most importantly, which image is being used in a decode call, but there is no "decode debuging".

Even if the decode commands weren't supported I'd still hapilly use RenderDoc to debug my own Vulkan issues with rendering and image processing, but right now I can't because the device creation fails (VK_ERROR_EXTENSION_NOT_PRESENT). Maybe RenderDoc could have some kind of extension bypass that allows it to work, but not capture the calls from certain extensions? I don't know how feasible that is though.

so a video player program using this API would not get any use out of RenderDoc and you would not be able to use it to solve your problem

Turns out that my issue was due to incorect vulkan image barriers which nsight was able to help with (although I would prefer using RenderDoc). The video was being correctly decoded, but I was using an incorect image layout which resulted in a green screen.

Below is screenshot of an nisght capture showing some of the commands going on. This doesn't show every single commands otherwise the screenshot would be too big, but as we can see it's mostly normal commands like barriers, fences and image copies.
image

@baldurk
Copy link
Owner

baldurk commented Sep 10, 2024

Yes in my previous comment I was talking about basic capture and replay with no introspection - the extensions were not designed with that in mind and I'm not sure it's practical. There is no way to 'skip' and extension or selectively implement certain parts of an extension except where allowed for by optional functionality, if it exists.

Note that you should check for extension availability via the enumeration functions before calling vkCreateDevice, the VK_ERROR_EXTENSION_NOT_PRESENT error in theory should not be encountered for a correct application and is there only as a guardrail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature An improvement or feature Unresolved Waiting for a fix or implementation
Projects
None yet
Development

No branches or pull requests

2 participants