Skip to content

Conversation

@RyanMetcalfeInt8
Copy link

@RyanMetcalfeInt8 RyanMetcalfeInt8 commented Jul 10, 2025

Description

This PR fixes an issue where zero-element tensors (typically when ort_ptr is nullptr) are not properly converted to OpenVINO tensors and set on the infer request.

Motivation and Context

For ORT GenAI use cases, it's possible that the application will pass zero-element tensors as input for inference. For example, for whisper pipeline through ORT GenAI, the first decode inference will set a 'past' KVCache tensor of shape [1, 8, 0, 64]. As this has 0 total elements, typically the ort_ptr associated with it is also nullptr.

Previously, this condition (if (cached_binding.ort_ptr != ort_ptr) {) needed to be true in order to create an ov::Tensor and set it on the infer request. But, for a zero-element tensor, both of these are NULL and so the tensor does not get set -- which causes various issues.

This PR adds an else that specifically checks for this 'zero-element' tensor case.

With this PR, I am able to run onnxruntime-genai whisper pipelines with OVEP.

Note: I didn't want to overcomplicate this PR, but I do think that this function should get revisited / refactored a bit. Comparing only the ptr is quite dangerous. For example, it's possible that the application can create a different shaped Ort tensor from the same buffer location -- so I think we should at least compared the ptr and the shape...

Copy link

@ankitm3k ankitm3k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, merging this change for now. We will revisit this change given it doesn't impact the other non genai models

@ankitm3k ankitm3k merged commit 8d36ad2 into intel:ovep-develop Jul 10, 2025
3 of 5 checks passed
ankitm3k pushed a commit that referenced this pull request Jul 17, 2025
ankitm3k pushed a commit that referenced this pull request Jul 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants