Attempt to get a name for a Tensor without names

[yinggeh]

Hi experts. Recently Triton team upgrades OpenVINO version from 23.3.0 to 24.4.0 for model generation. OpenVINO Inference Engine Python API is deprecated in the 2024.0 release (source). Therefore, we updated our OpenVINO model generation scripts with the new API. See commit 71ca0c5.

After the change, Triton server failed to load all new OpenVINO models with the following error (showing one as an example). Please advise.

+-------------------------+---------+-----------------------------------------------------------------------------------------------------------------+
| Model                   | Version | Status                                                                                                          |
+-------------------------+---------+-----------------------------------------------------------------------------------------------------------------+
| openvino_int8_int8_int8 | 3       | UNAVAILABLE: Internal: ModelInstanceState::Create ov::Exception: Exception from src/core/src/descriptor/tensor. |
|                         |         | cpp:84:                                                                                                         |
|                         |         | Attempt to get a name for a Tensor without names                                                                |
+-------------------------+---------+-----------------------------------------------------------------------------------------------------------------+

I1206 03:01:12.146728 1024 openvino.cc:1474] "TRITONBACKEND_ModelInitialize: openvino_float32_float32_float32 (version 3)"
W1206 03:01:12.151278 1024 openvino.cc:780] "model layout for model openvino_float32_float32_float32 does not support batching while non-zero max_batch_size is specified"
I1206 03:01:12.151301 1024 openvino.cc:846] "skipping input model configuration auto-complete for 'openvino_float32_float32_float32': input already specified"
I1206 03:01:12.151310 1024 openvino.cc:846] "skipping output model configuration auto-complete for 'openvino_float32_float32_float32': output already specified"
I1206 03:01:12.152004 1024 openvino.cc:192] "model configuration:\n{\n    \"name\": \"openvino_float32_float32_float32\",\n    \"platform\": \"\",\n    \"backend\": \"openvino\",\n    \"runtime\": \"\",\n    \"version_policy\": {\n        \"specific\": {\n            \"versions\": [\n                1,\n                3\n            ]\n        }\n    },\n    \"max_batch_size\": 8,\n    \"input\": [\n        {\n            \"name\": \"INPUT0\",\n            \"data_type\": \"TYPE_FP32\",\n            \"format\": \"FORMAT_NONE\",\n            \"dims\": [\n                16\n            ],\n            \"is_shape_tensor\": false,\n            \"allow_ragged_batch\": false,\n            \"optional\": false,\n            \"is_non_linear_format_io\": false\n        },\n        {\n            \"name\": \"INPUT1\",\n            \"data_type\": \"TYPE_FP32\",\n            \"format\": \"FORMAT_NONE\",\n            \"dims\": [\n                16\n            ],\n            \"is_shape_tensor\": false,\n            \"allow_ragged_batch\": false,\n            \"optional\": false,\n            \"is_non_linear_format_io\": false\n        }\n    ],\n    \"output\": [\n        {\n            \"name\": \"OUTPUT0\",\n            \"data_type\": \"TYPE_FP32\",\n            \"dims\": [\n                16\n            ],\n            \"label_filename\": \"output0_labels.txt\",\n            \"is_shape_tensor\": false,\n            \"is_non_linear_format_io\": false\n        },\n        {\n            \"name\": \"OUTPUT1\",\n            \"data_type\": \"TYPE_FP32\",\n            \"dims\": [\n                16\n            ],\n            \"label_filename\": \"\",\n            \"is_shape_tensor\": false,\n            \"is_non_linear_format_io\": false\n        }\n    ],\n    \"batch_input\": [],\n    \"batch_output\": [],\n    \"optimization\": {\n        \"priority\": \"PRIORITY_DEFAULT\",\n        \"input_pinned_memory\": {\n            \"enable\": true\n        },\n        \"output_pinned_memory\": {\n            \"enable\": true\n        },\n        \"gather_kernel_buffer_threshold\": 0,\n        \"eager_batching\": false\n    },\n    \"instance_group\": [\n        {\n            \"name\": \"openvino_float32_float32_float32_0\",\n            \"kind\": \"KIND_CPU\",\n            \"count\": 1,\n            \"gpus\": [],\n            \"secondary_devices\": [],\n            \"profile\": [],\n            \"passive\": false,\n            \"host_policy\": \"\"\n        }\n    ],\n    \"default_model_filename\": \"model.xml\",\n    \"cc_model_filenames\": {},\n    \"metric_tags\": {},\n    \"parameters\": {},\n    \"model_warmup\": []\n}"
I1206 03:01:12.152161 1024 openvino.cc:1519] "TRITONBACKEND_ModelInstanceInitialize: openvino_float32_float32_float32_0_0 (CPU device 0)"
E1206 03:01:12.152179 1024 openvino.cc:280] "error when reading parameters: Invalid argument - model configuration is missing the parameter CPU_EXTENSION_PATH"
E1206 03:01:12.152187 1024 openvino.cc:304] "error when reading parameters: Invalid argument - model configuration is missing the parameter SKIP_OV_DYNAMIC_BATCHSIZE"
E1206 03:01:12.152195 1024 openvino.cc:304] "error when reading parameters: Invalid argument - model configuration is missing the parameter ENABLE_BATCH_PADDING"
E1206 03:01:12.152202 1024 openvino.cc:304] "error when reading parameters: Invalid argument - model configuration is missing the parameter RESHAPE_IO_LAYERS"
W1206 03:01:12.154191 1024 openvino.cc:780] "model layout for model openvino_float32_float32_float32 does not support batching while non-zero max_batch_size is specified"
I1206 03:01:12.154212 1024 openvino.cc:846] "skipping input model configuration auto-complete for 'openvino_float32_float32_float32': input already specified"
I1206 03:01:12.154223 1024 openvino.cc:846] "skipping output model configuration auto-complete for 'openvino_float32_float32_float32': output already specified"
I1206 03:01:12.155050 1024 openvino.cc:192] "model configuration:\n{\n    \"name\": \"openvino_float32_float32_float32\",\n    \"platform\": \"\",\n    \"backend\": \"openvino\",\n    \"runtime\": \"\",\n    \"version_policy\": {\n        \"specific\": {\n            \"versions\": [\n                1,\n                3\n            ]\n        }\n    },\n    \"max_batch_size\": 8,\n    \"input\": [\n        {\n            \"name\": \"INPUT0\",\n            \"data_type\": \"TYPE_FP32\",\n            \"format\": \"FORMAT_NONE\",\n            \"dims\": [\n                16\n            ],\n            \"is_shape_tensor\": false,\n            \"allow_ragged_batch\": false,\n            \"optional\": false,\n            \"is_non_linear_format_io\": false\n        },\n        {\n            \"name\": \"INPUT1\",\n            \"data_type\": \"TYPE_FP32\",\n            \"format\": \"FORMAT_NONE\",\n            \"dims\": [\n                16\n            ],\n            \"is_shape_tensor\": false,\n            \"allow_ragged_batch\": false,\n            \"optional\": false,\n            \"is_non_linear_format_io\": false\n        }\n    ],\n    \"output\": [\n        {\n            \"name\": \"OUTPUT0\",\n            \"data_type\": \"TYPE_FP32\",\n            \"dims\": [\n                16\n            ],\n            \"label_filename\": \"output0_labels.txt\",\n            \"is_shape_tensor\": false,\n            \"is_non_linear_format_io\": false\n        },\n        {\n            \"name\": \"OUTPUT1\",\n            \"data_type\": \"TYPE_FP32\",\n            \"dims\": [\n                16\n            ],\n            \"label_filename\": \"\",\n            \"is_shape_tensor\": false,\n            \"is_non_linear_format_io\": false\n        }\n    ],\n    \"batch_input\": [],\n    \"batch_output\": [],\n    \"optimization\": {\n        \"priority\": \"PRIORITY_DEFAULT\",\n        \"input_pinned_memory\": {\n            \"enable\": true\n        },\n        \"output_pinned_memory\": {\n            \"enable\": true\n        },\n        \"gather_kernel_buffer_threshold\": 0,\n        \"eager_batching\": false\n    },\n    \"instance_group\": [\n        {\n            \"name\": \"openvino_float32_float32_float32_0\",\n            \"kind\": \"KIND_CPU\",\n            \"count\": 1,\n            \"gpus\": [],\n            \"secondary_devices\": [],\n            \"profile\": [],\n            \"passive\": false,\n            \"host_policy\": \"\"\n        }\n    ],\n    \"default_model_filename\": \"model.xml\",\n    \"cc_model_filenames\": {},\n    \"metric_tags\": {},\n    \"parameters\": {},\n    \"model_warmup\": []\n}"
I1206 03:01:12.155174 1024 openvino.cc:1519] "TRITONBACKEND_ModelInstanceInitialize: openvino_float32_float32_float32_0_0 (CPU device 0)"
E1206 03:01:12.155190 1024 openvino.cc:280] "error when reading parameters: Invalid argument - model configuration is missing the parameter CPU_EXTENSION_PATH"
E1206 03:01:12.155199 1024 openvino.cc:304] "error when reading parameters: Invalid argument - model configuration is missing the parameter SKIP_OV_DYNAMIC_BATCHSIZE"
E1206 03:01:12.155208 1024 openvino.cc:304] "error when reading parameters: Invalid argument - model configuration is missing the parameter ENABLE_BATCH_PADDING"
E1206 03:01:12.155216 1024 openvino.cc:304] "error when reading parameters: Invalid argument - model configuration is missing the parameter RESHAPE_IO_LAYERS"
I1206 03:01:12.156180 1024 openvino.cc:1553] "TRITONBACKEND_ModelInstanceFinalize: delete instance state"
E1206 03:01:12.156364 1024 backend_model.cc:692] "ERROR: Failed to create instance: ModelInstanceState::Create ov::Exception: Exception from src/core/src/descriptor/tensor.cpp:84:\nAttempt to get a name for a Tensor without names\n"
I1206 03:01:12.156377 1024 openvino.cc:1499] "TRITONBACKEND_ModelFinalize: delete model state"
E1206 03:01:12.156487 1024 model_lifecycle.cc:654] "failed to load 'openvino_float32_float32_float32' version 3: Internal: ModelInstanceState::Create ov::Exception: Exception from src/core/src/descriptor/tensor.cpp:84:\nAttempt to get a name for a Tensor without names\n"

cc @bstrzele @dtrawins @ilya-lavrenov @mc-nv

[ilya-lavrenov]

Looks like in some place you are trying to access inputs / outputs by tensor names, but they are not set in your test models.

I see here

openvino_backend/src/openvino.cc

Lines 513 to 516 in 0a3fb4c

	for (size_t i = 0; i < model_inputs.size(); i++) {
	model_inputs_names.insert(model_inputs[i].get_any_name());
	model_inputs_name_to_index[model_inputs[i].get_any_name()] = i;
	}

you are working with tensor names and get_any_name() here throws an exception.

[dtrawins]

If your model doesn't have named outputs, the following code can add the names:

import openvino as ov
from pathlib import Path

ov_model = ov.Core().read_model(Path(ir_model_dir + "/model.xml"))
for idx, out in enumerate(ov_model.outputs):  
   out.get_tensor().set_names({f"out_{idx}"})
ov.save_model(ov_model, "model_with_names.xml")  # it saves model with names

[nnshah1]

@dtrawins - the model prep scripts for the qa models is here:

triton-inference-server/server@71ca0c5

Can you recommend where the changes can be added there?
@yinggeh - are these changes we can make to the model generation?

longer term - maybe the model generation for qa models for openvino could be moved to the backend (@nvda-mesharma for viz if that makes sense / is feasible)

[yinggeh]

Hi @dtrawins. In our change 71ca0c5, we specified names to inputs and outputs. Why didn't it work as expected?

[yinggeh]

@yinggeh - are these changes we can make to the model generation?

@nnshah1 Yes. I have verified the workaround provided by @dtrawins works.

@dtrawins I noticed that another workaround is to revert the network version in model.xml. For example, changing
<net name="openvino_int8_int8_int8" version="11">
to
<net name="openvino_int8_int8_int8" version="10">
Was there any new requirement added to version 11 for tensor names?

[dtrawins]

@nnshah1 there were some changes in the model format to align the API with the models like from pytorch which allows outputs to be without a name. They can be accessed via an index. In order to use them in applications which requires a name, it has to be set explicitly. That can be done via output.get_tensor().set_names(name). I wasn't able to confirm if there an an alternative with setting it directly in openvino operations. Anyway the workaround I sent earlier should be valid. Your WA with downgrading the version indeed works but it is undocumented so I wouldn't call it recommended because for some models it might fail.
In the future we could allow in the openvino backend automatic output name assignment in case the name is missing.

[nnshah1]

@yinggeh, @dtrawins I think in this case we should add documentation that the backend currently requires models with named tensors (IIUC this is a limitation imposed by the backend and not the openvino framework).

And we should update our test model generation - to the latest. I also don't recommend downgrading the network version - we should align the model build and runtimes if we can.

@dtrawins - it would be good to get your feedback on the test model scripts and perhaps find a way to define and maintain that set in the openvino backend itself (@nvda-mesharma - does that fit our general plans of making the backends more independent from a build and test?)

[nnshah1]

Also if this is resolved for this release - let's convert this into a known issue and feature request for the backend -

[yinggeh]

@nnshah1 I am working on updating OpenVINO model generation script for this release.

[yinggeh]

I raised a PR triton-inference-server/server#7892 with output name assignment. Target for Triton release 25.01.