"Failed to convert to numpy array for model '_cortex_default'" when making a prediction on an ONNX model

[RobertLucian]

Version

Version 0.18.0

Description

When the input shape of an ONNX model has been set to a string (thus indicating that the axes are dynamic), then making a prediction will give an error of this kind:

cortex.lib.exceptions.UserException: error: key 'input_ids' for model '_cortex_default': failed to convert to NumPy array for model '_cortex_default': cannot reshape array of size 6 into shape (1,1)

Here's an example of a model's input shapes:

model input      type    shape
attention_mask   int64   (batch, sequence)
input_ids        int64   (batch, sequence)

Steps to reproduce

Within a given directory, run all the following steps.

Creating environment/model

Create a virtual environment for Python 3.6.9 and install the following pip dependencies:

onnxruntime==1.3.0
torch==1.5.0
transformers==3.0.0
scipy==1.4.1

Within that environment, run the following instructions to export the XLM-Roberta model in ONNX format:

from transformers.convert_graph_to_onnx import convert
convert(framework="pt", model="xlm-roberta-base", output="./output/xlm-roberta-base.onnx", opset=11)

Now, let's run the following:

python -m onnxruntime_tools.optimizer_cli --input ./output/xlm-roberta-base.onnx --output ./output/xlm-roberta-base.onnx --model_type bert --float16

Creating the Cortex deployment

Create a cortex.yaml config file with the following content:

# cortex.yaml

- name: api
  predictor:
    type: onnx
    model_path: ./output/xlm-roberta-base.onnx
    path: predictor.py
    image: cortexlabs/onnx-predictor-cpu:0.18.0

Create a predictor.py script with the following content:

# predictor.py

from transformers import XLMRobertaTokenizer
from scipy.special import softmax
import time


class ONNXPredictor:
    def __init__(self, onnx_client, config):
        self.client = onnx_client
        self.tokenizer = XLMRobertaTokenizer.from_pretrained("xlm-roberta-base")

    def predict(self, payload):
        start = time.time()
        model_inputs = self.tokenizer.encode_plus(payload["text"], max_length=512, return_tensors="pt", truncation=True)
        inputs_onnx = {k: v.cpu().detach().numpy() for k, v in model_inputs.items()}
        print(self.client._signatures)

        output = self.client.predict(inputs_onnx)
        output = softmax(output[0], axis=1)[0].tolist()

        end = time.time()
        return {"output": output, "time": end - start}

Copy-paste the pip dependencies as mentioned above into a requirements.txt file and within the same directory as that of the cortex.yaml config file, run cortex deploy -e local. Wait for the API to be live and then run:

curl http://localhost:8888 -X POST -H "Content-Type: application/json" -d '{"text": "That is a nice"}'

Error

The above command will return a non-200 response code. Inspect the logs with cortex get api. The expected error is:

cortex.lib.exceptions.UserException: error: key 'input_ids' for model '_cortex_default': failed to convert to numpy array for model '_cortex_default': cannot reshape array of size 6 into shape (1,1)