How to run ONNX model of GPT-2 ?

[JeeDevUser]

Hi all,

I am just playing with ONNX version of GPT-2

by using following Python script:

from onnxruntime_extensions import PyOrtFunction
from transformers import GPT2Tokenizer

gpt2_all = PyOrtFunction.from_model('gpt2-lm-head-bs-12.onnx')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.padding_side = "left"
tokenizer.pad_token = tokenizer.eos_token
encdict = tokenizer("hi", padding=True, return_tensors='np')

outputs = gpt2_all(encdict['input_ids'], encdict['attention_mask'].astype('float32'), 100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Wondering, Is it possible to employ this model in DJL, and how? Where to start?

[frankfliu]

@JeeDevUser
Please take a look this example: https://github.com/deepjavalibrary/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/nlp/TextGeneration.java#L164

[JeeDevUser]

@frankfliu , thanks for response!
A saw example you provided. No tried yet, but I suppose the example you gave won't work for the case I posed above?

So, here's my main problem: how to convert some Python code (for example, the one above) into an analogous Java code, which performs the same operation? If I had some rules to follow, it would be much easier. As it is, I can't figure out which part of the Java API I should use to get the same functionality.

All pretrained models are different from each other, I converted some of them to the ONNX format, for example this one , and I'm facing the same problem again - I don't know how to run them from Java code.

Can you elaborate on all this with a little more details?

[frankfliu]

@JeeDevUser

DJL is a library allows you load the model and do NDArray operations. You should be implement any model logic in Java.
To execute a Huggingface pretrained model, you need implement the same python code as transformers provided from Huggingface. Those code are task specific. In DJL we only implemented a few tasks (e.g. question_anwsering, fill_mask, text_embedding etc.). Text generation task is much more complicated, you need to understand how auto-regressive language model works. We only have a few examples which not being generalized to support other models. Yes, you need to adjust for each model based on our example, since we hard coded everything in the example.

[JeeDevUser]

you need implement the same python code as transformers provided from Huggingface. Those code are task specific. .... you need to understand how auto-regressive language model works.

@frankfliu , thanks for yuor valuable insight.
So, those who wrote that Python code had to understand how the model works, and a Java developer must understand the same thing in order to write analogous code. Now, the question is, where can one find information about how the model works in order to understand it?

Or, for example, how did you know all this information that you used to implement this?

I have a feeling that Java developers have a lot of work to do to be able to use pre-trained models, like Python developers can.
I think this is an unattainable feat for most developers...huh?

[JeeDevUser]

I think most Java developers just want to use a pre-trained model in a simple way to do some work in their applications. Just like Python developers can do it now, with a few lines of code. We're a long way from that now.

This work should probably involve the wider Java community, otherwise Java will be excluded from the future of AI development.