Encoding/decoding NLP model in tensorflow lite (fine-tuned GPT2)

[Guillaume-slize]

We are in the process of building a small virtual assistant and would like it to be able to run a fine-tuned version of GPT-2 on a raspberry-pi with a coral accelerator.

So far, we managed to convert our model to a tflite and to get first results. We know how to convert from words to indices with the previous tokenizer but then we need a bigger tensor as input to the interpreter. We miss the conversion from indices to tensors. Is there a way to do this simply?

You can find our pseudo-code here, we are stuck at step 2 and 6 :

import tensorflow as tf
 
#Prelude
TF_MODEL_PATH_LITE = "/path/model.tflite"
 
interpreter = tf.lite.Interpreter(model_path=TF_MODEL_PATH_LITE)
interpreter.allocate_tensors()
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
input_shape = input_details[0]['shape']
 
#1-Encode input, giving you indices
context_idx = tokenizer.encode("Hello world.", return_tensors = "tf")
 
#2-How to convert the context_idx to appropriate np.array ?
input_data = np.array(np.random.random_sample(input_shape), dtype=np.int32) #dummy input for now
 
#3- feed input
interpreter.set_tensor(input_details[0]['index'], input_data)
 
#4- Run model
interpreter.invoke()
 
#5- Get output as tensor
output_data = interpreter.get_tensor(output_details[0]['index'])
 
#6- How decode this np array to idx
output_idx=np.random.randint(100) #dummy for now ...
 
#7- Decode Output from idx to word
string_tf = tokenizer.decode(output_idx, skip_special_tokens=True)

[LysandreJik]

Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the forum instead?

Thanks!

[Guillaume-slize]

Sure, Thanks for your answer :)

…

On Mon, May 31, 2021 at 9:51 AM Lysandre Debut ***@***.***> wrote: Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests. Could you ask your question on the forum <https://discuss.huggingface.co> instead? Thanks! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#11947 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRJZ5KO6QL4EYWKMDXCXADTQM5WVANCNFSM45ZFOJYQ> .

-- Guillaume Slizewicz www.guillaumeslizewicz.com +32 496 53 6666

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.