Loading bert preprocess model in DLJ

[Tacacs-1101]

I want to load the BERT preprocess model from tensorflow hub in DLJ. I am not able to figure out what to use in
.setTypes() as Input and Output since output from model will be a dictionary . Below is the code i am trying to use in my application. modelUrl is pointing to location on tensorflow hub. The input to this model should be a list of strings and output is a map containing list of tokenized id, token types , attention mask.

Criteria<String[], Map<String, float[][]>> criteria = Criteria.builder()
.setTypes(String[].class, Map<String, float[][]>.class)
.optModelUrls(modelUrl)
.optProgress(new ProgressBar())
.build();

[zachgk]

The setTypes is based on your choice of Translator. Deep learning always requires that the inputs and outputs of a model be NDArrays. The translator is used to wrap the NDArray logic within more natural Java data types.

So, what Java data types make most sense in your application? Here, you chose String[] and float[][]. That would work, but you could also consider building your own classes to make it simpler. Then, in your translator you would convert between the natural Java data types and the NDArrays/NDLists expected by the model.

[Tacacs-1101]

Thanks for your response. My corresponding java data type would be Map<String, float[]> but what data type will i receive after model prediction that I am not able to figure out.

[zachgk]

The actual types used by deep learning models are always NDList. It's input is some kind of NDList and the output is some kind of NDList. So, in your post-processing code (defined in your Translator), you will need to write the conversion between however the model formats the data in the output NDList to your chosen Java data type. Or, you could use a NoopTranslator and have your model go directly from NDList to NDList without post-processing.

[reuschling]

I managed it to load the preprocess model from TF Hub with types of (String[].class, NDList.class). For this, I modified the MyTranslator from the UniversalSentenceEncoder example, getting its processInput method, implementing processOutput just by detaching and returning the second parameter. Predicting with this model works, I get the filled NDList Object.
But I am not able to feed this result into the Bert model in the second step. Loading a Bert model, e.g. this one with types (NDList.class, float[][].class) works. As Translator, I simply get the processOutput from MyTranslator, returning the input NDList parameter inside processInput.

But if I then invoke the predict method with the result NDList from the preprocessor, I get a core dump by crashing libtensorflow_cc.so.2+0xde703dd] tensorflow::TF_TensorToTensor(TF_Tensor const*, tensorflow::Tensor*)+0xd

...maybe there is a further conversion step necessary. Does anybody have loaded a Bert Text_Embeddings model sucessfully yet? In the examples I find only the Classification example, also in the Zoo models there is no Bert for text embeddings

[zachgk]

Can you try saving the output of the UniversalSentenceEncoder model to a file with NDList encode? Then, decode it and run it through the Bert model. Just to make sure no memory weirdness is happening. This should determine if the input is crashing the Bert model or if it is something in the transition from one model to another.

Also, we have an example of BertQA at http://docs.djl.ai/master/docs/demos/jupyter/BERTQA.html that you can look at. The translator is at https://github.com/deepjavalibrary/djl/blob/master/engines/pytorch/pytorch-model-zoo/src/main/java/ai/djl/pytorch/zoo/nlp/qa/PtBertQATranslator.java