Bert models / transformers package

[ghost]

Hi guys,

Hope you are doing all well !

I was wondering if DeepDetect is compatible to load models like camemBERT or flauBert with Tensorflow.

More widely, any models processed by the transformers package, for text classification/sentiment analysis tasks.

Cheers,
X

[beniz]

Hi, it's compatible with pytorch's models from or derived from Huggingface's package. It may not be fully documented yet, see #616 for inference and there are examples within the unit tests https://github.com/jolibrain/deepdetect/blob/master/tests/ut-torchapi.cc#L64

You'd need to export the models via torchscript so that that they are usable with torch C++ API, this can be achieved with this script, https://github.com/jolibrain/deepdetect/blob/master/tools/torch/trace_pytorch_transformers.py

If you decide to go this route, feel free to ask for help here.

[ghost]

There are 2 projects I wanted to load through deepdetect.

• https://github.com/TheophileBlard/french-sentiment-analysis-with-bert
• https://github.com/OthSay/bert-tweets-analysis

But I have no clue how to start

[fantes]

Hi
the general principle for current version of deepdetect is (1) that torch models have to be dumped/exported as scripted versions outside of deepdetect , and (2) then loaded as torch::jit::script inside of deepdetect.

1. First part can be done

• using
  https://github.com/jolibrain/deepdetect/blob/master/tools/torch/trace_pytorch_transformers.py . At the moment , there are only a few examples ie (vanilla) bert, roberta and gpt2. You'll have to extend this script in order to dump huggingface versions of camembert or flaubert.
• This also can be done directly from pytorch, all what is needed is [whatever] bert.pt (ie the model) + vocab.dat. see https://pytorch.org/docs/stable/jit.html and more precisely https://pytorch.org/docs/stable/jit.html#torch.jit.trace . this is what is done by previous script, it can be used as a somewhat complex example :)

2. Second part is done inside deepdetect/src/backends/torch/torchlib.cc , at first sight, you do not need to change anything here, because flaubert or camembert are basically the same as bert, at least from a structure / api point of view, so you can tell dede you are using bert, and just give the .pt and vocab.dat you want to use. up - to -date doc for doing so is at bert + gpt2 inference + training wrt torch 1.3.1 and transformers 2.2.1 #673