refactor(botonic-nlu): first approach nlu refactor

[vanbasten17]

Set as Draft PR if it's not ready to be merged.

PR best practices Reference

Description

A first approach of an improved version of current botonic-nlu engine after being migrated to Typescript.
We still need to modify botonic-cli and (probably) botonic-plugin-nlu, to have it working coordinately.

Context

The previous botonic-nlu engine was hard to maintain as it was written initially in JS. This is an improved version of that project which tries to solve several things:

• Be able to train with a custom tokenizer (it only needs to implement a tokenize method)
• Be able to train with a custom model declared explicitly by the dev
• Migration to TS. Self-descriptive code, more readable, typed. Better maintainability and flexibility
• Introduced some tests which were strongly necessary because is easy to break things in this kind of project
• Tried to use as many TFJS native methods as possible

Approach taken / Explain the design

This NLU provides a new kind of api to operate with the NLU engine. It can be used as a stand-alone node application to play with trainings and predictions. The main idea of this engine is the following:

1. We instantiate a BotonicNLU object which provides the main interface. Let's say we declare const nlu = new BotonicNLU
2. We load into memory the data structure which will contain a locale with a set of intents and each intent with a set of utterances with the method nlu.addExample(locale, intent, utterance).
3. Then we can initialize a "trainer" for a specific locale, like this: nlu.train('en').
4. Now we can tune this trainer like this:

const trainer = nlu
  .train('en')
  .withTokenizer(new natural.TreebankWordTokenizer())
  .withParams({ epochs: 30, validationSplit: 0.2 });

5. Then we only need to run it: async() => await trainer.run(). This will automatically trigger the steps of preprocessing, loading the word embeddings and train with a NN model.
6. Finally, we can save the results or try some predictions:

trainer.save(); // Saving the model
trainer.predict('How can i reach the park?');

For the development workflow, I have been using nodemon under the npm script npm run start:dev. (check index.ts).

To document / Usage example

const nlu = initFromDirectory('en');

const simpleTrainer = nlu
  .train('en')
  .withTokenizer(new natural.TreebankWordTokenizer())
  .withParams({ epochs: 30, validationSplit: 0.2 });

(async () => {
  await simpleTrainer.run();
  toPredict.forEach((input) => simpleTrainer.predict(input));
})();

Pending Steps

• Keep polishing everything: Adding more tests, improve execution flow, etc.
• Integrate all the pipeline: adapt botonic-cli to run the file from user directory, uploading generated files to Netlify, proper working with plugin-nlu and model deployed, think about retrieving the tokenizer used in botonic-nlu to be loaded in botonic-plugin-nlu.
• Add entities support, slot filling, ...

If you want to run the example:

1. yarn install
2. yarn build
3. node dev-example.js

Testing

• has unit tests
• has integration tests
• doesn't need tests because... [provide a description]

[vanbasten17]

Closing as the merged PR is this one: #984