Liga: the ML-enhanced SQL part of Rikai (The Hackathon)

[da-tubi]

Hackathon

by @da-tubi and @Renkai

Here is the result of the hackathon:

• pypi: https://pypi.org/project/liga/0.2.0.dev3/
• demo usage: https://github.com/liga-ai/liga-example
• github release: https://github.com/liga-ai/liga/releases/tag/v0.2.0.dev3
• milestone: https://github.com/liga-ai/liga/milestone/1

Liga v.s. Rikai

Liga v.s. Rikai (Linguistic)

The pronounciation of Liga in Wuu Chinese and Rikai in Japanese is almost the same.
The meaning of Liga or Rikai is understanding in English or 理解 in Chinese.

Liga v.s. Rikai (Demo Notebook)

• Rikai: https://github.com/eto-ai/rikai/blob/main/notebooks/Mojito.ipynb
• Liga: https://github.com/liga-ai/liga-example/blob/main/notebooks/RandomForestClassifier.ipynb

Liga v.s. Rikai (Software Engineering)

Rikai is too complicated to maintain and it is dedicated for computer vision. Liga is designed to modularize Rikai, we only need the ML-enhanced SQL part of Rikai:

• the ML_PREDICT magic syntax for Spark SQL (also ML_TRANSFORM , ML_FORECAST )
• MLflow integration
• ModelType design

- Rikai: ML_PREDICT/ModelType/MLflow/PyTorch/Computer Vision/Rikai format
+ Liga: ML_PREDICT/ModelType/MLflow/Sklearn

Liga v.s. Rikai (Pypi)

rikai

• rikai 0.1.15

liga

• liga 0.2.0dev3 0.2.0
  • liga-mlflow 0.2.0
  • liga-sklearn 0.2.0
  • liga-torch
  • liga-vision
    • liga-torchvision

Liga

Using the Pants build tool

Enable Python Linter and Checker

Before

# no linter and cheker

After

bin/lint
bin/check

How to test

Before

sbt publishLocal

cd python
# create a python virtualenv
pip install -e . # pip install -e .[all] to install all optional extras (see "Install from pypi")

Now

sbt assembly

bin/test

Launch the Jupyter Lab

Before

sbt publishLocal

cd python
# create a python virtualenv
pip install -e . # pip install -e .[all] to install all optional extras (see "Install from pypi")
jupyterlab

Now

sbt assembly

bin/lab sklearn

Plans

The Liga project is started as a hackathon project at Tubi. Here is the Plans:

1. Fork Rikai to Liga and remove the not-used code
2. Integrate liga-sklearn to verify the usability of Liga
3. Extensible Registeries #17