Serving HanLP NER model in a separate environment (#53)

* Serving HanLP NER model in a separate environment

* update README

* Everything is working locally

* replace

* re-enabling tests

* add tests

* style

* Deploy docker in production

* Self-review

* Self-review

* Self-review

* self-review

.github/pull_request_template.md

@@ -1,21 +1,4 @@
-Changelog
----------
+References
+----------
 
-### Added
-
-### Changed
-
-### Deprecated
-
-### Removed
-
-### Fixed
-
-### Security
-
-Checklist
----------
-
-- [ ] Test
-- [ ] Self-review
-- [ ] Documentation
+- []()

.github/workflows/ci-cd.yml

@@ -13,40 +13,38 @@ env:
   HASHICORP_PRODUCT_VERSION: "latest"
 
 jobs:
-#  test:
-#    runs-on: ubuntu-latest
-#    strategy:
-#      matrix:
-#        python-version: ["3.10"]
-#    steps:
-#      - uses: actions/checkout@v3
-#        with:
-#          fetch-depth: 0
-#      - name: Set up Python ${{ matrix.python-version }}
-#        uses: actions/setup-python@v4
-#        with:
-#          python-version: ${{ matrix.python-version }}
-#      - name: Install dependencies
-#        run: python3 -m pip install .
-#      - name: Setup tests
-#        run: |
-#          pip3 install '.[test]'
-#      - name: Run all tests
-#        run: |
-#          # "exported" env variables do NOT persist across action steps. So we co-locate them with test command
-#          export APP_CONFIG_FILE=${{ github.workspace }}/tests/settings.test.cfg
-#          pytest
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10"]
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: python3 -m pip install .
+      - name: Setup tests
+        run: pip3 install '.[test]'
+      - name: Run all tests
+        run: |
+          # "exported" env variables do NOT persist across action steps. So we co-locate them with test command
+          export APP_CONFIG_FILE=${{ github.workspace }}/tests/settings.test.cfg
+          pytest
 
   release:
     name: Release IAM image to AWS and Deploys its instance to EC2
-    # needs: test
+    needs: test
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
         uses: actions/checkout@v3
       - name: Load Flask App Config
-        run: |
-          echo '${{ secrets.FLASK_SETTINGS_FILE }}' > ${{ github.workspace }}/../settings.cfg
+        run: echo '${{ secrets.FLASK_SETTINGS_FILE }}' > ${{ github.workspace }}/../settings.cfg
       - name: Load SSL Certificates
         run: |
           echo '${{ secrets.NEXUSGRAPH_THERESA_SSL_CERT }}' > ${{ github.workspace }}/../nexusgraph_theresa_ssl.crt
@@ -56,15 +54,13 @@ jobs:
           echo '${{ secrets.NEXUSGRAPH_THERESA_NGINX_CONFIG_FILE }}' > ${{ github.workspace }}/../nexusgraph_theresa_nginx.conf
           echo '${{ secrets.PUBLIC_THERESA_NGINX_CONFIG_FILE }}' > ${{ github.workspace }}/../public_theresa_nginx.conf
       - name: Load Packer variable file
-        run: |
-          echo '${{ secrets.HASHICORP_PACKER_AUTO_VAR_FILE }}' > hashicorp/images/aws.auto.pkrvars.hcl
+        run: echo '${{ secrets.HASHICORP_PACKER_AUTO_VAR_FILE }}' > hashicorp/images/aws.auto.pkrvars.hcl
       - name: Load Terraform variable file
-        run: |
-          echo '${{ secrets.HASHICORP_TERRAFORM_AUTO_VAR_FILE }}' > hashicorp/instances/aws.auto.pkrvars.hcl
+        run: echo '${{ secrets.HASHICORP_TERRAFORM_AUTO_VAR_FILE }}' > hashicorp/instances/aws.auto.tfvars
       - name: Load Theresa executable
         run: |
-          tar -czf theresa.tar.gz *
-          mv theresa.tar.gz hashicorp/images
+          cd ../
+          tar --exclude-vcs -czvf theresa.tar.gz theresa
       - name: Publish Theresa AMI image and deploy it to EC2 through HashiCorp
         uses: QubitPi/aergia@master
         with:

.gitignore

@@ -14,3 +14,4 @@ build/
 .idea/
 *.swp
 *~
+mlflow_models/models/

README.md

@@ -1,7 +1,7 @@
-Theresa <sup>![Python Version Badge][Python Version Badge]</sup>
-================================================================
+Theresa <sup>![Python Version Badge](https://img.shields.io/badge/Python-3.10-brightgreen?style=flat-square&logo=python&logoColor=white)</sup>
+=======
 
-- Since this is a private-repo for a single person, documentations are all on this README file
+- Since this is a private-repo by a single person, documentations are all in this page
 - **The principle of Theresa is **SIMPLE, SIMPLE, and SIMPLE**. Theresa is a machine learning service deployed as a
   [separation-of-concern](https://stackoverflow.com/a/59492509) microservice. It does NOT handle caching, auth, or
   request pre-processing or response post-processing. **It simply loads some ML model, performs inference, and returns
@@ -21,7 +21,6 @@ python3 -m venv .venv
 
 ```bash
 python3 -m pip install .
-# pip3 install hanlp[amr,fasttext,full,tf]
 ```
 
 ### 3. Run Webservice Locally in Dev Mode
@@ -32,8 +31,8 @@ flask --app theresa run --debug
 ```
 
 - Note that `APP_CONFIG_FILE` has to be an _absolute_ path.
-- Running locally has [Flask debug mode] turned on
-- Swagger API (using [Flasgger]) is available at http://localhost:5000/apidocs/
+- Running locally has [Flask debug mode](https://flask.palletsprojects.com/en/latest/quickstart/#debug-mode) turned on
+- Swagger API (using [Flasgger](https://github.com/flasgger/flasgger)) is available at http://localhost:5000/apidocs/
 - The endpoints are available at http://127.0.0.1:5000. Example browser query:
 
   ```bash
@@ -73,8 +72,9 @@ CI/CD
   usage can be viewed at https://github.com/settings/billing
 - To protect API from unauthorized use:
 
-  - Rapid API instance uses Nginx to check [Rapid API gateway header][Rapid API firewall settings] so that only Rapid
-    API requests can hit public instance
+  - Rapid API instance uses Nginx to check
+    [Rapid API gateway header](https://docs.rapidapi.com/docs/security-threat-protection#firewall-settings) so that only
+    Rapid API requests can hit public instance
   - Although there is no check between EC2 and its load balancer, the load balancer binds to a Security Group called
     "Paion Data nexusgraph Theresa Load Balancer" whose inbound rules only allows requests from app.nexusgraph.com EC2
     instances
@@ -90,18 +90,6 @@ After developing Theresa, we'll want to make it available publicly to other user
 debugger, and reloader we use for local development should not be used in production. Instead, we use a dedicated
 [WSGI server](#wsgi-servers) listening at port **8000**
 
-> ⚠️⚠️⚠️
-> 
-> The number of worker process MUST be **1** to prevent multiple workers from downloading a HanLP pre-trained model to
-> the same location, which results in error of
-> 
-> ```bash
-> OSError: [Errno 39] Directory not empty: '/root/.hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210304_135840'
-> -> '/root/.hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210111_124159'
-> ```
-> 
-> ⚠️⚠️⚠️
-
 Entity Extraction
 -----------------
 
@@ -160,11 +148,59 @@ graph generator from text**, which leads me to the next approach
 
 ## Multi/Multiplex Parser
 
+MLflow
+------
+
+### Entity Extraction
+
+Create virtual environment and install dependencies:
+
+```bash
+cd mlflow_models/HanLPner
+python3 -m venv .venv
+. .venv/bin/activate
+pip3 install -r requirements.txt
+```
+
+Generate Model with
 
+```bash
+python3 HanLPner.py
+```
+
+A model directory called "HanLPner" appears under `mlflow_models/models`. Then build Docker image and run container with
+
+```bash
+cd mlflow_models/models/HanLPner
+mlflow models build-docker --name "entity-extraction"
+
+docker run --rm \
+  --memory=4000m \
+  -p 5001:8080 \
+  -v $NER_MODEL_PATH:/opt/ml/model \
+  -e GUNICORN_CMD_ARGS="--timeout 60 -k gevent --workers=1" \
+  "entity-extraction"
+```
+
+> ⚠️⚠️⚠️
+> 
+> The number of gunicorn worker process MUST be **1** (`--workers=1`) to prevent multiple workers from downloading a
+> HanLP pre-trained model to the same location, which results in runtime error in Docker container. In **native**
+> environment, this error can be
+> 
+> ```bash
+> OSError: [Errno 39] Directory not empty: '/root/.hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210304_135840'
+> -> '/root/.hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210111_124159'
+> ```
+> 
+> ⚠️⚠️⚠️
 
-[Flask debug mode]: https://flask.palletsprojects.com/en/latest/quickstart/#debug-mode
-[Flasgger]: https://github.com/flasgger/flasgger
+Example query:
 
-[Python Version Badge]: https://img.shields.io/badge/Python-3.10-brightgreen?style=flat-square&logo=python&logoColor=white
+```bash
+curl -X POST -H "Content-Type:application/json" \
+  --data '{"dataframe_split": {"columns":["text"], "data":[["我爱中国"], ["米哈游成立于2011年,致力于为用户提供美好的、超出预期的产品与内容。米哈游多年来秉持技术自主创新,坚持走原创精品之路,围绕原创IP打造了涵盖漫画、动画、游戏、音乐、小说及动漫周边的全产业链。"]]}}' \
+  http://127.0.0.1:5001/invocations
+```
 
-[Rapid API firewall settings]: https://docs.rapidapi.com/docs/security-threat-protection#firewall-settings
+[Note the JSON schema of the `--data` value](https://stackoverflow.com/a/75104855)

hashicorp/images/aws-base.pkr.hcl

@@ -8,6 +8,11 @@ variable "theresa_settings_config_path" {
   sensitive = true
 }
 
+variable "theresa_tar_gz_path" {
+  type =  string
+  sensitive = true
+}
+
 variable "skip_create_ami" {
   type =  bool
   sensitive = true

hashicorp/images/aws-nexusgraph.pkr.hcl

@@ -70,14 +70,14 @@ build {
 
   # Load Theresa executable
   provisioner "file" {
-    source = "./theresa.tar.gz"
+    source = "${var.theresa_tar_gz_path}"
     destination = "/home/ubuntu/theresa.tar.gz"
   }
 
   provisioner "shell" {
-    script = "../scripts/aws-base-pkr-setup.sh"
-  }
-  provisioner "shell" {
-    script = "../scripts/aws-nexusgraph-pkr-setup.sh"
+    scripts = [
+      "../scripts/aws-base-pkr-setup.sh",
+      "../scripts/aws-nexusgraph-pkr-setup.sh"
+    ]
   }
 }

hashicorp/images/aws-public.pkr.hcl

@@ -50,7 +50,7 @@ build {
 
   # Load Theresa executable
   provisioner "file" {
-    source = "./theresa.tar.gz"
+    source = "${var.theresa_tar_gz_path}"
     destination = "/home/ubuntu/theresa.tar.gz"
   }
 

hashicorp/scripts/aws-base-pkr-setup.sh

@@ -1,5 +1,6 @@
 #!/bin/bash
 set -x
+set -e
 
 sudo apt update
 sudo apt upgrade -y
@@ -8,17 +9,42 @@ sudo apt upgrade -y
 sudo apt-get -y install python3-pip
 pip3 install --upgrade pip
 sudo apt -y install python3.10-venv
+PATH=$PATH:/home/ubuntu/.local/bin
 
 python3 --version
 
 # Install Theresa
-mkdir -p /home/ubuntu/theresa
-cd /home/ubuntu/theresa
-mv ../theresa.tar.gz .
-tar -xf theresa.tar.gz
+cd /home/ubuntu
+tar -xvf theresa.tar.gz
 rm theresa.tar.gz
-cd /home/ubuntu/
 
 # Install Nginx and load SSL config
 sudo apt install -y nginx
 sudo mv /home/ubuntu/nginx.conf /etc/nginx/sites-enabled/default
+
+# Add Docker's official GPG key:
+sudo apt-get update
+sudo apt-get install -y ca-certificates curl gnupg
+sudo install -m 0755 -d /etc/apt/keyrings
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+sudo chmod a+r /etc/apt/keyrings/docker.gpg
+
+# Add the repository to Apt sources:
+echo \
+  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
+  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
+  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+sudo apt-get update
+
+# Install Docker
+sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+
+# Install HanLP MLflow model
+cd /home/ubuntu/theresa/mlflow_models/HanLPner
+pip3 install -r requirements.txt
+python3 HanLPner.py
+cd /home/ubuntu/theresa/mlflow_models/models/HanLPner
+sudo usermod -aG docker $USER
+sudo chmod o+rw /var/run/docker.sock # https://stackoverflow.com/a/76329637
+mlflow models build-docker --name "entity-extraction"
+cd /home/ubuntu/

hashicorp/scripts/aws-base-tf-init.sh

@@ -1,10 +1,19 @@
 #!/bin/bash
 set -x
+set -e
 
 touch /home/ubuntu/aws-base-tf-init.log
 export AWS_BASE_TF_INIT_LOG=/home/ubuntu/aws-base-tf-init.log
 echo "aws-base-tf-init started executing..."       >>$AWS_BASE_TF_INIT_LOG 2>&1
 
+sudo docker run --detach --rm \
+  --memory=4000m \
+  -p 5001:8080 \
+  -v /home/ubuntu/theresa/mlflow_models/models/HanLPner:/opt/ml/model \
+  -e GUNICORN_CMD_ARGS="--timeout 60 -k gevent --workers=1" \
+  "entity-extraction"                              >>$AWS_BASE_TF_INIT_LOG 2>&1
+echo "HanLP container started"                     >>$AWS_BASE_TF_INIT_LOG 2>&1
+
 sudo nginx -t                                      >>$AWS_BASE_TF_INIT_LOG 2>&1
 sudo nginx -s reload                               >>$AWS_BASE_TF_INIT_LOG 2>&1
 echo "nginx init done..."                          >>$AWS_BASE_TF_INIT_LOG 2>&1

hashicorp/scripts/aws-nexusgraph-pkr-setup.sh

@@ -1,5 +1,6 @@
 #!/bin/bash
 set -x
+set -e
 
 sudo mv /home/ubuntu/server.crt /etc/ssl/certs/server.crt
 sudo mv /home/ubuntu/server.key /etc/ssl/private/server.key