The deployment
directory is dedicated to facilitating the one-click deployment of our human evaluation tool using Docker-compose. This directory includes all the necessary scripts and configuration files to ensure an automated deployment process.
- YAML Files for Docker-compose: These files are used to configure and orchestrate the deployment of the tool's various components through Docker-compose.
.env
Environment File: Contains essential environment variables and configurations for deployment. This file should be modified to ensure that all components are correctly set up.nginx
Directory: Houses the configuration files for our web server. Files in this directory should be modified for your own deployment.
The server architecture is segmented into five distinct parts, as illustrated in the figure below. This setup includes:
- Front-end Web Interface: Accessible here.
- Web Server: Accessible here.
- Database: Utilises MongoDB.
- Model Connector: Utilises Redis middleware.
- ToD System Model Servers: Described here, marked as dashed yellow rounded rectangles in the figure.
To successfully set up the evaluation tool, deployment of three types of servers is necessary:
- Model Connector
- ToD System Model Servers
- Web Servers
For optimal reliability, deploying the Model Connector and Web Servers using Docker containers is recommended. Command-line methods are suggested for setting up the ToD System Model Servers.
The Celery Queue functions as the model connector, handling the messaging process between the web interface and specific models. Celery requires Redis as a middleware for task distribution and result reception. To build the infrastructure, you need firstly deploy redis on one of your machines.
-
On your chosen machine for deploying Redis, clone the project and navigate to the
deployment
folder. Edit the.env
file in this folder, setting theREDIS_PASSWORD
andREDIS_ADDRESS
. TheREDIS_ADDRESS
should be the IP address where the Redis service will be deployed. -
Deploy the Redis service with the following command in the deployment folder:
bash deploy.sh --component=redis
Note: We assume Redis will run on the default port (6379). If you have an existing Redis server on this port, it can be used directly. Alternatively, follow the provided instructions to deploy a new Redis instance.
The ToD System Model Servers are the pivotal components of our tool, responsible for processing task-specific requests within a dialogue system.
-
Model Preparation:
- Before deploying these servers, it's essential to prepare your models for the ToD system. For demonstration, We offer two pretrained mt5 models for deployment testing purposes:
songbo/multi3woz-en-dst-mt5-small
andsongbo/multi3woz-en-rg-mt5-small
. These models are provided to verify the deployment process. - If you're interested in training custom models for your ToD system, comprehensive guidance for training and preparing your models is available at tod_system/README.md.
- Before deploying these servers, it's essential to prepare your models for the ToD system. For demonstration, We offer two pretrained mt5 models for deployment testing purposes:
-
Setup Variables and Configurations for Microservice Deployment:
- Environment Setup: Configure the
.env
file in the deployment folder as per the provided instructions. - Model Configuration: Create and modify the config file in tod_system/human_eval_service/config with your model name and other paths. Example configurations like mt5.cfg are provided.
- Environment Setup: Configure the
-
Deployment Command:
-
Deployment Options By default, this human evaluation tool follow the microservice architecture. It means that a system is are composed with two independent servers: dst and rg. You have to deploy them independently. To make our tool capable with other systems developed outside our toolkit, we provide another deployment option e2e. To connect the tool with your own system, your system have to implement this specified interface tod_system/human_eval_service/your_own_cool_e2e_system.py.
-
Example Command:
bash deploy.sh --component tod --env prod --name mt5 --type dst
- Command Components:
-c/--component
: Specifies the deployment component, e.g.,tod
for ToD system microservice.-e/--env
: Sets the environment, either 'dev' (which will be introduced later on) for development or 'prod' for production. Useprod
here.-n/--name
: Indicates the model name. It will be used as MODEL_NAME variable in the.env
file. This name should match your config file in tod_system/human_eval_service/config. mt5 is the model name in this example.-t/--type
: Denotes the model type (e.g.,dst
,rg
, ore2e
). This value will set up the MODEL_TYPE variable
- Command Components:
-
Example Setup: For our testing setup, it is necessary to deploy both a DST model and an RG model. This can be accomplished by running the following commands:
bash deploy.sh --component tod --env prod --name mt5 --type dst bash deploy.sh --component tod --env prod --name mt5 --type rg
-
GPU Support: If deploying on an NVIDIA GPU server, modify deployment/docker-compose_tod_prod.yml as per the Docker documentation on GPU support. If you are using a server with NVIDIA GPUs, you may add the following lines to the deployment/docker-compose_tod_prod.yml below the volume section (ensure to keep the indent and start at the same indentation level as the
volume
):
deploy:
resources:
reservations:
devices:
- driver: nvidia
#count: the number of gpu you want to use
capabilities: [gpu]
-
Initial Setup:
- Clone this repository on the machine intended for deployment.
- Modify the
.env
file in this directory according to the instructions within. - For MongoDB setup, either:
- Configure your own MongoDB instance on a cloud service and enter the MongoDB URL into the
.env
file. - To deploy MongoDB on your server, execute:
bash deploy.sh --component mongodb
- Configure your own MongoDB instance on a cloud service and enter the MongoDB URL into the
-
Configure the Web Interface:
- In the
.env
file, setREACT_APP_SERVER_URL
to your domain name or the IP address of the deployment machine. - If
USE_SSL
is set to true, and you have an SSL certificate, you can access your page via HTTPS.
- In the
-
Modify Environment Variables:
SECRET_KEY
: Used for JWT token encryption.MONGO_URI
: The URI of your MongoDB database.TASK_PATH
: Represents the task for testers, defined in human_eval_tool/server/config. Follow the format in human_eval_tool/server/config/test_goals.json.- For SSL: Place your certificate (
ssl.crt
andssl.key
) in deployment/nginx/ssl. SetUSE_SSL
to true and rename example.ssl.conf to ssl.conf in this folder. REACT_APP_SERVER_URL
: Should be your server IP or domain address if using an SSL certificate.
-
Deploying the Human Evaluation Tool:
- Deploy with the command:
bash deploy.sh --env=prod --component=tools --name {Your model name here}
- In the above example,
Your model name here
ismt5
.
- Deploy with the command:
-
Container Verification:
- Use
docker ps
to check the containers. Ensure that server, nginx, and client containers are running. The client container will disappear once compiling is finished.
- Use
Once deployed, access the tool via the specified IP or domain name and enjoy its functionality. 🚀
Quick Test Example:
For a quick test with our provided mt5 model, ensure your machine meets the minimum requirements (8GB RAM, 4-core x86 server with Docker and Docker Compose). Set
MONGO_URI
(if you deploy mongodb with this tutorial, you don't need to change this),ADMIN_EMAIL
, andREACT_APP_SERVER_URL
(your machine IP address or domain name). Then execute the following deployment commands sequentially:bash deploy.sh --component=redis
If you want to deploy the mongodb by yourself, then
bash deploy.sh --component=mongodb
Otherwise, you could use cloud mongo service with given
MONGO_URI
. Then,bash deploy.sh --env=prod --component=tools --name mt5 bash deploy.sh --component tod --env prod --name mt5 --type rg bash deploy.sh --component tod --env prod --name mt5 --type dst
If you prefer to deploy this tool without Docker, or if you plan to modify the tool further, you will need to set up the development environment as follows:
-
Redis Service Setup:
- Clone the project on the deployment machine and navigate to the project folder.
- Modify the
.env
file in the docker folder as per the instructions provided. As a first step, you only need to set theREDIS_PASSWORD
.
-
Deploying Redis:
- Use the following command in the project repository to deploy the Redis service:
bash deploy.sh --component=redis
- Use the following command in the project repository to deploy the Redis service:
For production environments, we highly recommend deploying this component using a Docker container. We have not provided a specific command-line method for production deployment. For detailed insights into this complicated process, please refer to our Docker file. For development purposes, you can follow these steps:
-
Web Interface Setup:
- Clone this repository on the machine where you plan to deploy the human evaluation tools.
- Modify the
.env
file in this directory according to the provided instructions.
-
Deploying the Human Evaluation Tool:
- Deploy the tool with the following command:
bash deploy.sh --env=dev --component=tools --name {Your model name}
- Once completed, visit the human_eval_tool folder and follow the development instructions there.
- Deploy the tool with the following command:
This setup allows you to test the web interface without encountering CORS issues. 😊
For developing and deploying your own ToD system models, we recommend using the command-line method, especially for production environments. Due to potential issues with CUDA in Docker, a command-line setup might be more reliable.
- Development and Evaluation:
- Configure the
.env
file in this folder and follow the instructions provided in the Multilingual E2E ToD System Development and Evaluation guide.
- Configure the