Quick-start example for preprocessing, training and deploying ranking models

[gabrielspmoreira]

Fixes NVIDIA-Merlin/Merlin#916 , fixes NVIDIA-Merlin/Merlin#917 , fixes NVIDIA-Merlin/Merlin#918, fixes #680, fixes #681, fixes #666

Goals ⚽

This PR introduces a quick-start example for preprocessing, training, evaluating and deploying ranking models.
It is composed by a set of scripts and markdown documents. We use in the example the TenRec dataset, but the scripts are generic and can be used with customer own data, provided that they have the right shape: positive and potentially negative user-item events with tabular features.

Implementation Details 🚧

• preprocessing.py - Generic script for preprocessing with CLI arguments for preprocessing a raw dataset (CSV or parquet) with NVTabular. It contains arguments to configure input path and format, categorical and continuous features, configuring the features tagging (user_id, item_id, ...), to filter interactions by using min/max frequency for users or items and dataset split.
  Example command line for TenRec dataset:

python preprocessing.py --input_data_format=csv --csv_na_values=\\N --input_data_path /data/QK-video.csv --output_path=$OUT_DATASET_PATH --categorical_features=user_id,item_id,video_category,gender,age --binary_classif_targets=click,follow,like,share --regression_targets=watching_times --to_int32=user_id,item_id --to_int16=watching_times --to_int8=gender,age,video_category,click,follow,like,share --user_id_feature=user_id --item_id_feature=item_id --min_user_freq 5 --persist_intermediate_files --dataset_split_strategy=random --random_split_eval_perc=0.2 	

• ranking_train_eval.py - Generic script for training and evaluation of ranking models. It takes the preprocessed dataset from preprocessing.py and schema as input. You can set many different training and model hparams for train both single-task learning (MLP, DCN, DLRM, Wide&Deep, DeepFM) and multi-task learning specific models (e.g. MMOE, CGC, PLE).

python  ranking_train_eval.py --train_path $OUT_DATASET_PATH/final_dataset/train --eval_path $OUT_DATASET_PATH/final_dataset/eval --output_path ./outputs/ --tasks=click --stl_positive_class_weight 4 --model dlrm --embeddings_dim 64 --l2_reg 1e-5 --embeddings_l2_reg 1e-6 --dropout 0.05 --mlp_layers 64,32  --lr 1e-4 --lr_decay_rate 0.99 --lr_decay_steps 100 --train_batch_size 4096 --eval_batch_size 4096 --epochs 1 --train_steps_per_epoch 10 

Testing Details 🔍

• The preprocessing and training ranking scripts are going to be added as integration tests.

Tasks

Implementation

• [Task] Create a generic dataset preprocessing template for ranking models Merlin#916
• Update the research STL/MTL ranking training script to use the latest API (PredictionBlock instead of PredictionTask) Merlin#917
• Create a generic STL/MTL ranking training/eval script based on the research scripts Merlin#918
• Refine preprocessing.py to provide additional dataset split strategies (e.g. random_by_user, temporal).
• Adapt preprocessing.py to use Dask Distributed client for preprocessing larger/full dataset (single or multiple GPU)
• Refine scripts to accept both CLI args or YAML args

Experimentation

• Run hypertuning experiments for the STL ranking models: MLP, Wide&Deep, DCN-v2, DeepFM, DLRM for Tenrec dataset Merlin#680
• Run hypertuning experiments for the MTL ranking models: MMoE, PLE for Tenrec dataset Merlin#681

Documentation

• Create documentation for Quick-Start for Ranking (CLI args, best practices and tutorials) Merlin#666 - Create a markdown document providing best practices on setting hyperparameters for ranking models based on the empirical results from our research experimentation (e.g. hparam optimization search space, best hparams found, comparison of the accuracy of STL and MTL models for each task)

Deployment and inference with Triton

• Adjust synthetic dataset generator to use Merlin dtypes (in schemas etc) #912
• [BUG] Error when loading the session-based model, layer saved without config #913
• [Task] Update multi-task learning models and tests to use OutputBlock instead of PredictionsTasks #914

Testing

• Create integration tests for ranking models on CI to track accuracy or performance regressions over time Merlin#667

[github-actions]

Documentation preview

https://nvidia-merlin.github.io/models/review/pr-988

[gabrielspmoreira]

Closing this PR, as it was moved to Merlin Repo: NVIDIA-Merlin/Merlin#915