Super-gradients knowledge distillation

Example of training using Knowledge distillation

The student is a small model resnet18 with 11M parameters pre-trained on cifar10 and the teacher is a beit_base_patch16_224 model with 87M parameters pre-trained on ImageNet-22k.

Prerequisites

• az cli and az ml extension installed

Set up workspace config

az configure --defaults workspace=<your-workspace>
az configure --defaults group=<your-resource-group>

Create a compute cluster

Create an AzureML compute cluster with enough resource to run your training.

Create an AzureML environment

az ml environment create -f azureml/environment.yaml

Run the training

az ml job create -f azureml/job.yaml