3. Tutorials

0. Framework Design

1. How to build a pipeline

init:

• self.config: get the overall config from called function

run:

• logger: no need for changes, if you want to add some new functions to the logger please raise an issue

  from openood.utils import setup_logger
  
  # generate output directory and save the full config file
  setup_logger(self.config)

• preprocessor: config file in configs/preprocessors, add new preprocessor in openood/preprocesors, if you don’t need a preprocessor then assigend with a base preprocessor in config file

  from openood.preprocessors import get_preprocessor
  
  # get preprocessor
  preprocessor = get_preprocessor(self.config)

• dataloader

  from openood.datasets import get_dataloader
  
  # get dataloader
  loader_dict = get_dataloader(self.config.dataset, preprocessor)

  config file in configs/datasets/

  use ImageList data class by default

  add new get dataloader method in openood/datasets/utils.py if you cannot fit into existing methods

  add new data class if needed, though idealy all dataset should use ImageList data class

• network

  from openood.networks import get_network
  
  # init network
  net = get_network(self.config.network)

  config file in configs/networks

  add new network in openood/networks

  make proper tweaks in openood/networks/utils.py to intialize the network(where you set up hyper parameters for the network)

• trainer

  from openood.trainers import get_trainer
  
  # init trainer
  trainer = get_trainer(net, train_loader, self.config)

  trainer defines the optimizer, scheduler and loss for the training

  train_epoch function in trainer runs during the training process which will be called later in the pipeline

  config file in config/pipelines/train

  add a new trainer in openood/trainers

• evaluator

  from openood.evaluators import get_evaluator
  
  # init evaluator
  evaluator = get_evaluator(self.config)

  evaluator defines ,well, how you evaluate your model eval() function is called when perform evaluation on test/eval dataset

  config file in

  configs/pipelines/test and configs/pipelines/train depends on which pipeline this is

  add new evaluator in openood/evaluators

• recorder

  from openood.recorders import get_recorder
  
  # init recorder
  recorder = get_recorder(self.config)

  recorder does three things:

  • report() called after each epoch to report information regarding training
  • save_model() called to save the model can be modified to store the best model base on different indicaion or just store the model after each/certain epoch the suffix of the checkpoints should be .ckpt
  • summary() sum up the result of the entire training

  config file in configs/pipelines/train

  add a new recoder in openood/recorders

• start training

  # example code:
  for epoch_idx in range(1, self.config.optimizer.num_epochs + 1):
        # train the model
        net, train_metrics = trainer.train_epoch(epoch_idx)
        test_metrics = evaluator.eval(net,
                                      test_loader,
                                      epoch_idx=epoch_idx)
        # save model and report the result
        recorder.save_model(net, test_metrics)
        recorder.report(train_metrics, test_metrics)
    recorder.summary()

  during the training process

  train_epoch() from trainer,

  eval() from evaluator,

  save_model() and report() from recorder are called per epoch

• evaluation

  eval() from evaluator is called and return a metrix containg results then print out the results