Model_choice.py: refactor model, loss and optimizer instantiation and loading

[remtav]

Model_choice.py has gotten messy and cluttered over the years and needs a bit of refactoring. This refactoring should be done before addressing #246 and #152.

Current state of things

• set_hyperparamers() function has very vague prupose of "set[ting] hyperparameters based on values provided in yaml config file";
• net() function is supposed to "Define the neural net", but in reality it's an all-in-one vague function that does the following:

1. Defines net architecture;
2. Reads a checkpoint to memory with load_checkpoint() (from a .pth.tar file as created by torch.save);
3. Returns if net() is called from inference or continues with the following if net() is called from train mode:
4. If more than one gpu is requested, determines which gpus are available based on user-inputted threshold for GPU's available RAM and usage %;
5. Sets model to DataParallel if more than one gpu is requested and available;
6. Sets main device with set_device() function;
7. Pushes model to main device;
8. Calls set_hyperparamters() (see above);
9. Pushes loss to device;
10. Returns 7 (!!) objects: model, model_name, loss, etc.

Suggested solution (high level)

All these steps could be better separated in small, dedicated functions of their own:

• read_checkpoint(): renamed version of load_checkpoint (prevents confusion with load_state_dict function). Although it derives from torch.load(checkpoint)'s function, this function really just reads a checkpoint in memory from a .pth.tar file to a Python dict containing weights, optimizer, etc.
• define_net_architecture(): define the model architecture from config parameters (i.e. create model with randomly initialized weights)
• adapt_checkpoint_to_dp_model(): for use at test loop during training only, adapts a generic checkpoint to be loaded to a DataParallel model as is done in load_from_checkpoint (if model is DataParallel object)
• define_loss(): calls verify_weights() and instantiates a loss criterion
• define_optimizer(): instantiates optimizer with learning rate, weight decay, etc.

These functions would be called only when necessary in 3 main places:

1. Beginning of train_segmentation:

• read checkpoint to be loaded checkpoint for model weights and optimizer;
• define net architecture;
• load model weights with pytorch's [model_object].load_state_dict() method;
• define loss;
• define optimizer;
• load optimizer from checkpoint;

2. Test loop in train_segmentation:

• Load best checkpoint to model (adapt checkpoint keys if model is a nn.DataParallel instance using dedicated function);

3. Beginning of inference:

• override architecture, input bands, output classes from checkpoint's params;
• define net architecture
• load weights from provided checkpoint to model using pytorch's [model_object].load_state_dict() method