Implement optuna hyperparameter optimization #278
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…on (add returns to train_segmentation.py and run_gdl()) metrics.py: add try/except if missing key model_choice.py: temporarily raise gpu threshold to 100% until issue NRCan#246 is addressed train_segmentation.py: - fix bug for dontcare value in metrics - always calculate metrics at test time inference_segmentation.py: warn if inference has only background values test hyperparameter optimization in CI with github actions
Merged
remtav
commented
Mar 16, 2022
models/model_choice.py
Outdated
| @@ -244,7 +244,7 @@ def net(model_name: str, | |||
| else: | |||
| checkpoint = None | |||
| # list of GPU devices that are available and unused. If no GPUs, returns empty list | |||
| gpu_devices_dict = get_device_ids(num_devices) | |||
| gpu_devices_dict = get_device_ids(num_devices, max_used_perc=100, max_used_ram_perc=100) # FIXME: set back to default after issue #246 | |||
There was a problem hiding this comment.
the gpus don't empty between each run, so gpu usage threshold must be 100% otherwise each training after the first one runs on cpu (ie gpus get excluded)
There was a problem hiding this comment.
Or get_device_ids should be called once, before all runs.
remtav
commented
Mar 16, 2022
| @@ -432,12 +435,13 @@ def evaluation(eval_loader, | |||
| a, segmentation = torch.max(outputs_flatten, dim=1) | |||
| eval_metrics = iou(segmentation, labels_flatten, batch_size, num_classes, eval_metrics) | |||
| eval_metrics = report_classification(segmentation, labels_flatten, batch_size, eval_metrics, | |||
| ignore_index=eval_loader.dataset.dontcare) | |||
| elif (dataset == 'tst') and (batch_metrics is not None): | |||
| ignore_index=dontcare) | |||
There was a problem hiding this comment.
fixes missing attribute bug. Dontcare value was previously stored in dataloader object, but not anymore (since when?)
remtav
commented
Mar 16, 2022
| elif (dataset == 'tst') and (batch_metrics is not None): | ||
| ignore_index=dontcare) | ||
| elif dataset == 'tst': | ||
| batch_metrics = True |
There was a problem hiding this comment.
force metrics at test time. I don't see any reason why we shouldn'y systematically output metrics at test time.
# Conflicts: # gdl_hyperopt_template.py # train_segmentation.py
mpelchat04
previously approved these changes
Mar 17, 2022
models/model_choice.py
Outdated
| @@ -244,7 +244,7 @@ def net(model_name: str, | |||
| else: | |||
| checkpoint = None | |||
| # list of GPU devices that are available and unused. If no GPUs, returns empty list | |||
| gpu_devices_dict = get_device_ids(num_devices) | |||
| gpu_devices_dict = get_device_ids(num_devices, max_used_perc=100, max_used_ram_perc=100) # FIXME: set back to default after issue #246 | |||
There was a problem hiding this comment.
Or get_device_ids should be called once, before all runs.
valhassan
previously approved these changes
Mar 17, 2022
inception: unrelevant without classification ternausnet: unused fcn_resnet101: unadapted for >3 bands
…ce.py to DeepLabV3_dualhead class (NRCan#290) * replace torchvision's version of deeplabv3 to smp's version (>3 band implementation, reuse pretrained weights, etc.) encapsulate dual head deeplabv3 to deeplabv3_dualhead.py create temporary unit test in model_choice.py fcn_resnet101: unadapted for >3 bands for baseline deeplabv3, use smp's version as is. * oups, forgot to commit deeplabv3_dualhead.py
- set gpu max ram threshold to 100% only when using optuna (don't hardcode in model_choice.py)
| @@ -1,5 +1,5 @@ | |||
| # @package _global_ | |||
| model: | |||
| model_name: deeplabv3_resnet101 | |||
| model_name: deeplabv3_pretrained | |||
There was a problem hiding this comment.
I think that keeping the encoder in the name can be nice
CharlesAuthier
previously approved these changes
Mar 21, 2022
|
It seems geo-deep-learning will rely more and more on pytorch lightning (PL) and torchgeo (TG) for training. The hyperparameter optimization add-on is not a priority until the transition to PL and TG is cleared. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…on (add returns to train_segmentation.py and run_gdl())
metrics.py: add try/except if missing key
model_choice.py: temporarily raise gpu threshold to 100% until issue #246 is addressed
train_segmentation.py:
inference_segmentation.py: warn if inference has only background values
test hyperparameter optimization in CI with github actions
closes #243