smart ratio v3 init #25
Conversation
i'm sure i broke a few things, but this should be more amenable to autodiff
turns out the device_error was due to this, hiding under the pretense of multithreading
SmartRatio.py
Outdated
|
|
||
| # if we use modified version of smart ratio | ||
| if parser_args.sr_version == 2: | ||
| if parser_args.sr_version >= 2: | ||
| if parser_args.arch.lower() != 'resnet20': |
There was a problem hiding this comment.
This stuff is okay while debugging but make sure to remove it before we merge :)
There was a problem hiding this comment.
sry I didn't get it.. so we need to remove this line 93 handling different SR versions?
SmartRatio.py
Outdated
|
|
||
| if parser_args.sr_version == 2: | ||
| # followed the result in https://github.com/ksreenivasan/results_repo_pruning/blob/master/per_layer_sparsity_resnet20/hc_iter.csv | ||
| if parser_args.smart_ratio == 0.9856: # 1.44% sparsity |
There was a problem hiding this comment.
Can we move these things to the config or a csv file?
There was a problem hiding this comment.
yeah let me change it to load values from a csv file :)
| @@ -74,7 +76,7 @@ def main_worker(gpu, ngpus_per_node): | |||
| # model = model.module | |||
| if parser_args.random_subnet: | |||
| test_random_subnet(model, data, criterion, parser_args, result_root, parser_args.smart_ratio) | |||
| return | |||
| exit() | |||
There was a problem hiding this comment.
as I know, we want to end program from here, since test_random_subnet will do until finetune & save the result.
I found that the program is not exiting if we use "return" here, so added exit() instead.
main_utils.py
Outdated
| def init_smart_ratio(parser_args): | ||
| if parser_args.arch.lower() == 'resnet20': | ||
| if parser_args.target_sparsity == 3.72: | ||
| parser_args.init_sr = np.array([41.43518518518518, 24.305555555555557, 21.875, 19.57465277777778, 17.36111111111111, |
There was a problem hiding this comment.
Like above. Would be preferable if this is a config thing, or reads from a csv.
| @@ -90,6 +97,23 @@ def backward(ctx, g_1, g_2): | |||
| return g_1, g_2, None, None, None | |||
|
|
|||
|
|
|||
| class GetRandomSubnet(autograd.Function): | |||
There was a problem hiding this comment.
This is the new function. The reason I had to do this is because the parameters are now 1 dimensional. So the gradients are of different shape
There was a problem hiding this comment.
Thank you so much!
| if parser_args.algo == 'pt_sr': | ||
| #self.layer_score = nn.Parameter(torch.Tensor([parser_args.init_sr[parser_args.current_layer]])) | ||
| self.layer_weight_ratio = nn.Parameter(torch.Tensor(1)) | ||
| self.layer_weight_ratio.data = torch.Tensor([0.5]) |
There was a problem hiding this comment.
this is where ratios are initialized at 0.5
|
https://github.com/ksreenivasan/pruning_is_enough/blob/master/utils/net_utils.py#L531 |
Do not merge for now