Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

smart ratio v3 init #25

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open

smart ratio v3 init #25

wants to merge 34 commits into from

Conversation

jysohn1108
Copy link
Collaborator

Do not merge for now

jysohn1108 and others added 7 commits February 4, 2022 01:19
i'm sure i broke a few things, but this should be more amenable
to autodiff
turns out the device_error was due to this, hiding under the pretense
of multithreading
SmartRatio.py Outdated

# if we use modified version of smart ratio
if parser_args.sr_version == 2:
if parser_args.sr_version >= 2:
if parser_args.arch.lower() != 'resnet20':
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This stuff is okay while debugging but make sure to remove it before we merge :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sry I didn't get it.. so we need to remove this line 93 handling different SR versions?

SmartRatio.py Outdated

if parser_args.sr_version == 2:
# followed the result in https://github.com/ksreenivasan/results_repo_pruning/blob/master/per_layer_sparsity_resnet20/hc_iter.csv
if parser_args.smart_ratio == 0.9856: # 1.44% sparsity
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we move these things to the config or a csv file?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah let me change it to load values from a csv file :)

@@ -74,7 +76,7 @@ def main_worker(gpu, ngpus_per_node):
# model = model.module
if parser_args.random_subnet:
test_random_subnet(model, data, criterion, parser_args, result_root, parser_args.smart_ratio)
return
exit()
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm guessing this is debug?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as I know, we want to end program from here, since test_random_subnet will do until finetune & save the result.
I found that the program is not exiting if we use "return" here, so added exit() instead.

main_utils.py Outdated
def init_smart_ratio(parser_args):
if parser_args.arch.lower() == 'resnet20':
if parser_args.target_sparsity == 3.72:
parser_args.init_sr = np.array([41.43518518518518, 24.305555555555557, 21.875, 19.57465277777778, 17.36111111111111,
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like above. Would be preferable if this is a config thing, or reads from a csv.

@@ -90,6 +97,23 @@ def backward(ctx, g_1, g_2):
return g_1, g_2, None, None, None


class GetRandomSubnet(autograd.Function):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the new function. The reason I had to do this is because the parameters are now 1 dimensional. So the gradients are of different shape

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much!

if parser_args.algo == 'pt_sr':
#self.layer_score = nn.Parameter(torch.Tensor([parser_args.init_sr[parser_args.current_layer]]))
self.layer_weight_ratio = nn.Parameter(torch.Tensor(1))
self.layer_weight_ratio.data = torch.Tensor([0.5])
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is where ratios are initialized at 0.5

@ksreenivasan
Copy link
Owner

https://github.com/ksreenivasan/pruning_is_enough/blob/master/utils/net_utils.py#L531
You'll have to change some code here (because it calls GetSubnet() when it should really be calling GetRandomSubnet()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants