How to get the super weight for the OLMo-1B-0724-hf

[daidaiershidi]

Thanks for bringing the great work.

I would like to know how to get the super weight for the OLMo-1B-0724-hf, which is [(1, 1764, 1710), (2, 1764, 8041)] in outliers/model.py.

When I run

python analyze.py plot-down-proj-input-output allenai/OLMo-1B-0724-hf mlp.down_proj

I get

Activation spikes for mlp.down_proj input:
 - layer 1, value -401.75, index (0, 0, 1710)
 - layer 2, value 68.4375, index (0, 0, 8041)
 - layer 3, value -63.65625, index (0, 0, 1902)
 - layer 4, value 72.875, index (0, 0, 556)
 - layer 15, value -283.0, index (0, 0, 6840)
Plot saved to 'outputs/figures/input_down_proj.pdf'
largest_activations_down_proj_input=[-3.236328125, -401.75, 68.4375, -63.65625, 72.875, -10.828125, -26.546875, 3.421875, 2.5625, -9.390625, -3.912109375, 3.091796875, -13.0390625, -4.73828125, -7.7734375, -283.0]
Activation spikes for mlp.down_proj output:
 - layer 1, value -262.0, index (0, 0, 1764)
 - layer 3, value -64.75, index (0, 0, 1764)
 - layer 15, value 415.0, index (0, 0, 1764)
Plot saved to 'outputs/figures/output_down_proj.pdf'
largest_activations_down_proj_output=[4.6015625, -262.0, -41.8125, -64.75, -36.65625, -3.052734375, -9.03125, -1.447265625, 0.94921875, 1.439453125, 1.041015625, 0.9912109375, 5.9453125, 0.77978515625, 0.98974609375, 415.0]

How do I get the right results?

[kaiyliu]

And I found that the analyse code doesn't work with hooks on some models, for example WizardLMTeam/WizardLM-13B-V1.2, why is that?