How to get the mean value of massive activation

[pengyao96]

1. How to get the mean value of massive activation？e.g. 2546.8/-1502.0 in hook.py
2. Mean value is still large, what is the difference between using the mean value and using the original value?

[Eric-mingjie]

Hi, Thanks for you interest in our work.

To get the mean value, we simply evaluate 100 sequences from RedPajama and record the value of massive activations of each sequence.

In practice, we find no performance difference between using the mean value or the original value. But the original value may vary by each sequence, see Table 2, so it might be hard to justify which original value to use.