Does anyone know if this work is implemented on llama?

[wyxscir]

Does anyone know if this work is implemented on llama? Or is there any similar dynamic pruning work on llama?

[XieWeikai]

This method doesn't work very well on the llama. Llama uses the SiLU activation function, and its inherent sparsity is not very high.
A work mentioned that it is possible to replace SiLU with ReLU and retrain llama to improve sparsity.

[wyxscir]

This method doesn't work very well on the llama. Llama uses the SiLU activation function, and its inherent sparsity is not very high. A work mentioned that it is possible to replace SiLU with ReLU and retrain llama to improve sparsity.

thank you

[wyxscir]

This method doesn't work very well on the llama. Llama uses the SiLU activation function, and its inherent sparsity is not very high. A work mentioned that it is possible to replace SiLU with ReLU and retrain llama to improve sparsity.

“ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models”，
What you mentioned maybe be this work

[quaternior]

This method doesn't work very well on the llama. Llama uses the SiLU activation function, and its inherent sparsity is not very high. A work mentioned that it is possible to replace SiLU with ReLU and retrain llama to improve sparsity.

Hello, where can I find about the mention?

[XieWeikai]

This method doesn't work very well on the llama. Llama uses the SiLU activation function, and its inherent sparsity is not very high. A work mentioned that it is possible to replace SiLU with ReLU and retrain llama to improve sparsity.

“ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models”， What you mentioned maybe be this work