Feature Request: Meta releases Layer Skip, an end-to-end solution for accelerating LLMs

[mirek190]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

https://x.com/AIatMeta/status/1851327605716435011?t=uCwZiiCcZqPQz0O9NjLfoQ&s=19

Motivation

Meta releases Layer Skip, an end-to-end solution for accelerating LLMs

Possible Implementation

No response

[BarfingLemurs]

From the title, sounds a lot like #3565, which the gain is dependent on the efficient GPU kernels (especially quantized models) to pull off the speed boost, the # FLOPS would also make a big difference. Since meta has done some work on this, check if they already support quantized models. Most researchers test their optimizations with H100, A100 at fp16.