[MODEL] `Apertus` and `XIELU` #23068

EduardDurech · 2025-08-18T00:54:02Z

Pre-release of Apertus from the Swiss AI Initiative

Main modifications from Llama

xIELU Activation
QK-norm

Associated Transformers PR huggingface/transformers#39381

github-actions · 2025-08-18T00:54:10Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

gemini-code-assist

Code Review

This pull request introduces support for the Apertus model and its xIELU activation function. The changes include a new model definition for Apertus, which correctly implements QK-norm, and a new XIELU activation layer. The model is also added to the test suite and model registries.

My review focuses on the implementation of the new activation function. I've found a critical issue in the XIELU's forward method that would prevent the use of its CUDA kernel under torch.compile, leading to a performance degradation. The provided suggestion should resolve this issue. The rest of the implementation appears to be correct and follows vLLM's conventions.

vllm/model_executor/layers/activation.py

mergify · 2025-08-26T02:43:51Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @EduardDurech.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

DarkLight1337

Thanks for contributing! I think the implementation looks good overall but please fix pre-commit

EduardDurech · 2025-08-28T10:09:47Z

@DarkLight1337 I'm not sure what the errors are, does it just want to convert to forward_native and forward_cuda? Or to change to NN.Module if we keep forward? The activation is still in development so not sure the right way to do it

Comments @AllenHaoHuang?

DarkLight1337 · 2025-08-28T10:13:30Z

I think you can just assert self._xielu_cuda_obj is not None

martinjaggi · 2025-08-28T11:19:06Z

just a quick comment that hugging face transformers has already merged it into their main branch now, which should make next steps here easier?

also beta and eps additional parameters are now also saved in the HF checkpoint. is this taken into account here yet?

EduardDurech · 2025-08-28T11:50:39Z

@martinjaggi yes all my implementations are the most up to date

Co-authored-by: AllenHaoHuang <allenhuangdd@gmail.com> Signed-off-by: EduardDurech <39579228+EduardDurech@users.noreply.github.com>

Signed-off-by: EduardDurech <39579228+EduardDurech@users.noreply.github.com>

EduardDurech · 2025-08-28T17:29:09Z

@DarkLight1337 not sure what the current issues are but fixed pre-commit

DarkLight1337 · 2025-08-28T17:35:12Z

Fastcheck is optional so you can ignore it

EduardDurech · 2025-08-28T23:18:06Z

@DarkLight1337 I think the PR ci is just because huggingface/transformers#39381 was only recently merged

tests/models/registry.py

Signed-off-by: EduardDurech <39579228+EduardDurech@users.noreply.github.com>

EduardDurech · 2025-09-02T13:08:19Z

@DarkLight1337 we have a small fix for activation parameter loading for the model #24100

Signed-off-by: EduardDurech <39579228+EduardDurech@users.noreply.github.com> Co-authored-by: AllenHaoHuang <allenhuangdd@gmail.com>

Pre-release of Apertus from the Swiss AI Initiative Main modifications from Llama - xIELU Activation - QK-norm Associated Transformers PR huggingface/transformers#39381 Associated vLLM PR vllm-project/vllm#23068 Associated SGLang PR sgl-project/sglang#9774 GSM8K <img width="430" height="262" alt="image" src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed" /> <img width="436" height="266" alt="image" src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08" />

omaraljasem272144 · 2025-09-14T13:08:34Z

ا

Pre-release of Apertus from the Swiss AI Initiative Main modifications from Llama - xIELU Activation - QK-norm Associated Transformers PR huggingface/transformers#39381 Associated vLLM PR vllm-project/vllm#23068 Associated SGLang PR sgl-project/sglang#9774 GSM8K <img width="430" height="262" alt="image" src="https://github.com/user-attachments/assets/8b2d5188-834b-4a8c-828e-2d0aa2ccffed" /> <img width="436" height="266" alt="image" src="https://github.com/user-attachments/assets/57241a73-3150-474a-a4fb-222e33a0de08" />

Signed-off-by: EduardDurech <39579228+EduardDurech@users.noreply.github.com> Co-authored-by: AllenHaoHuang <allenhuangdd@gmail.com>

mergify bot added the new-model Requests to new models label Aug 18, 2025

gemini-code-assist bot reviewed Aug 18, 2025

View reviewed changes

vllm/model_executor/layers/activation.py Outdated Show resolved Hide resolved

EduardDurech force-pushed the model/apertus branch 3 times, most recently from 732dd4d to e26ab90 Compare August 18, 2025 21:35

mergify bot added the needs-rebase label Aug 26, 2025

EduardDurech force-pushed the model/apertus branch from f61e6c0 to 505301f Compare August 26, 2025 12:45

mergify bot removed the needs-rebase label Aug 26, 2025

EduardDurech force-pushed the model/apertus branch from 505301f to a3e809c Compare August 27, 2025 23:24

EduardDurech marked this pull request as ready for review August 27, 2025 23:24

EduardDurech requested review from DarkLight1337 and ywang96 as code owners August 27, 2025 23:24

DarkLight1337 reviewed Aug 28, 2025

View reviewed changes

Apertus and XIELU

75a7eec

Co-authored-by: AllenHaoHuang <allenhuangdd@gmail.com> Signed-off-by: EduardDurech <39579228+EduardDurech@users.noreply.github.com>

EduardDurech force-pushed the model/apertus branch from 581b566 to 4e29220 Compare August 28, 2025 16:51

Assert XIELU CUDA obj is not None

81ab8f7

Signed-off-by: EduardDurech <39579228+EduardDurech@users.noreply.github.com>

EduardDurech force-pushed the model/apertus branch from 4e29220 to 81ab8f7 Compare August 28, 2025 17:21

DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 28, 2025

DarkLight1337 approved these changes Aug 28, 2025

View reviewed changes

DarkLight1337 enabled auto-merge (squash) August 28, 2025 17:30

DarkLight1337 reviewed Aug 29, 2025

View reviewed changes

tests/models/registry.py Show resolved Hide resolved

Version constraint

5d7b48c

Signed-off-by: EduardDurech <39579228+EduardDurech@users.noreply.github.com>

auto-merge was automatically disabled August 29, 2025 04:04
Head branch was pushed to by a user without write access

DarkLight1337 enabled auto-merge (squash) August 29, 2025 04:34

test_registry_imports Transformers version compatibility

5f37650

Signed-off-by: EduardDurech <39579228+EduardDurech@users.noreply.github.com>

auto-merge was automatically disabled August 29, 2025 09:56
Head branch was pushed to by a user without write access

DarkLight1337 merged commit 1cf3753 into vllm-project:main Aug 29, 2025
42 checks passed

malte-aws mentioned this pull request Sep 1, 2025

feat: Swiss ai apertus aws-samples/sagemaker-genai-hosting-examples#112

Merged

EduardDurech mentioned this pull request Sep 1, 2025

[model] feat: Add Apertus volcengine/verl#3295

Merged

theo77186 mentioned this pull request Sep 2, 2025

Feature Request: Apertus models support ggml-org/llama.cpp#15748

Closed

4 tasks

DarkLight1337 mentioned this pull request Sep 8, 2025

[Bugfix] Fix Apertus HF repo name #24447

Merged

5 tasks

eicherseiji pushed a commit to eicherseiji/vllm that referenced this pull request Sep 9, 2025

[MODEL] Apertus and XIELU (vllm-project#23068)

eea03c5

Signed-off-by: EduardDurech <39579228+EduardDurech@users.noreply.github.com> Co-authored-by: AllenHaoHuang <allenhuangdd@gmail.com>

FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request Sep 25, 2025

[MODEL] Apertus and XIELU (vllm-project#23068)

36f7479

Signed-off-by: EduardDurech <39579228+EduardDurech@users.noreply.github.com> Co-authored-by: AllenHaoHuang <allenhuangdd@gmail.com>

Uh oh!

[MODEL] Apertus and XIELU #23068

[MODEL] Apertus and XIELU #23068

Uh oh!

Conversation

EduardDurech commented Aug 18, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 18, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

mergify bot commented Aug 26, 2025

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

EduardDurech commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkLight1337 commented Aug 28, 2025

Uh oh!

martinjaggi commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EduardDurech commented Aug 28, 2025

Uh oh!

EduardDurech commented Aug 28, 2025

Uh oh!

DarkLight1337 commented Aug 28, 2025

Uh oh!

EduardDurech commented Aug 28, 2025

Uh oh!

Uh oh!

Uh oh!

EduardDurech commented Sep 2, 2025

Uh oh!

omaraljasem272144 commented Sep 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[MODEL] `Apertus` and `XIELU` #23068

[MODEL] `Apertus` and `XIELU` #23068

EduardDurech commented Aug 18, 2025 •

edited by github-actions bot

Loading

EduardDurech commented Aug 28, 2025 •

edited

Loading

martinjaggi commented Aug 28, 2025 •

edited

Loading