Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

llama : fix n_rot default #8348

Merged
merged 1 commit into from
Jul 7, 2024
Merged

llama : fix n_rot default #8348

merged 1 commit into from
Jul 7, 2024

Conversation

ggerganov
Copy link
Owner

@ggerganov ggerganov commented Jul 7, 2024

fix #8246 #8251

The logic for determining default n_rot parameter did not take into account LLM_KV_ATTENTION_KEY_LENGTH overrides. This lead to invalid context shift for Gemma2 models:

# gemma-2-27b-it
n_embd = 4608
n_head = 32
n_embd / n_head = 144
n_embd_head_k   = 128   # (from KV header)

# old
n_rot = 144

# new
n_rot = 128
# gemma-2-9b-it
n_embd = 3584
n_head = 16
n_embd / n_head = 224
n_embd_head_k   = 256   # (from KV header)

# old
n_rot = 224

# new
n_rot = 256

@ggerganov ggerganov merged commit b504008 into master Jul 7, 2024
59 checks passed
@ggerganov ggerganov deleted the gg/fix-n_rot branch July 7, 2024 11:59
Nexesenex pushed a commit to Nexesenex/croco.cpp that referenced this pull request Jul 8, 2024
Nexesenex pushed a commit to Nexesenex/croco.cpp that referenced this pull request Jul 8, 2024
Nexesenex pushed a commit to Nexesenex/croco.cpp that referenced this pull request Jul 11, 2024
Nexesenex pushed a commit to Nexesenex/croco.cpp that referenced this pull request Jul 11, 2024
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Jul 13, 2024
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Jul 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug: gemma 2 27B GGML_ASSERT n_dims <= ne0
1 participant