Code

Add GroupedQueryAttention layer#18488

Code

Merged

fchollet merged 38 commits intokeras-team:masterkeras-team/keras:masterfrom awsaf49:gqaawsaf49/keras:gqaCopy head branch name to clipboardOct 22, 2023

+643

Commits

Commits on Sep 25, 2023

• add: gqa barebone

  awsaf49

  committed
  e6bc383

  Copy full SHA for e6bc383

Commits on Sep 26, 2023

• update: avoid abbr. in signature

  awsaf49

  committed
  04b6fe7

  Copy full SHA for 04b6fe7

• update: use fully-spelled snake_case

  awsaf49

  committed
  85717ea

  Copy full SHA for 85717ea

Commits on Oct 14, 2023

• add: initializers, regularizers, constraint

  awsaf49

  committed
  18e319c

  Copy full SHA for 18e319c

• update: mask to attention_mask

  awsaf49

  committed
  2bc6e46

  Copy full SHA for 2bc6e46

Commits on Oct 15, 2023

• add: dropout

  awsaf49

  committed
  004dea4

  Copy full SHA for 004dea4

• add: softmax from ops

  awsaf49

  committed
  acd8a7e

  Copy full SHA for acd8a7e

• update: use softmax for masking

  Show description for b0d45db
  awsaf49

  committed
  b0d45db

  Copy full SHA for b0d45db

• add: import in __init__

  awsaf49

  committed
  cbfc9d2

  Copy full SHA for cbfc9d2

• update: filename

  awsaf49

  committed
  660429a

  Copy full SHA for 660429a

• add: compute_output_shape

  awsaf49

  committed
  df13321

  Copy full SHA for df13321

• add: query-key-value & causal mask

  awsaf49

  committed
  1f93104

  Copy full SHA for 1f93104

• update: use EinsumDense and einsum

  awsaf49

  committed
  7c6f063

  Copy full SHA for 7c6f063

• update: remove Dense import

  awsaf49

  committed
  7d4b8c9

  Copy full SHA for 7d4b8c9

• fix: __init__ in layer for isort

  awsaf49

  committed
  c6b73cc

  Copy full SHA for c6b73cc

• update: docstring

  awsaf49

  committed
  d34de7b

  Copy full SHA for d34de7b

• add: simple test

  awsaf49

  committed
  321914d

  Copy full SHA for 321914d

• update: support_masking False in test

  awsaf49

  committed
  b361be5

  Copy full SHA for b361be5

• fix: error due to query & key-value seq_len mismatch

  awsaf49

  committed
  c0010e3

  Copy full SHA for c0010e3

• Revert "update: support_masking False in test"

  Show description for 5ea5379
  awsaf49

  committed
  5ea5379

  Copy full SHA for 5ea5379

• update: use_bias = True as default

  awsaf49

  committed
  6cbfd39

  Copy full SHA for 6cbfd39

• add: support_masking

  awsaf49

  committed
  af9051f

  Copy full SHA for af9051f

• add: more tests

  awsaf49

  committed
  8b1489d

  Copy full SHA for 8b1489d

• update: code format with black

  awsaf49

  committed
  334c776

  Copy full SHA for 334c776

• remove: high dim attention test

  Show description for 81ca49c
  awsaf49

  committed
  81ca49c

  Copy full SHA for 81ca49c

• fix: remove undefined arg num_head

  awsaf49

  committed
  710a661

  Copy full SHA for 710a661

• add: shape mismatch test

  awsaf49

  committed
  2a92866

  Copy full SHA for 2a92866

• add: initializer test

  awsaf49

  committed
  30b88e8

  Copy full SHA for 30b88e8

• update: softmax for mask propagation

  awsaf49

  committed
  07ebe3b

  Copy full SHA for 07ebe3b

• update: code format

  awsaf49

  committed
  5514345

  Copy full SHA for 5514345

Commits on Oct 16, 2023

• add: mask propagation test

  awsaf49

  committed
  e70dbb1

  Copy full SHA for e70dbb1

• add: masking test

  awsaf49

  committed
  227325a

  Copy full SHA for 227325a

• add: correctness test

  awsaf49

  committed
  35f7e80

  Copy full SHA for 35f7e80

• update: output shape test for mqa, gqa & mha

  Show description for d018a7e
  awsaf49

  committed
  d018a7e

  Copy full SHA for d018a7e

• fix: code format for isort

  awsaf49

  committed
  a3b89dc

  Copy full SHA for a3b89dc

Commits on Oct 19, 2023

• add: divisible error check

  Show description for 51219fc
  awsaf49

  committed
  51219fc

  Copy full SHA for 51219fc

• add: shape of attention_scores

  awsaf49

  committed
  75b309c

  Copy full SHA for 75b309c

• update: different letters for query and key-value heads

  Show description for 1535b7e
  awsaf49

  committed
  1535b7e

  Copy full SHA for 1535b7e