Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Umt5] Add google's umt5 to transformers #24477

Merged
merged 85 commits into from
Jul 3, 2023
Merged
Changes from 1 commit
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
7997b9d
add tokenization template
ArthurZucker Jun 20, 2023
92ddd5f
update conversion script
ArthurZucker Jun 26, 2023
9a50e90
update modeling code
ArthurZucker Jun 26, 2023
9f38d02
update
ArthurZucker Jun 26, 2023
5be546f
update convert checkpoint
ArthurZucker Jun 26, 2023
ef0f5ab
update modeling
ArthurZucker Jun 27, 2023
ec27200
revert changes on convert script
ArthurZucker Jun 27, 2023
74cb2b1
new conversion script for new format
ArthurZucker Jun 27, 2023
df57c75
correct position bias
ArthurZucker Jun 27, 2023
0f5c507
cleaning a bit
ArthurZucker Jun 27, 2023
eac93b8
Credit co authors
ArthurZucker Jun 28, 2023
083e3a2
styling
ArthurZucker Jun 28, 2023
5a45e1b
Add docq
ArthurZucker Jun 28, 2023
ed9da8f
fix copies
ArthurZucker Jun 28, 2023
a5cfeb8
add co author
Jun 28, 2023
abf02ed
Other Author
Jun 28, 2023
24e89b1
Merge branch 'main' of https://github.com/huggingface/transformers in…
ArthurZucker Jun 28, 2023
6c5d9d4
Merge branch 'main' of https://github.com/huggingface/transformers in…
ArthurZucker Jun 28, 2023
6def1cc
add testing
ArthurZucker Jun 28, 2023
7e4f4e6
nit
ArthurZucker Jun 28, 2023
252e123
Update docs/source/en/model_doc/umt5.mdx
ArthurZucker Jun 28, 2023
25507fa
fix t5
ArthurZucker Jun 28, 2023
e5c0d9b
Merge branch 'add-umt5' of https://github.com/ArthurZucker/transforme…
ArthurZucker Jun 28, 2023
c804387
actual fix?
ArthurZucker Jun 28, 2023
4482271
revert wrong changes
ArthurZucker Jun 28, 2023
3ce31e2
remove
ArthurZucker Jun 28, 2023
f22c72c
update test
ArthurZucker Jun 28, 2023
d058081
more fixes
ArthurZucker Jun 29, 2023
f8e4518
revert some changes
ArthurZucker Jun 29, 2023
e8c30c6
Merge branch 'add-umt5' of https://github.com/ArthurZucker/transforme…
ArthurZucker Jun 29, 2023
1ccf79e
add SPIECE_UNDERLINE
ArthurZucker Jun 29, 2023
d761343
add a commone xample
ArthurZucker Jun 29, 2023
62987df
Merge branch 'main' of https://github.com/huggingface/transformers in…
ArthurZucker Jun 29, 2023
0e6bc03
Merge branch 'add-umt5' of https://github.com/ArthurZucker/transforme…
ArthurZucker Jun 29, 2023
376feb0
upfate
ArthurZucker Jun 29, 2023
94abb37
fix copies
ArthurZucker Jun 29, 2023
52ccdfe
revert changes on t5 conversion script
ArthurZucker Jun 29, 2023
4e2afe1
revert bytefallback changes since there was no addition yet
ArthurZucker Jun 29, 2023
26c130a
Merge branch 'add-umt5' of https://github.com/ArthurZucker/transforme…
ArthurZucker Jun 29, 2023
8edffae
fixup
ArthurZucker Jun 29, 2023
e1431c4
fixup
ArthurZucker Jun 29, 2023
606b3ca
ingore umt5 cutom testing folder
ArthurZucker Jun 29, 2023
afecb5c
fix readmes
ArthurZucker Jun 29, 2023
4a34822
revertT5 changes
ArthurZucker Jun 29, 2023
28617b0
same outputs
ArthurZucker Jun 29, 2023
0dc9440
fixup
ArthurZucker Jun 29, 2023
7f489c8
update example
ArthurZucker Jun 29, 2023
38b425b
Apply suggestions from code review
ArthurZucker Jun 29, 2023
a420c73
style
ArthurZucker Jun 29, 2023
f71414b
Merge branch 'add-umt5' of https://github.com/ArthurZucker/transforme…
ArthurZucker Jun 29, 2023
88585a7
draft addition of all new files
ArthurZucker Jun 30, 2023
e9113ee
Merge branch 'main' of https://github.com/huggingface/transformers in…
ArthurZucker Jun 30, 2023
c6bcb35
current update
ArthurZucker Jun 30, 2023
c6dd239
fix attention and stuff
ArthurZucker Jun 30, 2023
273e56f
Merge branch 'main' of https://github.com/huggingface/transformers in…
ArthurZucker Jun 30, 2023
be06c14
finish refactoring
ArthurZucker Jun 30, 2023
6217a0b
auto config
ArthurZucker Jun 30, 2023
cd7c8f1
fixup
ArthurZucker Jun 30, 2023
23ad8ba
more nits
ArthurZucker Jun 30, 2023
dea1b02
add umt5 to init
ArthurZucker Jun 30, 2023
b473697
use md format
ArthurZucker Jun 30, 2023
5bdb793
Update README.md
ArthurZucker Jun 30, 2023
3180b55
revert changes on mt5
ArthurZucker Jun 30, 2023
3b0e3af
revert mt4 changes
ArthurZucker Jun 30, 2023
d9e4aa5
update test
ArthurZucker Jun 30, 2023
1d3d449
Merge branch 'add-umt5' of https://github.com/ArthurZucker/transforme…
ArthurZucker Jun 30, 2023
9c5c46a
more fixes
ArthurZucker Jun 30, 2023
f49cd73
add to mapping
ArthurZucker Jun 30, 2023
223151c
fix-copies
ArthurZucker Jun 30, 2023
50b03e9
fix copies
ArthurZucker Jun 30, 2023
48b0c41
foix retain grad
ArthurZucker Jun 30, 2023
ee6a9b8
fix some tests
ArthurZucker Jun 30, 2023
f2a2046
nits
ArthurZucker Jun 30, 2023
f772cbe
done
ArthurZucker Jun 30, 2023
a00270b
Update src/transformers/models/umt5/modeling_umt5.py
ArthurZucker Jun 30, 2023
324acb3
Update docs/source/en/model_doc/umt5.md
ArthurZucker Jun 30, 2023
d13c544
Update src/transformers/models/umt5/__init__.py
ArthurZucker Jul 1, 2023
4fbae39
Update docs/source/en/model_doc/umt5.md
ArthurZucker Jul 1, 2023
00063ff
Update src/transformers/models/umt5/modeling_umt5.py
ArthurZucker Jul 1, 2023
3e26583
update conversion script + use google checkpoints
ArthurZucker Jul 2, 2023
0a3d470
nits
ArthurZucker Jul 2, 2023
cdbaf71
update test and modelling
ArthurZucker Jul 2, 2023
5be4769
stash slow convert
ArthurZucker Jul 2, 2023
00c7c87
update fixupd
ArthurZucker Jul 2, 2023
c0b0a67
don't change slow
ArthurZucker Jul 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
current update
ArthurZucker committed Jun 30, 2023
commit c6bcb350d0a93bd640e3b7ac4c6e7351609d59b5
8 changes: 6 additions & 2 deletions src/transformers/models/umt5/__init__.py
Original file line number Diff line number Diff line change
@@ -14,7 +14,7 @@

from typing import TYPE_CHECKING

from ...utils import OptionalDependencyNotAvailable, is_torch_available
from ...utils import OptionalDependencyNotAvailable, _LazyModule, is_torch_available

_import_structure = {}

@@ -40,4 +40,8 @@
UMT5Model,
UMT5PreTrainedModel,
UMT5Stack,
)
)
else:
import sys

sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure, module_spec=__spec__)
25 changes: 12 additions & 13 deletions src/transformers/models/umt5/modeling_umt5.py
Original file line number Diff line number Diff line change
@@ -44,7 +44,6 @@
logging,
replace_return_docstrings,
)
from ...configuration_umt5 import UMT5Config


logger = logging.get_logger(__name__)
@@ -80,7 +79,7 @@ def forward(self, hidden_states):

# Copied from transformers.models.t5.modeling_t5.T5DenseActDense with T5->UMT5
class UMT5DenseActDense(nn.Module):
def __init__(self, config: UMT5Config):
def __init__(self, config):
super().__init__()
self.wi = nn.Linear(config.d_model, config.d_ff, bias=False)
self.wo = nn.Linear(config.d_ff, config.d_model, bias=False)
@@ -103,7 +102,7 @@ def forward(self, hidden_states):

# Copied from transformers.models.t5.modeling_t5.T5DenseGatedActDense with T5->UMT5
class UMT5DenseGatedActDense(nn.Module):
def __init__(self, config: UMT5Config):
def __init__(self, config):
super().__init__()
self.wi_0 = nn.Linear(config.d_model, config.d_ff, bias=False)
self.wi_1 = nn.Linear(config.d_model, config.d_ff, bias=False)
@@ -133,7 +132,7 @@ def forward(self, hidden_states):

# Copied from transformers.models.t5.modeling_t5.T5LayerFF with T5->UMT5
class UMT5LayerFF(nn.Module):
def __init__(self, config: UMT5Config):
def __init__(self, config):
super().__init__()
if config.is_gated_act:
self.DenseReluDense = UMT5DenseGatedActDense(config)
@@ -151,7 +150,7 @@ def forward(self, hidden_states):


class UMT5Attention(nn.Module):
def __init__(self, config: UMT5Config, has_relative_attention_bias=False):
def __init__(self, config, has_relative_attention_bias=False):
super().__init__()
self.is_decoder = config.is_decoder
self.has_relative_attention_bias = has_relative_attention_bias
@@ -518,7 +517,7 @@ class UMT5PreTrainedModel(PreTrainedModel):
models.
"""

config_class = UMT5Config
# config_class = UMT5Config
base_model_prefix = "transformer"
supports_gradient_checkpointing = True
_no_split_modules = ["UMT5Block"]
@@ -982,10 +981,10 @@ class UMT5Model(UMT5PreTrainedModel):
>>> hidden_states = outputs.last_hidden_state
```"""
model_type = "uumt5"
config_class = UMT5Config
# config_class = UMT5Config
_tied_weights_keys = ["encoder.embed_tokens.weight", "decoder.embed_tokens.weight"]

def __init__(self, config: UMT5Config):
def __init__(self, config):
super().__init__(config)
self.shared = nn.Embedding(config.vocab_size, config.d_model)

@@ -1148,10 +1147,10 @@ class UMT5ForConditionalGeneration(UMT5PreTrainedModel):
```"""

model_type = "umt5"
config_class = T5Config
# config_class = T5Config
_tied_weights_keys = ["encoder.embed_tokens.weight", "decoder.embed_tokens.weight", "lm_head.weight"]

def __init__(self, config: UMT5Config):
def __init__(self, config):
super().__init__(config)
self.model_dim = config.d_model

@@ -1435,10 +1434,10 @@ class UMT5EncoderModel(UMT5PreTrainedModel):
```"""

model_type = "umt5"
config_class = UMT5Config
# config_class = UMT5Config
_tied_weights_keys = ["encoder.embed_tokens.weight"]

def __init__(self, config: UMT5Config):
def __init__(self, config):
super().__init__(config)
self.shared = nn.Embedding(config.vocab_size, config.d_model)

@@ -1527,7 +1526,7 @@ class UMT5ForQuestionAnswering(UMT5PreTrainedModel):
_tied_weights_keys = ["encoder.embed_tokens.weight", "decoder.embed_tokens.weight"]

# Copied from transformers.models.t5.modeling_t5.T5ForQuestionAnswering.__init__ with T5->UMT5
def __init__(self, config: UMT5Config):
def __init__(self, config):
super().__init__(config)
self.model_dim = config.d_model