Add XLMRoBERTaModel in paddlenlp #9720

jie-z-0607 · 2024-12-31T03:41:54Z

PR types

New features

PR changes

Models

Description

在PaddleNLP中增加对于XLM-RoBERTa系列模型的支持，已支持相关预训练模型如下：

model name	model type
BAAI/bge-m3	XLMRobertaModel
BAAI/bge-reranker-v2-m3	XLMRobertaForSequenceClassification
BAAI/bge-reranker-large	XLMRobertaForSequenceClassification
BAAI/bge-reranker-base	XLMRobertaForSequenceClassification
BAAI/bge-m3-unsupervised	XLMRobertaModel

JunnYu · 2024-12-31T03:53:12Z

paddlenlp/transformers/xlm_roberta/configuration.py

+    Examples:
+
+    ```python
+    >>> from ppdiffusers.transformers import XLMRobertaConfig, XLMRobertaModel


改一下文档

JunnYu · 2024-12-31T03:54:00Z

paddlenlp/transformers/xlm_roberta/configuration.py

+        classifier_dropout=None,
+        **kwargs,
+    ):
+        kwargs["return_dict"] = kwargs.pop("return_dict", True)


这里我当时是跟transformers逻辑一样，默认值return_dict为True，而paddlenlp基本上所有模型都是False，需要决策一下

改为False吧

JunnYu · 2024-12-31T03:55:17Z

paddlenlp/transformers/xlm_roberta/modeling.py

+            if self.gradient_checkpointing and not hidden_states.stop_gradient:
+                layer_outputs = self._gradient_checkpointing_func(


gradient_checkpointing -> recompute，参照paddlenlp的改一下吧

JunnYu · 2024-12-31T03:55:28Z

paddlenlp/transformers/xlm_roberta/modeling.py

+        all_self_attentions = () if output_attentions else None
+        all_cross_attentions = () if output_attentions and self.config.add_cross_attention else None
+
+        if self.gradient_checkpointing and self.training:


这里也是

JunnYu · 2024-12-31T03:55:41Z

paddlenlp/transformers/xlm_roberta/modeling.py

+        super().__init__()
+        self.config = config
+        self.layer = nn.LayerList([XLMRobertaLayer(config) for _ in range(config.num_hidden_layers)])
+        self.gradient_checkpointing = False


这里也改了吧

改成self.enable_recompute=False

JunnYu · 2024-12-31T03:56:20Z

paddlenlp/transformers/xlm_roberta/modeling.py

+    _deprecated_dict = {
+        "key": ".self_attn.q_proj.",
+        "name_mapping": {
+            # common
+            "encoder.layers.": "encoder.layer.",
+            # embeddings
+            "embeddings.layer_norm.": "embeddings.LayerNorm.",
+            # transformer
+            ".self_attn.q_proj.": ".attention.self.query.",
+            ".self_attn.k_proj.": ".attention.self.key.",
+            ".self_attn.v_proj.": ".attention.self.value.",
+            ".self_attn.out_proj.": ".attention.output.dense.",
+            ".norm1.": ".attention.output.LayerNorm.",
+            ".linear1.": ".intermediate.dense.",
+            ".linear2.": ".output.dense.",
+            ".norm2.": ".output.LayerNorm.",
+        },
+    }


这里删了，没有用

JunnYu · 2024-12-31T03:58:42Z

paddlenlp/transformers/xlm_roberta/tokenizer.py

+
+from paddlenlp.transformers.tokenizer_utils import AddedToken
+from paddlenlp.transformers.tokenizer_utils import (
+    PretrainedTokenizer as PPNLPPretrainedTokenizer,


这里不用as直接PretrainedTokenizer

改为相对路径

JunnYu · 2024-12-31T03:58:47Z

paddlenlp/transformers/xlm_roberta/tokenizer.py

+__all__ = ["XLMRobertaTokenizer"]
+
+
+class XLMRobertaTokenizer(PPNLPPretrainedTokenizer):


这里也修改

JunnYu · 2024-12-31T04:00:06Z

auto部分也要加

JunnYu · 2024-12-31T04:01:56Z

paddlenlp/transformers/model_utils.py

+class ModuleUtilsMixin:
+    """
+    A few utilities for `nn.Layer`, to be used as a mixin.
+    """
+
+    # @property
+    # def device(self):
+    #     """
+    #     `paddle.place`: The device on which the module is (assuming that all the module parameters are on the same
+    #     device).
+    #     """
+    #     try:
+    #         return next(self.named_parameters())[1].place
+    #     except StopIteration:
+    #         try:
+    #             return next(self.named_buffers())[1].place
+    #         except StopIteration:
+    #             return paddle.get_device()


这部分的代码加入可能会影响已有的很多模型，得仔细看一下

DrownFish19 · 2024-12-31T04:02:15Z

paddlenlp/transformers/xlm_roberta/configuration.py

@@ -0,0 +1,133 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.


这里少一个paddle的copyright

DrownFish19 · 2024-12-31T04:02:52Z

paddlenlp/transformers/xlm_roberta/configuration.py

+        classifier_dropout=None,
+        **kwargs,
+    ):
+        kwargs["return_dict"] = kwargs.pop("return_dict", True)


改为False吧

DrownFish19 · 2024-12-31T04:03:12Z

paddlenlp/transformers/xlm_roberta/modeling.py

@@ -0,0 +1,1517 @@
+# coding=utf-8


增加paddle的copyright

DrownFish19 · 2024-12-31T04:04:06Z

paddlenlp/transformers/xlm_roberta/modeling.py

+from paddle import nn
+from paddle.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
+
+from paddlenlp.transformers.activations import ACT2FN


from paddlenlp 这些都改成相对路径吧

DrownFish19 · 2024-12-31T04:04:49Z

paddlenlp/transformers/xlm_roberta/modeling.py

+        super().__init__()
+        self.config = config
+        self.layer = nn.LayerList([XLMRobertaLayer(config) for _ in range(config.num_hidden_layers)])
+        self.gradient_checkpointing = False


改成self.enable_recompute=False

DrownFish19 · 2024-12-31T04:06:10Z

paddlenlp/transformers/xlm_roberta/modeling.py

+        Example:
+
+        ```python
+        >>> from ppdiffusers.transformers import AutoTokenizer, XLMRobertaForCausalLM, AutoConfig


同上修改文档

DrownFish19 · 2024-12-31T04:07:34Z

paddlenlp/transformers/xlm_roberta/tokenizer.py

+
+from paddlenlp.transformers.tokenizer_utils import AddedToken
+from paddlenlp.transformers.tokenizer_utils import (
+    PretrainedTokenizer as PPNLPPretrainedTokenizer,


改为相对路径

DrownFish19 · 2024-12-31T04:09:53Z

在PaddleNLP/paddlenlp/transformers/auto文件里增加对应的模型、tokenizer映射

codecov · 2024-12-31T04:16:18Z

Codecov Report

Attention: Patch coverage is 79.34641% with 158 lines in your changes missing coverage. Please review.

Project coverage is 52.39%. Comparing base (dff62a1) to head (473e6e0).
Report is 20 commits behind head on develop.

Files with missing lines	Patch %	Lines
paddlenlp/transformers/xlm_roberta/modeling.py	78.89%	137 Missing ⚠️
paddlenlp/transformers/xlm_roberta/tokenizer.py	75.86%	21 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #9720      +/-   ##
===========================================
- Coverage    53.20%   52.39%   -0.81%     
===========================================
  Files          719      727       +8     
  Lines       115583   115095     -488     
===========================================
- Hits         61493    60304    -1189     
- Misses       54090    54791     +701

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ZHUI · 2025-01-02T09:36:04Z

加两个单测，测试一下，模型初始化，tokenier 加载。

JunnYu · 2025-01-02T10:42:27Z

新增对应的单测脚本

ZHUI · 2025-01-07T03:46:00Z

paddlenlp/transformers/xlm_roberta/modeling.py

+    # See all XLM-RoBERTa models at https://huggingface.co/models?filter=xlm-roberta
+]
+
+


缺少 __all__ = [""] 说明一下可以import哪些模型名称

add xlm_roberta in paddlenlp

521a424

JunnYu reviewed Dec 31, 2024

View reviewed changes

DrownFish19 reviewed Dec 31, 2024

View reviewed changes

jie-z-0607 added 3 commits December 31, 2024 16:08

fix1

46ab26e

fix_2

964ac27

fix_configuration

e4c1f12

ZHUI reviewed Jan 7, 2025

View reviewed changes

add_test

473e6e0

jie-z-0607 changed the title ~~add XLM-RoBERTa in paddlenlp~~ Add XLMRoBERTaModel in paddlenlp Jan 8, 2025

ZHUI merged commit 1d74d62 into PaddlePaddle:develop Jan 8, 2025
9 of 12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add XLMRoBERTaModel in paddlenlp #9720

Add XLMRoBERTaModel in paddlenlp #9720

jie-z-0607 commented Dec 31, 2024 •

edited

Loading

JunnYu Dec 31, 2024

JunnYu Dec 31, 2024

DrownFish19 Dec 31, 2024

JunnYu Dec 31, 2024

JunnYu Dec 31, 2024

JunnYu Dec 31, 2024

DrownFish19 Dec 31, 2024 •

edited

Loading

JunnYu Dec 31, 2024

JunnYu Dec 31, 2024

DrownFish19 Dec 31, 2024

JunnYu Dec 31, 2024

JunnYu commented Dec 31, 2024 •

edited

Loading

JunnYu Dec 31, 2024 •

edited

Loading

DrownFish19 Dec 31, 2024

DrownFish19 Dec 31, 2024

DrownFish19 Dec 31, 2024

DrownFish19 Dec 31, 2024

DrownFish19 Dec 31, 2024 •

edited

Loading

DrownFish19 Dec 31, 2024

DrownFish19 Dec 31, 2024

DrownFish19 commented Dec 31, 2024

codecov bot commented Dec 31, 2024 •

edited

Loading

ZHUI commented Jan 2, 2025

JunnYu commented Jan 2, 2025

ZHUI Jan 7, 2025

		if self.gradient_checkpointing and not hidden_states.stop_gradient:
		layer_outputs = self._gradient_checkpointing_func(

		__all__ = ["XLMRobertaTokenizer"]


		class XLMRobertaTokenizer(PPNLPPretrainedTokenizer):

		@@ -0,0 +1,133 @@
		# coding=utf-8
		# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.

		# See all XLM-RoBERTa models at https://huggingface.co/models?filter=xlm-roberta
		]

Add XLMRoBERTaModel in paddlenlp #9720

Add XLMRoBERTaModel in paddlenlp #9720

Conversation

jie-z-0607 commented Dec 31, 2024 • edited Loading

PR types

PR changes

Description

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DrownFish19 Dec 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JunnYu commented Dec 31, 2024 • edited Loading

JunnYu Dec 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DrownFish19 Dec 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DrownFish19 commented Dec 31, 2024

codecov bot commented Dec 31, 2024 • edited Loading

Codecov Report

ZHUI commented Jan 2, 2025

JunnYu commented Jan 2, 2025

Choose a reason for hiding this comment

jie-z-0607 commented Dec 31, 2024 •

edited

Loading

DrownFish19 Dec 31, 2024 •

edited

Loading

JunnYu commented Dec 31, 2024 •

edited

Loading

JunnYu Dec 31, 2024 •

edited

Loading

DrownFish19 Dec 31, 2024 •

edited

Loading

codecov bot commented Dec 31, 2024 •

edited

Loading