Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PaddleMIX ppdiffusers中升级attention相关代码 #262

Closed
chenjjcccc opened this issue Oct 31, 2023 · 10 comments
Closed

PaddleMIX ppdiffusers中升级attention相关代码 #262

chenjjcccc opened this issue Oct 31, 2023 · 10 comments
Labels
HappyOpenSource Pro 快乐开源issue与PR，更具挑战的任务

Comments

@chenjjcccc
Copy link

chenjjcccc commented Oct 31, 2023 •
edited by guoshengCS
Loading

PaddleMIX ppdiffusers中升级attention相关代码

任务描述

任务背景

  • models模块代码升级。
  • 升级脚本：
    • models/attention.py
    • models/attention_processor.py

完成步骤

  1. 对比ppdiffusers (v0.19.3)和diffusers (v0.21.1)的代码差异，可使用页面diff工具。
  2. fork PaddleMix仓库并对题目中提到的代码文件进行更新
  3. 学习掌握pytest测试工具的使用，并确保升级后的代码文件能够通过tests/models目录下的单元测试用例（如有单测代码更新需要先进行对应单测升级再测试），确保升级后的代码正确。
  4. 提交PR到主仓库的develop分支，并描述任务具体完成情况（附上测试信息、推理用例）

提交内容：

  1. 升级后的代码文件。
  2. 测试情况。

题目更新：

  1. 对比ppdiffusers (v0.19.3)和diffusers (v0.21.1)的代码差异，可使用页面diff工具

当前还请按照diffusers最新稳定版本0.23.1更新升级
@shiyutang shiyutang added the HappyOpenSource Pro 快乐开源issue与PR，更具挑战的任务 label Oct 31, 2023
@shiyutang shiyutang mentioned this issue Oct 31, 2023
Closed
@Ligoml Ligoml mentioned this issue Nov 9, 2023
Closed
@unicornshell
Copy link

unicornshell commented Nov 21, 2023 •
edited
Loading

ppdiffusers里attention.py里GEGLU模块缺少
def gelu(self, gate): if gate.device.type != "mps": return F.gelu(gate)，但这并不在代码差异中，需要更改吗

@unicornshell
Copy link

似乎缺少models/attention.py的单测

@LokeZhou
Copy link
Collaborator

ppdiffusers里attention.py里GEGLU模块缺少 def gelu(self, gate): if gate.device.type != "mps": return F.gelu(gate)，但这并不在代码差异中，需要更改吗

不需要

@LokeZhou
Copy link
Collaborator

似乎缺少models/attention.py的单测

如果torch 的diffusers里有这个单测则添加上去，如果没有就不需要添加

@unicornshell
Copy link

unicornshell commented Nov 22, 2023 •
edited
Loading

diffusers的attention_processor.py里存在的pytorch2.0版本的类需要在ppdiffusers里实现吗 像：
AttnAddedKVProcessor2_0，AttnProcessor2_0，LoRAAttnProcessor2_0

@unicornshell
Copy link

unicornshell commented Nov 22, 2023 •
edited
Loading

缺少2_0的类会导致新增的

ADDED_KV_ATTENTION_PROCESSORS = (
    AttnAddedKVProcessor,
    SlicedAttnAddedKVProcessor,
    AttnAddedKVProcessor2_0,
    XFormersAttnAddedKVProcessor,
    LoRAAttnAddedKVProcessor,
)

CROSS_ATTENTION_PROCESSORS = (
    AttnProcessor,
    AttnProcessor2_0,
    XFormersAttnProcessor,
    SlicedAttnProcessor,
    LoRAAttnProcessor,
    LoRAAttnProcessor2_0,
    LoRAXFormersAttnProcessor,
)

AttentionProcessor = Union[
    AttnProcessor,
    AttnProcessor2_0,
    XFormersAttnProcessor,
    SlicedAttnProcessor,
    AttnAddedKVProcessor,
    SlicedAttnAddedKVProcessor,
    AttnAddedKVProcessor2_0,
    XFormersAttnAddedKVProcessor,
    CustomDiffusionAttnProcessor,
    CustomDiffusionXFormersAttnProcessor,
    # depraceted
    LoRAAttnProcessor,
    LoRAAttnProcessor2_0,
    LoRAXFormersAttnProcessor,
    LoRAAttnAddedKVProcessor,
]

部分出现类似于"AttnProcessor2_0" is not defined的warning

@co63oc co63oc mentioned this issue Nov 24, 2023
Merged
@shiyutang
Copy link
Collaborator

@LokeZhou 请问这个PR #322 是否很快就能合入。

@LokeZhou
Copy link
Collaborator

@LokeZhou 请问这个PR #322 是否很快就能合入。

ci通过后合入

@LokeZhou
Copy link
Collaborator

diffusers的attention_processor.py里存在的pytorch2.0版本的类需要在ppdiffusers里实现吗 像： AttnAddedKVProcessor2_0，AttnProcessor2_0，LoRAAttnProcessor2_0

如果功能一致不需要；如果是因为2.0版本导致新增功能，则需要。

JunnYu pushed a commit that referenced this issue Dec 14, 2023
@co63oc
【Hackathon 5th No.83】PaddleMIX ppdiffusers models模块功能升级同步HF (#322)
c4eb91e 
PaddleMIX ppdiffusers models模块功能升级同步HF


https://github.com/PaddlePaddle/community/blob/master/hackathon/hackathon_5th/%E3%80%90PaddlePaddle%20Hackathon%205th%E3%80%91%E5%BC%80%E6%BA%90%E8%B4%A1%E7%8C%AE%E4%B8%AA%E4%BA%BA%E6%8C%91%E6%88%98%E8%B5%9B%E5%A5%97%E4%BB%B6%E5%BC%80%E5%8F%91%E4%BB%BB%E5%8A%A1%E5%90%88%E9%9B%86.md#no83paddlemix-ppdiffusers-models%E6%A8%A1%E5%9D%97%E5%8A%9F%E8%83%BD%E5%8D%87%E7%BA%A7%E5%90%8C%E6%AD%A5hf

#259
#260
#262
#264
#265

升级 models
增加scale
增加get_processor
diffusers上LoRA类逐渐取消，改为使用lora_layer
部分测试误差增加，测试修改前代码同样有误差
增加tests/lora目录
flax模型paddle没有使用，没有迁移

test_attention_processor.py

![image](https://github.com/PaddlePaddle/PaddleMIX/assets/4617245/3a7cdfcf-6313-42f7-8bb3-9fd7d46b36cf)

test_models_unet_2d.py

![image](https://github.com/PaddlePaddle/PaddleMIX/assets/4617245/51f9ec37-e0f2-4010-b621-966d596bf451)

test_lora_layers.py

![image](https://github.com/PaddlePaddle/PaddleMIX/assets/4617245/d4761fde-1eda-4329-8f94-79d4ed6535b3)

test_models_unet_motion.py

![image](https://github.com/PaddlePaddle/PaddleMIX/assets/4617245/f3ff8f8c-89f2-46e2-a733-4b6c442e36b5)
@shiyutang
Copy link
Collaborator

任务已完成 by @co63oc ，close issue。

@isLinXu isLinXu mentioned this issue Dec 30, 2023
Closed
26 tasks
@Harryoung Harryoung mentioned this issue Feb 2, 2024
Closed
westfish pushed a commit to westfish/PaddleMIX that referenced this issue Sep 25, 2024
@co63oc
【Hackathon 5th No.83】PaddleMIX ppdiffusers models模块功能升级同步HF (PaddlePa…
85ac514 
…ddle#322)

PaddleMIX ppdiffusers models模块功能升级同步HF


https://github.com/PaddlePaddle/community/blob/master/hackathon/hackathon_5th/%E3%80%90PaddlePaddle%20Hackathon%205th%E3%80%91%E5%BC%80%E6%BA%90%E8%B4%A1%E7%8C%AE%E4%B8%AA%E4%BA%BA%E6%8C%91%E6%88%98%E8%B5%9B%E5%A5%97%E4%BB%B6%E5%BC%80%E5%8F%91%E4%BB%BB%E5%8A%A1%E5%90%88%E9%9B%86.md#no83paddlemix-ppdiffusers-models%E6%A8%A1%E5%9D%97%E5%8A%9F%E8%83%BD%E5%8D%87%E7%BA%A7%E5%90%8C%E6%AD%A5hf

PaddlePaddle#259
PaddlePaddle#260
PaddlePaddle#262
PaddlePaddle#264
PaddlePaddle#265

升级 models
增加scale
增加get_processor
diffusers上LoRA类逐渐取消，改为使用lora_layer
部分测试误差增加，测试修改前代码同样有误差
增加tests/lora目录
flax模型paddle没有使用，没有迁移

test_attention_processor.py

![image](https://github.com/PaddlePaddle/PaddleMIX/assets/4617245/3a7cdfcf-6313-42f7-8bb3-9fd7d46b36cf)

test_models_unet_2d.py

![image](https://github.com/PaddlePaddle/PaddleMIX/assets/4617245/51f9ec37-e0f2-4010-b621-966d596bf451)

test_lora_layers.py

![image](https://github.com/PaddlePaddle/PaddleMIX/assets/4617245/d4761fde-1eda-4329-8f94-79d4ed6535b3)

test_models_unet_motion.py

![image](https://github.com/PaddlePaddle/PaddleMIX/assets/4617245/f3ff8f8c-89f2-46e2-a733-4b6c442e36b5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
HappyOpenSource Pro 快乐开源issue与PR，更具挑战的任务
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@LokeZhou @shiyutang @unicornshell @chenjjcccc