New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

internvideo2中文支持，如何正确调用internvideo2_clip模型 #199

Open

normanShan0917 opened this issue Oct 15, 2024 · 0 comments

normanShan0917 commented Oct 15, 2024

想要在自有中文检索数据集上应用internvideo2，看了下最接近的是vatex_cn中使用internvideo2_clip模型，但是在加载过程中遇到些疑问。

共计需要加载哪些ckpt？
目前看是这几个组件
chinese_alpaca_lora_7b
InternVideo2-stage2_1b-224p-f4.pt
1B_clip.pth
internvl_c_13b_224px.pth
配置文件如下
tokenizer_path="chinese_alpaca_lora_7b",
vision_ckpt_path="OpenGVLab__InternVideo2-Stage2_1B-224p-f4/InternVideo2-stage2_1b-224p-f4.pt",
load_vision_ckpt_from_internvideo2_stage2=True,
text_ckpt_path="internvl_c_13b_224px.pth"
extra_ckpt_path="OpenGVLab__InternVideo2-CLIP-1B-224p-f8/1B_clip.pth"

但是找不到可以配置1B_clip.pth的地方，在demo中直接在config中设置pretrained_path还是在配置文件中设置extra_ckpt_path?

2.加载ckpt的代码不知道是不是有问题，我最终产出的排序是这样的，不太符合预期。目前使用的是在配置文件中设置extra_ckpt_path，也就是1里的配置

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment