Why do we need to extract the language embeddings first?

[redagavin]

Hi, I don't know why we need to extract the language embeddings first. Doesn't it work like we give box prompts and images and then the model outputs segmentation masks and labels? Why we need language embeddings? How does it work with the model?

[HarborYuan]

Hi @redagavin ,

The language embeddings serve as a "dictionary." During the inference, a visual embedding will be extracted to match the "dictionary." You can refer to the code and paper for details. Please let me know if you have any other questions.