SAM-HQ implementation in transformers

[IskanderNakipov]

Model description

HQ-SAM is an upgrade SAM for high-quality zero-shot segmentation. It's design mostly repeats the design of SAM, but additionally early VisionEncoder features are used in enhanced MaskDecoder for fine-grained segmentation

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

Code is available here https://github.com/SysCV/sam-hq
Paper https://arxiv.org/abs/2306.01567

Authors @lkeab et al.

[LysandreJik]

cc @amyeroberts @qubvel @molbap

[LysandreJik]

Linking PR #31150

[sushmanthreddy]

@qubvel @amyeroberts Is anyone working on this PR ?
I would like to contribute on this issue the above pr has been closed actually .
if its not done will start working on pr

[molbap]

Hi @sushmanthreddy , AFAIK there is no one currently working on this PR, @IskanderNakipov has done some work on this model on above PR that you can use. Feel free to start and ping me for a first review :)

Since SamHQ is based off Sam, we should use modular transformers for the addition. modular allows using inheritance in transformers, I recommend checking https://huggingface.co/docs/transformers/en/modular_transformers as well as https://github.com/huggingface/transformers/blob/main/src/transformers/models/gemma2/modular_gemma2.py, https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm/modular_glm.py . That will allow you to reuse components of Sam implicitly, and write explicitly only what's new.

[sushmanthreddy]

@molbap thanks for the directions ,will start working on it