ViT + CLIP

[gboduljak]

Would it be worth implementing ViT and CLIP example?

[awni]

Yea that ones on our list of examples to add! Are you interested in contributing it? If so which model would you use?

[gboduljak]

Yea that ones on our list of examples to add! Are you interested in contributing it? If so which model would you use?

I would like to contribute :) However, I would like to complete the implementation of norm first (ml-explore/mlx#187). I would use models from the official CLIP repository: https://github.com/openai/CLIP. If you have an alternative idea, please let me know.

[nkasmanoff]

@gboduljak I submitted a PR to your existing PR, which creates a local implementation of the CLIPImageProcessor. gboduljak#1

This should eliminate the dependency on transformers, aside from using it for downloading the model & tokenizer.

[gboduljak]

@nkasmanoff Thanks for the help. I will take a look at your work now.

[gboduljak]

@nkasmanoff I merged your PR, corrected the nits and I refactored your implementation so that everything is in preprocessing folder. Many thanks for the help. In future, we might drop this 'copy-paste' implementation from HuggingFace. Ideally, we should use mlx-data. If you have time, it would be awesome to have mlx-data implementation of CLIPImageProcessor.