-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
the pretrained model weights #4
Comments
No. But maybe it's possible to make it work with the pretrained OPT weights recently released by facebook and the pretrained CLIP weights. |
great idea :) will still need to train the cross attention blocks (perceiver + gated), but should be doable |
It would really be great to somehow allow loading pretrained weights for one (some?) of the available pretrained models. Do you think it's really difficult (and unintuitive) to smh allow pulling weights from arbitrary pretrained models (to an extent, smth like GPTJ, GPTNeo(X), GPT2, etc.)? |
Are we planning to release any pretrained model weights in the future? |
Its actually not a good idea since CLIP is not task agnostic, as explained in CLIP's paper tasks with non natural images like EuroSAT perform poorly. So you would be basically making Flamingo look like a model being one year older. |
Flamingo's encoder backbone is trained in a similar approach to CLIP with contrastive text-image training (ref to Section 3 of the paper). The data from CLIP is also scraped from the web, I think in a very similar way to Flamingo. Therefore, the training process of Flamingo or the data that it uses is not necessarily more task agnostic than CLIP. If you want to have a proper task agnostic backbone, it's probably better to use a backbone trained in a self-supervised approach similar to BYOL. Nevertheless, the type of data is important. For example, you mentioned X-ray in the other thread, which could be quite different than typically scraped data from the web. |
I tried this but i counted on flamingo to help me to do few short learning because my dataset is not a per se real dataset as we can imagine it but more few examples of xray images |
You might be able to use Flamingo for your use case by prompting (I'm not sure if it works but its worth a try). By prompting I mean that u give it one x-ray image, describe what's important in the image, give it a second image and ask your query. I think this leads us out of the topic here, if you want we can discuss this in another thread. |
Any news on the model weights? |
any pretrained model weights released?
The text was updated successfully, but these errors were encountered: