CLIP result seems wrong

[awsaf49]

Current Behavior:

I tried out recently added pre-trained CLIP for zero-shot classification and the result seems wrong. You can verify that in this kaggle notebook. Instead of classifying the image as cat it is giving all candid labels equal probability of 0.33. I tried with multiple samples and the result is same for all, {'man': 0.33333847, 'dog': 0.33333862, 'cat': 0.33332297}.

Steps To Reproduce:

notebook: https://www.kaggle.com/code/awsaf49/kerascv-clip

Version:

• HEAD

Reference:

• Add CLIP to KerasCV #2331

[awsaf49]

cc: @divyashreepathihalli

[awsaf49]

I tried this code for zero-shot classification

processed_image = processor.process_images(["cat1.jpg"])
candid_labels = ["man", "dog", "cat"]
processed_text, attention_mask = processor.process_texts(
  [f"Picture of a {x}." for x in candid_labels]
  )
image_logits, text_logits = model(
        {
            "image": processed_image,
            "text": processed_text,
            "attention_mask": attention_mask,
        }
    )
preds = keras.ops.softmax(image_logits).numpy().squeeze()
result = dict(zip(candid_labels, preds))
print(result)

[awsaf49]

In the same kaggle notebook, I've compared results with open-ai CLIP . And open-AI's CLIP works nicely in contrast to KerasCV's CLIP.

[divyashreepathihalli]

That doesn't seem right. I'll take a look.

[divyashreepathihalli]

Looks like the model is working fine with the weights file but from_preset() doesn't seem to be working.