Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CrossEncoder object has no attribute model_card_data #1192

Closed
isaac-chung opened this issue Sep 3, 2024 · 1 comment
Closed

CrossEncoder object has no attribute model_card_data #1192

isaac-chung opened this issue Sep 3, 2024 · 1 comment
Assignees

Comments

@isaac-chung
Copy link
Collaborator

isaac-chung commented Sep 3, 2024

#1186 (comment)

Getting WARNING:mteb.models:Failed to extract metadata from model: 'CrossEncoder' object has no attribute 'model_card_data'. Upgrading to sentence-transformers v3.0.0 or above is recommended. even though I'm using the below

on the following snippet:

from mteb import MTEB
import mteb
from sentence_transformers import CrossEncoder, SentenceTransformer

cross_encoder = CrossEncoder("cross-encoder/ms-marco-TinyBERT-L-2-v2")
dual_encoder = SentenceTransformer("all-MiniLM-L6-v2")

tasks = mteb.get_tasks(tasks=["NFCorpus"], languages=["eng"])

subset = "default" # subset name used in the NFCorpus dataset
eval_splits = ["test"]

evaluation = MTEB(tasks=tasks)
evaluation.run(
    dual_encoder,
    eval_splits=eval_splits,
    save_predictions=True,
    output_folder="results/stage1",
)
evaluation.run(
    cross_encoder,
    eval_splits=eval_splits,
    top_k=5,
    save_predictions=True,
    output_folder="results/stage2",
    previous_results=f"results/stage1/NFCorpus_{subset}_predictions.json",
)
@x-tabdeveloping
Copy link
Collaborator

@isaac-chung Just for future reference, I ended up talking to Tom about this and he said that model_card_data will probably be implemented for CrossEncoders in a couple of months.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants