Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix metadata_update for verified evaluations #1214

Merged
merged 11 commits into from
Nov 29, 2022
Merged

Fix metadata_update for verified evaluations #1214

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
c951108
Fix metadata_update for verified evaluations
lewtun Nov 21, 2022
23467e6
Add dataset metadata to model index conversion
lewtun Nov 21, 2022
49b5d36
Add regression test for #1208
lewtun Nov 27, 2022
cfaee4e
Add regression test for #1214
lewtun Nov 27, 2022
8e4eae2
Use unique ID
lewtun Nov 27, 2022
2c00a83
Fix typing
lewtun Nov 27, 2022
8cec496
Merge branch 'main' into fix-metadata-update
lewtun Nov 28, 2022
da6ae02
Fix hashing
lewtun Nov 28, 2022
b2bb6d6
Refactor
lewtun Nov 28, 2022
ad02d45
Fix args
lewtun Nov 29, 2022
7ec1406
Merge branch 'main' into fix-metadata-update
lewtun Nov 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/huggingface_hub/repocard.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -807,6 +807,8 @@ def metadata_update(
)
result_found = True
existing_result.metric_value = new_result.metric_value
if existing_result.verified is True:
existing_result.verify_token = new_result.verify_token
if not result_found:
card.data.eval_results.append(new_result)
else:
Expand Down
22 changes: 17 additions & 5 deletions src/huggingface_hub/repocard_data.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,9 @@ def is_equal_except_value(self, other: "EvalResult") -> bool:
for key, _ in self.__dict__.items():
if key == "metric_value":
continue
if getattr(self, key) != getattr(other, key):
# For metrics computed by Hugging Face's evaluation service, `verify_token` is derived from `metric_value`,
# so we exclude it here in the comparison.
if key != "verify_token" and getattr(self, key) != getattr(other, key):
return False
return True

Expand Down Expand Up @@ -514,12 +516,22 @@ def eval_results_to_model_index(
# Here, we make a map of those pairs and the associated EvalResults.
task_and_ds_types_map = defaultdict(list)
for eval_result in eval_results:
task_and_ds_pair = (eval_result.task_type, eval_result.dataset_type)
task_and_ds_pair = (
eval_result.task_type,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unique combination for evaluation results is (task, dataset, config, split), so we extend the logic here to avoid erasing this information from the evaluation results

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice solving this bug @lewtun ! Do you see any way we could avoid a break in the future if another field is added to EvalResult ? Or is it most likely not gonna happen ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just had a look to the code, I guess it would make sense to add dataset_revision and dataset_args as well right ?

Maybe we could have a EvalResult.unique_identifier property that would be a tuple/hash that depends on task and dataset properties: task_type, dataset_type, dataset_config,... Because in the end, an EvalResult is "just" a config (task+dataset) and a value associated to it right (+a attributes for verification) ? I don't know how we could name that but it would also be very convenient also for the is_equal_except_value method.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @nateraw who worked on that

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea! Added in 8e4eae2

eval_result.dataset_type,
eval_result.dataset_config,
eval_result.dataset_split,
)
task_and_ds_types_map[task_and_ds_pair].append(eval_result)

# Use the map from above to generate the model index data.
model_index_data = []
for (task_type, dataset_type), results in task_and_ds_types_map.items():
for (
task_type,
dataset_type,
dataset_config,
dataset_split,
), results in task_and_ds_types_map.items():
data = {
"task": {
"type": task_type,
Expand All @@ -528,8 +540,8 @@ def eval_results_to_model_index(
"dataset": {
"name": results[0].dataset_name,
"type": dataset_type,
"config": results[0].dataset_config,
"split": results[0].dataset_split,
"config": dataset_config,
"split": dataset_split,
"revision": results[0].dataset_revision,
"args": results[0].dataset_args,
},
Expand Down