Skip to content
This repository has been archived by the owner on Oct 2, 2024. It is now read-only.

Officially add new annotators #550

Merged
merged 11 commits into from
Sep 5, 2024
Merged

Officially add new annotators #550

merged 11 commits into from
Sep 5, 2024

Conversation

bkorycki
Copy link
Contributor

@bkorycki bkorycki commented Sep 5, 2024

This PR updates the actual v1.0 test to use new annotators.

  • The test specifies the set of individual annotators instead of interfacing with a single ensemble annotator object. This enables finer-grained caching at the individual annotator level. The ensemble class also ran into caching issues due to polymorphic serialization.
  • The test uses ensemble’s aggregation logic during the measurement phase. This requires a tiny change to the other repo.
  • Does NOT require any environment variables to be set. yay

Problem: A core test is relying on a private repo. If you don’t have that secret repo installed, everything breaks.

Copy link

github-actions bot commented Sep 5, 2024

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

Copy link
Contributor

@wpietri wpietri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great progress for now, and I'm happy to have it merged. For the future, I think we can extract an object there. Having a parallel series of ifs can be a great time to introduce a superclass and then two subclasses, each taking a side of the if.

@bkorycki bkorycki merged commit 280bb1e into main Sep 5, 2024
4 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Sep 5, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants