Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify Offline Evals #17

Merged
merged 3 commits into from
Dec 9, 2024
Merged

Unify Offline Evals #17

merged 3 commits into from
Dec 9, 2024

Conversation

zentavious
Copy link
Owner

This PR introduces the iter_hyperparameters function to streamline hyperparameter tuning by generating and managing combinations of topic_theta and locality_theta values. It enhances the recommendation pipeline by supporting parallelized processing of profiles paired with these theta combinations, with an option for random sampling to reduce runtime. Additionally, it integrates key metrics for analysis and debugging.

New Functionality:

  1. iter_hyperparameters: A generator function that yields cross-product combinations of topic_theta and locality_theta values paired with profiles.
  2. Random sampling capability to reduce computational overhead (suggested by @mdekstrand).
  3. Parallelization-ready structure for downstream components.

Pipeline Integration:

  1. Added theta parameters to the rerank component.
  2. Wired these parameters through the recommendation pipeline.

Metrics Logging:

  1. KL Divergence: Measures the divergence between distributions.
  2. Treatment Ratio: Tracks the ratio of profiles treated with specific configurations.
  3. Within Locality Threshold Flag: Indicates compliance with locality-based constraints.

@zentavious zentavious changed the title Zent/uni evals Unify Offline Evals Dec 6, 2024
@zentavious zentavious merged commit 5a7c810 into main Dec 9, 2024
2 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant