update base code #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Initial Topic Models
…guide Topic model rating filter guide update
Topic seed token prefix adjustments and model updates
Guide updates for new thresholds at which writing ability is locked and unlocked.
This commit splits the monolithic scoring binary into two separate scoring binaries (that may still be run sequentially): 1. Prescoring: do expensive pre-computation to learn user and note parameters 2. [Final] Scoring: ingest prescoring outputs in order to save computation time, then run scoring like it is today. In this commit, the final result of scoring is the same. In the future though, this unlocks much work to simplify the final scorer.
…scorer Split scoring binary into separate prescoring and final scoring binaries
…udoraters Final scoring and prescoring each run about ~10mins faster now when run in parallel on one large CPU machine, due to sharing large dataframes in memory across multiple processes instead of re-reading them. Also, the core scorer itself now runs about ~10mins faster due to cleanup of unused pseudorater computations (uncertainty estimation)
…_memory_optimization Optimize scorer: used shared memory across processes, and streamline uncertainty estimation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.