Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interleaving pos/neg prompts to match what the repe code expects #59

Merged
merged 1 commit into from
Jan 11, 2024

Conversation

chanind
Copy link
Collaborator

@chanind chanind commented Jan 11, 2024

It appears that the original Repe reading vector code expects every positive example to be paired with its corresponding negative example next to it in a big list passed to the find_directions() method. This isn't documented anywhere, but means we can't simply pass prompts to the find_directions() method unimpeded. Also, the Repe PCA code doesn't properly reflect each difference vector across the origin, so it's important to have a balance of <neg, pos> and <pos, neg> pairs passed to the find_directions() method.

This PR addresses these issues by adding a multi_answer_method param to the RepeReadingControl algorithm. By default, this will select just the first incorrect answer to pair with the correct answer, but can also be set to random_incorrect to pick at random, or repeat_correct to instead duplicate the correct answer to pair with every incorrect answer. This PR also alternates the order of the <neg, pos> for each example to try to ensure a good balance of directions for PCA.

@chanind chanind requested a review from dtch1997 January 11, 2024 20:09
@chanind chanind force-pushed the interleaved-pos-neg-repe branch from 55a8432 to b77ef3e Compare January 11, 2024 20:55
@chanind
Copy link
Collaborator Author

chanind commented Jan 11, 2024

Merging this now for the sake of speed to test it on colab. @dtch1997 Please feel free to code-review this still, any changes can be addressed in a follow-up PR

@chanind chanind merged commit 8eb3b75 into main Jan 11, 2024
2 checks passed
@chanind chanind deleted the interleaved-pos-neg-repe branch January 11, 2024 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant