Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improves completion api with new input parameters for LLMs #23

Merged
merged 4 commits into from
Nov 5, 2024

Conversation

LeonRuggiero
Copy link
Contributor

@LeonRuggiero LeonRuggiero commented Oct 31, 2024

Improves completion api with new options for LLMs

https://github.com/becomposable/studio/issues/17

Supports:

  • max_tokens
  • stop_sequences
  • top_k
  • top_p
  • presence_penalty
  • frequency_penalty

as supported by the models/api/provider.

Also updates the test file to reflect this.

Noteable TODOs:

  • OpenAI o1 models use the max_completion_tokens rather than max_tokens
  • top_logprobs option currently ignored (commented out on supporting models) as it's output is not supported.

Instruction to validate:

Use change in test file to set new options, try extreme values that are expected to be invalid, this confirms that the model/provider is reading the option correctly. Or large values that produce identifiable behaviour, i.e. lots of repetitions with -2.0 frequency_penalty.

@LeonRuggiero LeonRuggiero added the enhancement New feature or request label Oct 31, 2024
@LeonRuggiero LeonRuggiero requested a review from ebarroca October 31, 2024 10:01
@LeonRuggiero LeonRuggiero self-assigned this Oct 31, 2024
@ebarroca
Copy link
Contributor

ebarroca commented Nov 4, 2024

Cohere fails with https://github.com/llumiverse/llumiverse/actions/runs/11609834444/job/32457539201
Error: ValidationException: Malformed input request: #/p: 1 is not less or equal to 0.99, please reformat your input and try again.

… to be more conservative to prevent long repetition and have better compatibility for testing.
@LeonRuggiero
Copy link
Contributor Author

Cohere fails with https://github.com/llumiverse/llumiverse/actions/runs/11609834444/job/32457539201 Error: ValidationException: Malformed input request: #/p: 1 is not less or equal to 0.99, please reformat your input and try again.

Cohere top_p failure, fixed by setting to a lower value for the tests.

@bstefanescu bstefanescu merged commit a585389 into main Nov 5, 2024
6 of 8 checks passed
@bstefanescu bstefanescu deleted the completion_options branch November 5, 2024 15:51
@LeonRuggiero LeonRuggiero linked an issue Nov 6, 2024 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add parameters to completion API
3 participants