[API] Add octoai back-end

[vvchernov]

Need OCTOAI_API_KEY to use it
Add octoai (https://octoai.cloud/) as back-end for models.
Greedy_until path is enabled (while likelihood is WIP)

[CLAassistant]

All committers have signed the CLA.

[vvchernov]

Hello @haileyschoelkopf, @lintangsutawika! Could you review the PR?

[StellaAthena]

Thank you for this PR! I notice that you say "no support for logits"... Logits are a crucial resource for machine learning researchers and lacking logits will heavily restrict the tasks that can be run.

Are there plans to add support for this in the future? If not, what is the overriding value that OctoAI adds? For example, are there popular models that are only available through that platform?

[vvchernov]

Thank you for this PR! I notice that you say "no support for logits"... Logits are a crucial resource for machine learning researchers and lacking logits will heavily restrict the tasks that can be run.

Are there plans to add support for this in the future?

Hello @StellaAthena! I think you meant loglikelyhood approach which works with logits inside. Yes, I work to support it just now and it will be done in the closest future

[binarybana]

Hi @StellaAthena,

As @vvchernov mentioned, we'll be supporting logprobs (and enabling loglikelihood evals) shortly and can hold this PR until that lands if you'd like.

One benefit of adding OpenAI endpoint support is because our LLM's are exposing an OpenAI compatible API, then these integrations will be useful for people beyond just OctoAI's endpoints. Also, for people using fine-tuned endpoints (we have a whole roadmap for this), it's helpful to be able to run these evals against their finetuned models without needing to host them locally.

Also, we've got some other improvements/fixes for some lm-evaluation-harness workflows incoming as well separate from these integrations.

[StellaAthena]

OpenAI compatible API

Okay this is really helpful. We already support multiple websites with an OpenAI-compatible API. Right now these are all bundled together for convenience of maintenance and controlled via aliases. It looks like we can do something similar here to make support easier to maintain.

That said OpenAI is rapidly depreciating its support for functionality other than chat-completions. It's been on my radar that we might want to separate the "official" implementation (which necessarily will need to change to track their actual API) and support for a "legacy" version that represents how the API worked before they depreciated things like loglikelihood.

I assume that you're an OctoAI employee? Do you know if the current plan is to maintain consistency with the OpenAI API regardless of how it changes or if you're planning on preserving the original verison that you currently support?

@haileyschoelkopf @lintangsutawika We should ask this of goose.ai as well. It may be that the best path forward is to have a primary class for the official OpenAI API and subclasses for different frameworks that implement and extend it?

[haileyschoelkopf]

We should ask this of goose.ai as well. It may be that the best path forward is to have a primary class for the official OpenAI API and subclasses for different frameworks that implement and extend it?

Yes, definitely agree. vLLM and GGML are other frameworks that could be used via a self-hosted API endpoint with this. I'm not certain about goose.ai but will check

Will take a look at this PR and leave comments, though will wait until logprobs land to merge.

Also of note: we have a new version release in big-refactor that we are planning to have replace the current main very soon (once documentation is tidied up slightly). If you all at OctoAI would be willing to direct your changes to that branch that would be very helpful, though not mandatory (thank you for the other PRs so far!)

[vvchernov]

Also of note: we have a new version release in big-refactor that we are planning to have replace the current main very soon (once documentation is tidied up slightly). If you all at OctoAI would be willing to direct your changes to that branch that would be very helpful, though not mandatory (thank you for the other PRs so far!)

Hello @haileyschoelkopf! Yes, we follow to big-refactor branch. The reason that the master branch is used is Hugging Face leaderboard uses it. Highly likely when big-refactor will be released HF will go to it and we will do it too.

[binarybana]

I assume that you're an OctoAI employee? Do you know if the current plan is to maintain consistency with the OpenAI API regardless of how it changes or if you're planning on preserving the original verison that you currently support?

Yes. We will maintain API compatibility with the core functionality (eg, most LLM providers don't support the plugins pieces) around both /completions and /chat/completions even though /completions is deprecated. Luckily, I've also heard that OpenAI is planning on adding logprobs into /chat/completions at a future date as well.

The idea to support OpenAI API with plugins/mixins to indicate which aspects of the API are supported sounds great to us.

[haileyschoelkopf]

We've now switched to v0.4.0 / main as the new version of lm_eval. We'd be happy to help you port this to the updated version of the codebase--let us know if you have any questions, or, if you'd like us to port it, then if there is an API key we can use for developing the integration.