Use Canonical Models to set context window sizes

[katzdave]

Summary

Replace context_limits across all providers with data from canonical model if available.

• API change of adding provider to model config hits a lot of files.
• Also nesting the fast model config, so we actually use the correct context limits for that; There's already a fallback in complete_fast to use the regular model if it fails.
• Replace pricing api with a more general canonical model fetch. Will layer more info there as the client needs.

Going to do one more push down this path after and try to nuke this KNOWN_MODELS concept completely from both the server and client.

[Copilot]

Pull request overview

Copilot reviewed 51 out of 52 changed files in this pull request and generated 4 comments.

[Copilot]

Switching Anthropic to model_config.max_output_tokens() changes the fallback from the previous Claude-specific defaults (often 64k/32k) to the global 4096 when canonical output limits are missing, which can severely truncate responses for models not in the canonical registry; consider preserving the prior Anthropic heuristic as a fallback when max_tokens is unset and canonical output is unavailable.

[Copilot]

The previous config route tests were removed and replaced with an empty mod tests {} even though this PR adds a new /config/canonical-model-info endpoint; add coverage for at least the 200 path (known model) and 404 path (unknown model) to prevent regressions.

Suggested change

	mod tests {}
	mod tests {
	use super::*;
	use std::sync::Arc;
	use axum::body::Body;
	use axum::http::{Request, StatusCode};
	use serde_json::json;
	use tower::ServiceExt; // for `oneshot`
	fn test_app() -> Router {
	// Construct a test instance of the app router with a default AppState.
	let state = Arc::new(AppState::default());
	routes(state)
	}
	#[tokio::test]
	async fn canonical_model_info_known_model_returns_200() {
	let app = test_app();
	// Use a model name that is expected to be treated as a known/valid model.
	let body = serde_json::to_vec(&json!({ "model": "known-model" })).unwrap();
	let request = Request::builder()
	.method("POST")
	.uri("/config/canonical-model-info")
	.header("content-type", "application/json")
	.body(Body::from(body))
	.unwrap();
	let response = app.oneshot(request).await.unwrap();
	assert_eq!(response.status(), StatusCode::OK);
	}
	#[tokio::test]
	async fn canonical_model_info_unknown_model_returns_404() {
	let app = test_app();
	// Use a clearly invalid/unknown model name to trigger the 404 path.
	let body = serde_json::to_vec(&json!({ "model": "this-model-should-not-exist" })).unwrap();
	let request = Request::builder()
	.method("POST")
	.uri("/config/canonical-model-info")
	.header("content-type", "application/json")
	.body(Body::from(body))
	.unwrap();
	let response = app.oneshot(request).await.unwrap();
	assert_eq!(response.status(), StatusCode::NOT_FOUND);
	}
	}

[Copilot]

with_fast() builds the fast model config via ModelConfig::new(...), so any settings already applied to the main config (e.g., temperature, toolshim, toolshim_model, request_params, max_tokens overrides) won't carry over to the fast model when complete_fast() calls use_fast_model(); consider deriving the fast config from self.clone() and only overriding the model identity/limits (or merging fast-model limits into a cloned base config).

[Copilot]

use_fast_model() returns the nested fast_model_config as-is, which can drop runtime/session configuration from the primary ModelConfig (e.g., temperature/toolshim/request params) and cause fast completions to behave differently beyond just model name/limits; consider returning a clone of self with only the model name + canonical limits replaced, or explicitly merging the fast config into the base config.

Suggested change

	*fast_config.clone()
	// Start from the current configuration and override only the model identity
	// and canonical limits from the fast model, preserving runtime/session settings.
	let mut merged = self.clone();
	merged.model = fast_config.model.clone();
	merged.context_limit = fast_config.context_limit;
	merged.max_tokens = fast_config.max_tokens;
	merged

[DOsinga]

have a look at some of what copilot says, but let's get this in

[DOsinga]

here and elsewhere do we need to map the error? we're just bubbling the other things up

[DOsinga]

yeah, let's remove the 404 handler

[Copilot]

Pull request overview

Copilot reviewed 64 out of 65 changed files in this pull request and generated 3 comments.

[Copilot]

response.json() will throw if the server returns HTTP 200 with an empty body and no Content-Length: 0 header (a case the removed fallback previously handled), which can turn a successful response into a client-side error; consider restoring the empty-body-safe JSON parsing path (read as text, parse if non-empty, else {}) or broadening the empty-body detection to also handle empty text bodies.

[Copilot]

Using sessionInputTokens || localInputTokens (and the similar output/token assignments) treats 0 as “missing” and will incorrectly fall back when the true token count is zero; use ?? (if values can be null/undefined) or an explicit check (e.g., sessionInputTokens > 0 ? ... : ...) so zero is preserved.

[Copilot]

with_canonical_limits and max_output_tokens() now affect request shaping (e.g., provider payload max_tokens) but there are no unit tests covering the canonical-registry fill behavior; add a focused test that uses a known bundled canonical model to assert context/output limits get populated as expected.

[Copilot]

Pull request overview

Copilot reviewed 52 out of 53 changed files in this pull request and generated 1 comment.

[Copilot]

The canonical limits lookup is being called with the provider name (name) instead of the model name (model_name). This will always fail to find canonical model data because it's passing the provider name as both the provider and model parameters to with_canonical_limits.

Suggested change

	.with_canonical_limits(name)
	.with_canonical_limits(model_name)

[Copilot]

Pull request overview

Copilot reviewed 54 out of 55 changed files in this pull request and generated no new comments.