Skip to content

Conversation

@roomote
Copy link
Contributor

@roomote roomote bot commented Nov 26, 2025

This PR attempts to address Issue #9597 where newer OpenRouter models (like gpt-5.1, gemini-3-pro-preview, and grok-4.1-fast:free) were being incorrectly marked as invalid after the model cache was populated.

Problem

The validation logic was too strict - it would reject any model not in the cached list when the cache had more than 1 model. This caused newer models to fail validation after a few minutes when the full model list was fetched.

Solution

  1. Increased validation threshold: Changed from >1 to >10 models before validation kicks in, preventing false negatives during initial cache population
  2. Added pattern matching: For OpenRouter specifically, added regex patterns to allow known model formats even when not in cache:
    • OpenAI models (e.g., openai/gpt-5.1)
    • Google models (e.g., google/gemini-3-pro-preview)
    • X-AI models (e.g., x-ai/grok-4.1-fast:free)
    • Anthropic, Meta Llama, and Mistral models

Testing

  • Added comprehensive tests for the new validation logic
  • Tests cover both the new pattern matching and the threshold change
  • All existing tests pass

Fixes #9597

Feedback and guidance are welcome!


Important

Improve OpenRouter model validation by increasing threshold and adding pattern matching for newer models.

  • Behavior:
    • In validate.ts, increase model validation threshold from >1 to >10 models to prevent false negatives.
    • Add regex pattern matching for OpenRouter models in validateDynamicProviderModelId() to allow known formats (e.g., openai/gpt-5.1, google/gemini-3-pro-preview).
  • Testing:
    • Add tests in validate.spec.ts for new validation logic, covering pattern matching and threshold changes.
    • Rename validate.test.ts to validate.spec.ts for consistency.
  • Misc:
    • Update error messages in validate.ts to use settings:validation namespace.

This description was created by Ellipsis for e3c1bc0. You can customize this summary. It will automatically update as commits are pushed.

- Changed validation threshold from >1 to >10 models to avoid false negatives
- Added pattern matching for known model providers (OpenAI, Google, X-AI, etc.)
- Allows newer models like gpt-5.1, gemini-3-pro-preview, and grok-4.1-fast
- Added comprehensive tests for the new validation logic

Fixes #9597
@roomote roomote bot requested review from cte, jr and mrubens as code owners November 26, 2025 04:21
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working labels Nov 26, 2025
@roomote
Copy link
Contributor Author

roomote bot commented Nov 26, 2025

Rooviewer Clock   See task on Roo Cloud

Review completed. No issues found in the changes made by this PR.

The implementation looks good:

  • The threshold increase from >1 to >10 models is reasonable to avoid false negatives during initial cache population
  • The pattern matching for known OpenRouter model formats (OpenAI, Google, X-AI, Anthropic, Meta Llama, Mistral) provides a good fallback for newer models
  • The comprehensive test coverage validates both the threshold change and pattern matching behavior
  • The file rename from validate.test.ts to validate.spec.ts aligns with project conventions

Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues.

const knownPatterns = [
/^openai\/gpt-\d+(\.\d+)?/, // Matches gpt-5, gpt-5.1, etc.
/^google\/gemini-\d+/, // Matches gemini-3, gemini-3-pro, etc.
/^x-ai\/grok-\d+/, // Matches grok-4, grok-4.1, etc.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regex for x-ai/grok only matches digits (e.g. /^x-ai\/grok-\d+/). To support versions with decimals (e.g. 4.1), consider updating it to /^x-ai\/grok-\d+(\.\d+)?/ for consistency with the OpenAI pattern.

Suggested change
/^x-ai\/grok-\d+/, // Matches grok-4, grok-4.1, etc.
/^x-ai\/grok-\d+(\.\d+)?/, // Matches grok-4, grok-4.1, etc.

@hannesrudolph hannesrudolph added the Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. label Nov 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. size:L This PR changes 100-499 lines, ignoring generated files.

Projects

Status: Triage

3 participants