Skip to content

LCORE-1206add extra e2e tests to align with OLS #1091

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
radofuchs merged 1 commit into from
Feb 2, 2026
Merged

LCORE-1206add extra e2e tests to align with OLS #1091

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 31 additions & 11 deletions tests/e2e/features/query.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,16 +76,6 @@ Feature: Query endpoint API tests
Then The status code of the response is 401
And The body of the response contains No token found in Authorization header

Scenario: Check if LLM responds to sent question with error when model does not exist
Given The system is in default state
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
When I use "query" to ask question with authorization header
"""
{"query": "Write a simple code for reversing string", "model": "does-not-exist", "provider": "does-not-exist"}
"""
Then The status code of the response is 404
And The body of the response contains Model not found

Scenario: Check if LLM responds to sent question with error when attempting to access conversation
Given The system is in default state
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
Expand All @@ -96,6 +86,16 @@ Feature: Query endpoint API tests
Then The status code of the response is 404
And The body of the response contains Conversation not found

Scenario: Check if LLM responds to sent question with error when attempting to access conversation with incorrect conversation ID format
Given The system is in default state
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
When I use "query" to ask question with authorization header
"""
{"conversation_id": "123e4567", "query": "Write a simple code for reversing string", "model": "{MODEL}", "provider": "{PROVIDER}"}
"""
Then The status code of the response is 422
And The body of the response contains Value error, Improper conversation ID '123e4567'

Scenario: Check if LLM responds for query request with error for missing query
Given The system is in default state
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
Expand Down Expand Up @@ -138,8 +138,28 @@ Scenario: Check if LLM responds for query request with error for missing query
Then The status code of the response is 422
And The body of the response contains Value error, Provider must be specified if model is specified

Scenario: Check if LLM responds for query request with error for unknown model
Given The system is in default state
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
When I use "query" to ask question with authorization header
"""
{"query": "Say hello", "provider": "{PROVIDER}", "model":"unknown"}
"""
Then The status code of the response is 404
And The body of the response contains Model with ID unknown does not exist

Scenario: Check if LLM responds for query request with error for unknown provider
Given The system is in default state
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
When I use "query" to ask question with authorization header
"""
{"query": "Say hello", "model": "{MODEL}", "provider":"unknown"}
"""
Then The status code of the response is 404
And The body of the response contains Model with ID gpt-4o-mini does not exist

@skip-in-library-mode
Scenario: Check if LLM responds for query request with error for missing provider
Scenario: Check if LLM responds for query request with error for inability to connect to llama-stack
Given The system is in default state
And The llama-stack connection is disrupted
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
Expand Down
20 changes: 20 additions & 0 deletions tests/e2e/features/streaming_query.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,26 @@ Feature: streaming_query endpoint API tests
Then The status code of the response is 422
And The body of the response contains Value error, Provider must be specified if model is specified

Scenario: Check if LLM responds for query request with error for unknown model
Given The system is in default state
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
When I use "streaming_query" to ask question with authorization header
"""
{"query": "Say hello", "provider": "{PROVIDER}", "model":"unknown"}
"""
Then The status code of the response is 404
And The body of the response contains Model with ID unknown does not exist

Scenario: Check if LLM responds for query request with error for unknown provider
Given The system is in default state
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
When I use "streaming_query" to ask question with authorization header
"""
{"query": "Say hello", "model": "{MODEL}", "provider":"unknown"}
"""
Then The status code of the response is 404
And The body of the response contains Model with ID gpt-4o-mini does not exist

Scenario: Check if LLM responds properly when XML and JSON attachments are sent
Given The system is in default state
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
Expand Down
Loading