Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add error during rescheduling test to L0_generative_sequence #6550

Merged
merged 1 commit into from
Nov 13, 2023
Merged

Add error during rescheduling test to L0_generative_sequence #6550

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions qa/L0_generative_sequence/generative_sequence_e2e.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,25 @@
self.assertEqual(res_count, data_item.as_numpy("OUTPUT")[0][0])
self.assertEqual(0, res_count)

def test_reschedule_error(self):
# Use short idle timeout (< backend reschedule delay: 0.5s) so that
# the backend won't be able to reschedule the request as the scheduler
# will terminate the sequence early
config = r'"sequence_batching" : { "generative_sequence" : true, "max_sequence_idle_microseconds" : 200000 }'
with grpcclient.InferenceServerClient("localhost:8001") as triton_client:
triton_client.load_model(
"generative_sequence", config=MODEL_CONFIG_BASE.format(config)
)
with self.assertRaises(InferenceServerException) as context:
# Without specifying 'generative_sequence : true', the sequence
# batcher expects sequence parameters to be provided explicitly
self.test_grpc_stream()
print(str(context.exception))
self.assertTrue(
"must specify the START flag on the first request of the sequence"
in str(context.exception)
)
Comment on lines +144 to +147

Check notice

Code scanning / CodeQL

Imprecise assert Note

assertTrue(a in b) cannot provide an informative message. Using assertIn(a, b) instead will give more informative messages.

def test_unsupported_sequence_scheduler(self):
# Override model config with scheduler settings that do not support
# request rescheduling.
Expand All @@ -145,7 +164,6 @@
# batcher expects sequence parameters to be provided explicitly
self.test_grpc_stream(sequence_id=sid, sequence_start=True)
sid += 1
print(str(context.exception))
self.assertTrue(
"Request is released with TRITONSERVER_REQUEST_RELEASE_RESCHEDULE"
in str(context.exception)
Expand All @@ -164,7 +182,6 @@
)
with self.assertRaises(InferenceServerException) as context:
self.test_grpc_stream()
print(str(context.exception))
self.assertTrue(
"Request is released with TRITONSERVER_REQUEST_RELEASE_RESCHEDULE"
in str(context.exception)
Expand Down
2 changes: 1 addition & 1 deletion qa/L0_generative_sequence/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ RET=0

CLIENT_LOG="./generative_sequence_client.log"
TEST_PY=./generative_sequence_e2e.py
EXPECTED_NUM_TESTS="4"
EXPECTED_NUM_TESTS="5"
TEST_RESULT_FILE='test_results.txt'


Expand Down
2 changes: 2 additions & 0 deletions src/test/generative_sequence/src/generative_sequence.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -533,6 +533,8 @@ TRITONBACKEND_ModelInstanceExecute(
SET_TIMESTAMP(exec_end_ns);
max_exec_end_ns = std::max(max_exec_end_ns, exec_end_ns);

// wait for 0.5 second before rescheduling the request.
std::this_thread::sleep_for(std::chrono::milliseconds(500));
// Release the request first as the testing backend may be configured to
// receive error on request release, in such a case, the error will be
// propagated back through error response.
Expand Down
Loading