If you want to test the application that you just deployed in the cluster, run this integration test using the behave package.
Behave uses behavior-driven development (BDD), an agile software development technique that encourages and facilitates collaboration between developers, QA engineers and business participants.
-
Create a virtual environment and install your dependencies using thamos(the CLI tool for interacting with Thoth and included as a dependency to this repo) from the root directory of this tutorial. Be sure to select the appropriate overlay, which in this case is
test-model
.python3 -m venv venv/ && . venv/bin/activate && thamos install -r test-model
-
Run the behave command from the root directory of the tutorial.
DEPLOYED_MODEL_URL=http://elyra-aidevsecops-tutorial-thoth-deployment-examples.apps.zero.massopen.cloud behave
Below is an example of the output you should expect to see if the behave command ran correctly:
Scenario: Deployment metrics gathering # features/gather_deployment_metrics.feature:2 Given dataset is available # features/steps/model_deployment.py:42 Given deployment is accessible # features/steps/model_deployment.py:56 When I run test to gather metrics on predict endpoint # features/steps/model_deployment.py:72 Then I should get model and application metrics # features/steps/model_deployment.py:128 1 feature passed, 0 failed, 0 skipped 1 scenario passed, 0 failed, 0 skipped 4 steps passed, 0 failed, 0 skipped, 0 undefined Took 0m3.394s
-
You can also check the metrics produced by running the following command:
cat metrics.json
{'average_latency': 0.03846242621075362, 'average_error': 0.001}
Monitor your model and application with Prometheus and Grafana