epic: Basic Automated test for Installation & Inference on compatible Hardware & OS

[dan-homebrew]

Goal

• We should test for full e2e lifecycle of Cortex's main use case
• This only covers llama.cpp for v0.1

Existing Work

• Current e2e test
• Previous Google Spreadsheet
• Nicole to add Hardware Compatibility Matrix
• Cortex.cpp: Hardware & OS Compatibility Matrix [WIP] #1125

Existing Bugs

• Prev bug with appimage and Fedora bug: Cortex exited with code null immediately after loading a model #1105

Test Cases

Test Harness

• epic: unitest for cortex cpp #1069

Installation & Uninstallation

• Test successful installation on all OS and hardware Cortex.cpp: Hardware & OS Compatibility Matrix [WIP] #1125

Starting

• Starts within X seconds (previous bug: bug: Slow load on Windows on first time run of Cortex #929)

Model Running

• Successful load of cached model (e.g. tinyllama)
• Successful inference request of cached model
• Error messages are tested
• Successful unloading of model
• Ensure dylib issue is covered bug: libengine.dylib not found #953

Stopping

• Successfully stops with no dangling processes

Uninstallation

• Successfully uninstalls with no dangling files

[namchuai]

For end to end testing. I'm proposing to use pytest.

Advantages

• Easy testing our CLI and API server
• Easy to write test as well as generate from AI

[0xSage]

Current Manual QA Checklist

It is a combination of:

• unit tests (soon to be automated by feat: e2e tests for APIs #1357)
• integration & e2e tests (automation still needed)

Across the following:

• Windows 11 (online & offline)
• Ubuntu 24, 22 (online & offline)
• Mac Silicon OS 14/15 (online & offline)
• Mac Intel (online & offline)

I'm documenting this for @gabrielle-ong to ensure no tests fall through the cracks and eventually we have coverage across various kinds of tests.

Installation/Uninstallation

• it should install with network installer
• it should install with local installer
• it should uninstall with & without purging the data folder
• it should gracefully uninstall when server is still running
• it should reinstall without having conflict issues with existing cortex data folders
• it should install with correct folder permissions
• it should install with folders: /engines /models /logs, and .cortexrc, and dbs
• cortex update should update from ~3-5 versions ago to latest (+3 to 5 bump)
• cortex update should update from the previous version to latest (+1 bump)
• cortex update should update from previous stable version to latest (stable checking)
• it should gracefully update when server is actively running

Basic Commands

• cortex returns helpful text in a timely* way
• cortex -v should check output current version and check for updates
• it should correctly log to cortex-cli

Hardware Detection [WIP]

• TODO

Server

• it should start server
• it should stop server
• it should correctly log to cortex logs
• it should return server status ps

Engines

• it should CRUD engines
• it should gracefully refine engine installation if interrupted halfway
• it should update engines
• it should gracefully handle when users try to CRUD incompatible engines
• it should run gguf models on llamacpp
• it should run trtllm models on trt-llm

Model Management

• it should pull by built in model_ID
• it should pull by built-in model_id:variant
• it should pull by HF repo/model ID
• it should pull by partial HF url
• it should pull by full HF url (ending in .gguf)
• it should resume pull after interruption
• it should CRUD downloaded models
• it should correctly update state in /models
• it should import models

Model Running

• run should download missing models
• run works on already downloaded models
• run should autostart server
• chat works

With Hardware Acceleration

• it should auto offload max ngl
• it should correctly detect available GPUs
• it should gracefully detect missing dependencies/drivers

[gabrielle-ong]

Updated for v1.0.1 (Manual QA & API tests) #1535
to capture changes eg

• model management syntax
• recommend default model
• llama.cpp installed by default
• API tests
  Closing this issue in favour of iterating the QA list with each update

[dan-homebrew]

@gabrielle-ong Even though we are choosing to go with a Manual test for now, we should create an open Ticket for an Automated Test and put it in Icebox.

• We should be careful of "QA Debt", and not let it grow too much