planning: API for Active models

[gabrielle-ong]

Goal

• We currently have the CLI version of cortex ps to show active models and consumed resources (Ram/vram usage and how long its been running)
• we have an unofficial API GET http://localhost:39281/inferences/server/models
• we should make this an official /v1 API

Discussion / Success Criteria: @louis-jan

1. Should it 1 API to show all running models & consumed resources? ie == CLI cortex ps
2. Or separate APIs to show active status for /models (per model or all models?)
3. And separate API to show resources consumed /system

Tasklist

• add unofficial API to docs (interim) - @gabrielle-ong
• implement official API endpoint(s)

Current

GET http://localhost:39281/inferences/server/models

Response:

{
    "data": [
        {
              "engine": "cortex.llamacpp",
              "id": "llama3.2-1b-instruct",
              "model_size": 123,
              "object": "model",
              "ram": 123,
              "start_time": 123,
              "vram": 123,              
        }
    ],
    "object": "list"
}

Future: API / CLI

API

1. Feature

GET /v1/endpoint

Body:

{
    "key": "value"
}

Response

200
{
}
Error
{
}

User request