Support API Cost Breakdown

[madisonb]

Description

Per the discussion over here, I would like to be able to understand and calculate the cost of my various API endpoints running within my cortex cluster.

Right now, using AWS cost tags only gives me the overall cost of the cluster, but not my true Spot, CPU, Memory, or any other costs by replica. Using a formula or estimator is difficult considering all the variables at play.

Motivation

This has real implication for enabling teams to rapidly prototype new models and move them into production, because you can understand the cost of any individual API endpoint running within the cluster. This might allow you to focus on memory optimization, reduction in init time, speed, or anything else that helps keep your costs down.

From a contractual R&D perspective, this also helps understand costs specific for customers or programs without spinning up new dedicated clusters just for a single API endpoint or two. While that is practical is some circumstances, it gets unwieldy at large scales and would be easier to track costs based on a single cluster.

Additional context

Here is an example project that helps track kubernetes costs using Prometheus, it might be a great starting point.

https://github.com/kubecost/cost-model