Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(quickstart): use unified meters #652

Merged
merged 7 commits into from
Mar 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 16 additions & 5 deletions config.example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,21 @@ telemetry:
# tokenSecret: this-isnt-secure

meters:
- slug: m1
eventType: api-calls
valueProperty: $.duration_ms
# Sample meter to count API requests
- slug: api_requests_total # Unique identifier for the meter
description: API Requests
eventType: request # Filter events by type
aggregation: COUNT # Aggregation method: COUNT, SUM, etc.
groupBy:
method: $.method # HTTP Method: GET, POST, etc.
route: $.route # Route: /products/:product_id

# Sample meter to count LLM Token Usage
- slug: tokens_total
description: AI Token Usage
eventType: prompt # Filter events by type
aggregation: SUM
valueProperty: $.tokens # JSONPath to parse usage value
groupBy:
method: $.method
path: $.path
model: $.model # AI model used: gpt4-turbo, etc.
type: $.type # Prompt type: input, output, system
2 changes: 1 addition & 1 deletion examples/ingest-logs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,6 @@ Transformations in Vector uses the [Vector Remap Language](https://vector.dev/do

1. First, start OpenMeter. Refer to the [quickstart guide](/quickstart) for instructions.
2. Execute `docker compose up` within this example directory.
3. To query meters, use the following command: `curl http://localhost:8888/api/v1/meters/m1/values`.
3. To query meters, use the following command: `curl http://localhost:8888/api/v1/meters/api_requests_total/values`.

Note: It's important that you run quickstart's `docker compose` first.
43 changes: 22 additions & 21 deletions quickstart/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,15 +31,14 @@ curl -X POST http://localhost:8888/api/v1/events \
--data-raw '
{
"specversion" : "1.0",
"type": "api-calls",
"type": "request",
"id": "00001",
"time": "2023-01-01T00:00:00.001Z",
"source": "service-0",
"subject": "customer-1",
"data": {
"duration_ms": "1",
"method": "GET",
"path": "/hello"
"route": "/hello"
}
}
'
Expand All @@ -53,15 +52,14 @@ curl -X POST http://localhost:8888/api/v1/events \
--data-raw '
{
"specversion" : "1.0",
"type": "api-calls",
"type": "request",
"id": "00002",
"time": "2023-01-01T00:00:00.001Z",
"source": "service-0",
"subject": "customer-1",
"data": {
"duration_ms": "1",
"method": "GET",
"path": "/hello"
"route": "/hello"
}
}
'
Expand All @@ -75,15 +73,14 @@ curl -X POST http://localhost:8888/api/v1/events \
--data-raw '
{
"specversion" : "1.0",
"type": "api-calls",
"type": "request",
"id": "00003",
"time": "2023-01-02T00:00:00.001Z",
"source": "service-0",
"subject": "customer-1",
"data": {
"duration_ms": "1",
"method": "GET",
"path": "/hello"
"route": "/hello"
}
}
'
Expand All @@ -94,7 +91,7 @@ curl -X POST http://localhost:8888/api/v1/events \
Query the usage hourly:

```sh
curl 'http://localhost:8888/api/v1/meters/m1/query?windowSize=HOUR&groupBy=method&groupBy=path' | jq
curl 'http://localhost:8888/api/v1/meters/api_requests_total/query?windowSize=HOUR&groupBy=method&groupBy=route' | jq
```

```json
Expand All @@ -108,7 +105,7 @@ curl 'http://localhost:8888/api/v1/meters/m1/query?windowSize=HOUR&groupBy=metho
"subject": null,
"groupBy": {
"method": "GET",
"path": "/hello"
"route": "/hello"
}
},
{
Expand All @@ -118,7 +115,7 @@ curl 'http://localhost:8888/api/v1/meters/m1/query?windowSize=HOUR&groupBy=metho
"subject": null,
"groupBy": {
"method": "GET",
"path": "/hello"
"route": "/hello"
}
}
]
Expand All @@ -128,7 +125,7 @@ curl 'http://localhost:8888/api/v1/meters/m1/query?windowSize=HOUR&groupBy=metho
Query the total usage for `customer-1`:

```sh
curl 'http://localhost:8888/api/v1/meters/m1/query?subject=customer-1' | jq
curl 'http://localhost:8888/api/v1/meters/api_requests_total/query?subject=customer-1' | jq
```

```json
Expand All @@ -147,21 +144,25 @@ curl 'http://localhost:8888/api/v1/meters/m1/query?subject=customer-1' | jq

## 4. Configure additional meter(s) _(optional)_

Configure how OpenMeter should process your usage events.
In this example we will meter the execution duration per API invocation, groupped by method and path.
You can think about it how AWS Lambda [charges](https://aws.amazon.com/lambda/pricing/) by execution duration on a millisecond level.
In this example we will meter LLM token usage, groupped by AI model and prompt type.
You can think about it how OpenAI [charges](https://openai.com/pricing) by tokens for ChatGPT.

Configure how OpenMeter should process your usage events in this new `tokens_total` meter.

```yaml
# ...

meters:
- slug: m1
eventType: api-calls
valueProperty: $.duration_ms
# Sample meter to count LLM Token Usage
- slug: tokens_total
description: AI Token Usage
eventType: prompt # Filter events by type
aggregation: SUM
valueProperty: $.tokens # JSONPath to parse usage value
groupBy:
method: $.method
path: $.path
model: $.model # AI model used: gpt4-turbo, etc.
type: $.type # Prompt type: input, output, system

```

## Cleanup
Expand Down
27 changes: 15 additions & 12 deletions quickstart/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,21 +11,24 @@ sink:
namespaceRefetch: 1s

meters:
- slug: m1
description: API calls
eventType: api-calls
valueProperty: $.duration_ms
aggregation: SUM
# Sample meter to count API requests
- slug: api_requests_total # Unique identifier for the meter
description: API Requests
eventType: request # Filter events by type
aggregation: COUNT # Aggregation method: COUNT, SUM, etc.
groupBy:
method: $.method
path: $.path
- slug: m2
description: Open AI Tokens
eventType: openai
valueProperty: $.total_tokens
method: $.method # HTTP Method: GET, POST, etc.
route: $.route # Route: /products/:product_id

# Sample meter to count LLM Token Usage
- slug: tokens_total
description: AI Token Usage
eventType: prompt # Filter events by type
aggregation: SUM
valueProperty: $.tokens # JSONPath to parse usage value
groupBy:
model: $.model
model: $.model # AI model used: gpt4-turbo, etc.
type: $.type # Prompt type: input, output, system

portal:
enabled: true
Expand Down
2 changes: 1 addition & 1 deletion quickstart/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ services:
volumes:
- ./config.yaml:/etc/openmeter/config.yaml
healthcheck:
test: wget --no-verbose --tries=1 --spider http://openmeter:8888/api/v1/meters/m1/query || exit 1
test: wget --no-verbose --tries=1 --spider http://openmeter:8888/api/v1/meters/api_requests_total/query || exit 1
interval: 5s
timeout: 3s
retries: 100
Expand Down
15 changes: 6 additions & 9 deletions quickstart/ingest.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,14 @@ curl -X POST http://localhost:8888/api/v1/events \
--data-raw '
{
"specversion" : "1.0",
"type": "api-calls",
"type": "request",
"id": "00001",
"time": "2023-01-01T00:00:00.001Z",
"source": "service-0",
"subject": "customer-1",
"data": {
"duration_ms": "1",
"method": "GET",
"path": "/hello"
"route": "/hello"
}
}
'
Expand All @@ -23,15 +22,14 @@ curl -X POST http://localhost:8888/api/v1/events \
--data-raw '
{
"specversion" : "1.0",
"type": "api-calls",
"type": "request",
"id": "00002",
"time": "2023-01-01T00:00:00.001Z",
"source": "service-0",
"subject": "customer-1",
"data": {
"duration_ms": "1",
"method": "GET",
"path": "/hello"
"route": "/hello"
}
}
'
Expand All @@ -41,15 +39,14 @@ curl -X POST http://localhost:8888/api/v1/events \
--data-raw '
{
"specversion" : "1.0",
"type": "api-calls",
"type": "request",
"id": "00003",
"time": "2023-01-02T00:00:00.001Z",
"source": "service-0",
"subject": "customer-1",
"data": {
"duration_ms": "1",
"method": "GET",
"path": "/hello"
"route": "/hello"
}
}
'
2 changes: 1 addition & 1 deletion quickstart/verify.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

temp_file=$(mktemp)

curl http://localhost:8888/api/v1/meters/m1/query?windowSize=HOUR >$temp_file
curl http://localhost:8888/api/v1/meters/api_requests_total/query?windowSize=HOUR >$temp_file

[ $(cat $temp_file | jq '.data[0].value') == 2 ] || (echo "Unexpected value" && cat $temp_file && exit 1)
[ $(cat $temp_file | jq '.data[1].value') == 1 ] || (echo "Unexpected value" && cat $temp_file && exit 1)
Loading