Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrating recommended changes into telemetry doc #802

Merged
merged 1 commit into from
Jul 10, 2024
Merged

Integrating recommended changes into telemetry doc #802

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 25 additions & 15 deletions docs/how_to_guides/telemetry.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,11 @@ import Dashboard from './assets/dashboard.png';
import ViewTraces from './assets/view_traces.mp4';
import GrafanaOtelConfig from './assets/grafana_otel_config.mp4';

# Capture Metrics for your Guards
# Overview

In this document, we explain how to set up Guardrails with [OpenTelemetry (OTEL)](https://opentelemetry.io/) using either Grafana or a self-hosted OTEL collector. With this functionality enabled, you can measure latency of Guards, Large Language Models (LLMs), scuccess rates, and other metrics for your Guardrails-protected LLM calls.

## Metrics you can capture using OTEL

This package is instrumented using the OpenTelemetry Python SDK. By viewing the captured traces and derived metrics, we're able to get useful insights into how our Guards, and our LLM apps in general perform. Among other things, we're able to find:

Expand All @@ -12,12 +16,11 @@ This package is instrumented using the OpenTelemetry Python SDK. By viewing the
4. The rate at which validators Pass and Fail, within a Guard and across Guards
5. Deep dives into singular guard and validator calls

Since we are using OpenTelemetry, traces and metrics can be written to any OpenTelemetry enabled service or OTLP endpoint. This includes all major metrics providers like Grafana, New Relic, Prometheus, and Splunk.
Since we are using OpenTelemetry, traces and metrics can be written to any OpenTelemetry-enabled service or OTLP endpoint. This includes all major metrics providers like Grafana, New Relic, Prometheus, and Splunk.

This guide will show how to set up your python project to log traces to Grafana and an OTEL collector.
This guide will show how to set up your Python project to log traces to Grafana and to a self-hosted OTEL collector. For other OTEL endpoints, consult your metrics provider's documentation on OTEL support.


## Setup with Grafana
## Configure OTEL for Grafana

Grafana Cloud is a free offering by Grafana that's easy to setup and is our preferred location for storing metrics.

Expand All @@ -41,11 +44,11 @@ OTEL_EXPORTER_OTLP_HEADERS

### Setup a Guard with Telemetry

We first have to install the ```ValidLength``` guardrail from Guardrails Hub.
1. First, install the ```ValidLength``` guardrail from Guardrails Hub.

```guardrails hub install hub://guardrails/valid_length```

Then, set up your Guard the default tracer provided in the guardrails library. You can still use your desired validators
2. Next, set up your Guard the default tracer provided in the guardrails library. You can still use your desired validators:

<h5 a><strong><code>main.py</code></strong></h5>
```python
Expand All @@ -69,7 +72,7 @@ guard(
)
```

Before running the file, make sure to set the environment variables you got from Grafana
3. Before running the file, make sure to set the environment variables you got from Grafana

```bash
export GRAFANA_INSTANCE_ID=
Expand All @@ -82,34 +85,41 @@ export OTEL_EXPORTER_OTLP_ENDPOINT=
export OTEL_EXPORTER_OTLP_HEADERS=
```

Finally, run the python script
4. Finally, run the python script

```bash

python main.py

```

### Viewing traces
### View traces

There are two ways to view traces: using the Explore tab or using the Guardrails Grafana dashboard template.

The simplest way to do this is to go to your grafana stack and click on the "Explore" tab. You should see a list of traces that you can filter by service name, operation name, and more.
#### Use the Explore tab

The simplest way to do this is to go to your grafana stack and click on the "**Explore** tab. You should see a list of traces that you can filter by service name, operation name, and more.

<video autoPlay={true} style={{width: "100%"}} loop={true} controls={true}>
<source src={ViewTraces} type="video/mp4"/>
</video>

#### Use the Guardrails Grafana dashboard template

While this is easy to do, it's not the best way to get a big-picture view of how your guards are doing. For that, we should use the Guardrails Grafana dashboard template.

The template can be found here https://grafana.com/grafana/dashboards/20600-standard-guardrails-dash/
and instructions to use the template are found here https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/
**[Use the template](https://grafana.com/grafana/dashboards/20600-standard-guardrails-dash/)**

**[Template instructions](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/)**

<img src={Dashboard}/>


## OTEL Collector
## Configure OTEL for a self-hosted OpenTelemetry Collector

For advanced use cases (like if you have a metrics provider in a VPC), you can use a self-hosted OpenTelemetry Collector to receive traces and metrics from your Guard.
Standard [open telemetry environment variables](https://opentelemetry-python.readthedocs.io/en/stable/getting-started.html#configure-the-exporter) are used to configure the collector. Use the default_otel_collector_tracer when configuring your guard.
Standard [open telemetry environment variables](https://opentelemetry.io/docs/languages/python/exporters/) are used to configure the collector. Use the `default_otel_collector_tracer` when configuring your guard.

```python
from guardrails import Guard, OnFailAction
Expand Down
Loading