Skip to content

Commit

Permalink
Add target allocator guidance (#642)
Browse files Browse the repository at this point in the history
* initial ta doc structure

* Add config example

* Add TA to add-on index

* fix prom casing

* Update TA description

* Update src/docs/getting-started/adot-eks-add-on/target-allocator.mdx
  • Loading branch information
bryan-aguilar authored Oct 2, 2023
1 parent 87a4c8f commit 39dda8b
Show file tree
Hide file tree
Showing 3 changed files with 104 additions and 1 deletion.
1 change: 1 addition & 0 deletions src/docs/getting-started/adot-eks-add-on.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import operatorImg2 from "assets/img/docs/gettingStarted/operator/img2.png"
#### [Advanced Collector Configuration for Amazon Managed Prometheus](/docs/getting-started/adot-eks-add-on/config-advanced)
### [Injecting Auto-instrumentation](/docs/getting-started/adot-eks-add-on/config-auto-instrumentation)
### [Kubernetes Attributes Processor](/docs/getting-started/adot-eks-add-on/k8s-attr-processor)
### [Target Allocator](/docs/getting-started/adot-eks-add-on/target-allocator)
### [Update and Cleanup](/docs/getting-started/adot-eks-add-on/update-and-cleanup)


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -246,4 +246,4 @@ and alarms that can leverage Kubernetes resource attributes.

### [Advanced Collector Configuration for Amazon Managed Prometheus](/docs/getting-started/adot-eks-add-on/config-advanced)

## [Next Topic: Updating and Cleanup](/docs/getting-started/adot-eks-add-on/update-and-cleanup)
## [Next Topic: Target Allocator](/docs/getting-started/adot-eks-add-on/target-allocator)
102 changes: 102 additions & 0 deletions src/docs/getting-started/adot-eks-add-on/target-allocator.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
---
title: 'Target Allocator'
description:
This page provides an introduction to the Target Allocator
path: '/docs/getting-started/adot-eks-add-on/target-allocator'
---

Installing the EKS Add-on gives you access to the Target Allocator (TA). Enabling the Target Allocator for an
OpenTelemetry Collector custom resource decouples Prometheus service discovery and metrics collection. When enabled,
the Target Allocator manages the configuration of the Prometheus receiver. Specifically,
the Target Allocator allows for distribution of Prometheus scrape targets across a pool of Collectors. This functionality can
be used to avoid the manual sharding of Prometheus scrape targets that can be required when horizontally scaling
a single Prometheus receiver configuration.
Visit the [Target Allocator Readme](https://github.com/open-telemetry/opentelemetry-operator/blob/main/cmd/otel-allocator/README.md)
for a deeper introduction.

## Out of Order Samples

In some cases the Target Allocator can be used to solve Prometheus remote write out of order sample errors. A common
reason Collector users will see out of order sample erorrs is due to multiple collectors scraping the same metric
endpoint. Collector users may experience this when horizontally scaling Collectors with Prometheus
receivers or daemonset deployments without manually sharding scrape targets. Both of these scenarios can lead to
out of order sample erorrs without sharding of Prometheus scrape targets.

## Getting Started

Enabling the target allocator requires setting `OpenTelemeteryCollector.spec.targetAllocator.enabled` to `true`.
This will cause the Prometheus configuration within the Collector CRD to be rewritten to use an `http_sd_config`
from the Target Allocator. The operator will also create a new deployment and service for this http_sd_config
directive.

```
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: collector-with-ta
spec:
mode: statefulset
servicAccount: adot-collector
targetAllocator:
enabled: true
config: |
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
metric_relabel_configs:
- action: labeldrop
regex: (id|name)
replacement: $$1
- action: labelmap
regex: label_(.+)
replacement: $$1
processors:
batch:
exporters:
prometheusremotewrite:
endpoint: ${AMP_REMOTE_WRITE_URL}
auth:
authenticator: sigv4auth
extensions:
sigv4auth:
region: ${AWS_REGION}
service: "aps"
service:
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [prometheusremotewrite]
```

Please reference the [OpenTelemetry Operator documentation](https://github.com/open-telemetry/opentelemetry-operator/tree/main#target-allocator)
for more information on using the Target Allocator.

### Useful links

[Target Allocator API Spec](https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/api.md#opentelemetrycollectorspectargetallocator)

[Scaling OpenTelmetery Collectors](https://opentelemetry.io/docs/collector/scaling/)

---
## [Previous Topic: Kubernetes Attributes Processor](/docs/getting-started/adot-eks-add-on/k8s-attr-processor)

## Related Topics:

### [Collector Configuration for Amazon Managed Prometheus](/docs/getting-started/adot-eks-add-on/config-amp)

### [Collector Configuration for AWS CloudWatch](/docs/getting-started/adot-eks-add-on/config-cloudwatch)

### [Advanced Collector Configuration for Amazon Managed Prometheus](/docs/getting-started/adot-eks-add-on/config-advanced)

## [Next Topic: Updating and Cleanup](/docs/getting-started/adot-eks-add-on/update-and-cleanup)

0 comments on commit 39dda8b

Please sign in to comment.