From ceeb9414e152821d01e603a43901a1649087d3a4 Mon Sep 17 00:00:00 2001 From: Jay Clifford <45856600+Jayclifford345@users.noreply.github.com> Date: Thu, 25 Jul 2024 16:16:38 +0100 Subject: [PATCH] docs: Send Data Alloy page + Interactive Examples (#13367) Co-authored-by: J Stickler --- docs/sources/send-data/alloy/_index.md | 64 +++ .../alloy/examples/alloy-kafka-logs.md | 405 ++++++++++++++++++ .../alloy/examples/alloy-otel-logs.md | 292 +++++++++++++ 3 files changed, 761 insertions(+) create mode 100644 docs/sources/send-data/alloy/_index.md create mode 100644 docs/sources/send-data/alloy/examples/alloy-kafka-logs.md create mode 100644 docs/sources/send-data/alloy/examples/alloy-otel-logs.md diff --git a/docs/sources/send-data/alloy/_index.md b/docs/sources/send-data/alloy/_index.md new file mode 100644 index 0000000000000..daad0d6906db7 --- /dev/null +++ b/docs/sources/send-data/alloy/_index.md @@ -0,0 +1,64 @@ +--- +title: Ingesting logs to Loki using Alloy +menuTitle: Grafana Alloy +description: Configuring Grafana Alloy to send logs to Loki. +weight: 250 +--- + + +# Ingesting logs to Loki using Alloy + +Grafana Alloy is a versatile observability collector that can ingest logs in various formats and send them to Loki. We recommend Alloy as the primary method for sending logs to Loki, as it provides a more robust and feature-rich solution for building a highly scalable and reliable observability pipeline. + +{{< figure src="/media/docs/alloy/flow-diagram-small-alloy.png" alt="Alloy flow diagram" >}} + +## Installing Alloy + +To get started with Grafana Alloy and send logs to Loki, you need to install and configure Alloy. You can follow the [Alloy documentation](https://grafana.com/docs/alloy/latest/get-started/install/) to install Alloy on your preferred platform. + +## Components of Alloy for logs + +Alloy pipelines are built using components that perform specific functions. For logs these can be broken down into three categories: + +- **Collector:** These components collect/receive logs from various sources. This can be scraping logs from a file, receiving logs over HTTP, gRPC or ingesting logs from a message queue. +- **Transformer:** These components can be used to manipulate logs before they are sent to a writer. This can be used to add additional metadata, filter logs, or batch logs before sending them to a writer. +- **Writer:** These components send logs to the desired destination. Our documentation will focus on sending logs to Loki, but Alloy supports sending logs to various destinations. + +### Log components in Alloy + +Here is a non-exhaustive list of components that can be used to build a log pipeline in Alloy. For a complete list of components, refer to the [components list](https://grafana.com/docs/alloy/latest/reference/components/). + +| Type | Component | +|------------|-----------------------------------------------------------------------------------------------------| +| Collector | [loki.source.api](https://grafana.com/docs/alloy/latest/reference/components/loki.source.api/) | +| Collector | [loki.source.awsfirehose](https://grafana.com/docs/alloy/latest/reference/components/loki.source.awsfirehose/) | +| Collector | [loki.source.azure_event_hubs](https://grafana.com/docs/alloy/latest/reference/components/loki.source.azure_event_hubs/) | +| Collector | [loki.source.cloudflare](https://grafana.com/docs/alloy/latest/reference/components/loki.source.cloudflare/) | +| Collector | [loki.source.docker](https://grafana.com/docs/alloy/latest/reference/components/loki.source.docker/) | +| Collector | [loki.source.file](https://grafana.com/docs/alloy/latest/reference/components/loki.source.file/) | +| Collector | [loki.source.gcplog](https://grafana.com/docs/alloy/latest/reference/components/loki.source.gcplog/) | +| Collector | [loki.source.gelf](https://grafana.com/docs/alloy/latest/reference/components/loki.source.gelf/) | +| Collector | [loki.source.heroku](https://grafana.com/docs/alloy/latest/reference/components/loki.source.heroku/) | +| Collector | [loki.source.journal](https://grafana.com/docs/alloy/latest/reference/components/loki.source.journal/) | +| Collector | [loki.source.kafka](https://grafana.com/docs/alloy/latest/reference/components/loki.source.kafka/) | +| Collector | [loki.source.kubernetes](https://grafana.com/docs/alloy/latest/reference/components/loki.source.kubernetes/) | +| Collector | [loki.source.kubernetes_events](https://grafana.com/docs/alloy/latest/reference/components/loki.source.kubernetes_events/) | +| Collector | [loki.source.podlogs](https://grafana.com/docs/alloy/latest/reference/components/loki.source.podlogs/) | +| Collector | [loki.source.syslog](https://grafana.com/docs/alloy/latest/reference/components/loki.source.syslog/) | +| Collector | [loki.source.windowsevent](https://grafana.com/docs/alloy/latest/reference/components/loki.source.windowsevent/) | +| Collector | [otelcol.receiver.loki](https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.loki/) | +| Transformer| [loki.relabel](https://grafana.com/docs/alloy/latest/reference/components/loki.relabel/) | +| Transformer| [loki.process](https://grafana.com/docs/alloy/latest/reference/components/loki.process/) | +| Writer | [loki.write](https://grafana.com/docs/alloy/latest/reference/components/loki.write/) | +| Writer | [otelcol.exporter.loki](https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.loki/) | +| Writer | [otelcol.exporter.logging](https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.logging/) | + + +## Interactive Tutorials + +To learn more about how to configure Alloy to send logs to Loki within different scenarios, follow these interactive tutorials: + +- [Sending OpenTelemetry logs to Loki using Alloy]({{< relref "./examples/alloy-otel-logs" >}}) +- [Sending logs over Kafka to Loki using Alloy]({{< relref "./examples/alloy-kafka-logs" >}}) + + diff --git a/docs/sources/send-data/alloy/examples/alloy-kafka-logs.md b/docs/sources/send-data/alloy/examples/alloy-kafka-logs.md new file mode 100644 index 0000000000000..f75cfcc72ac74 --- /dev/null +++ b/docs/sources/send-data/alloy/examples/alloy-kafka-logs.md @@ -0,0 +1,405 @@ +--- +title: Sending Logs to Loki via Kafka using Alloy +menuTitle: Sending Logs to Loki via Kafka using Alloy +description: Configuring Grafana Alloy to recive logs via Kafka and send them to Loki. +weight: 250 +killercoda: + title: Sending Logs to Loki via Kafka using Alloy + description: Configuring Grafana Alloy to recive logs via Kafka and send them to Loki. + backend: + imageid: ubuntu +--- + + + +# Sending Logs to Loki via Kafka using Alloy + +Alloy natively supports receiving logs via Kafka. In this example, we will configure Alloy to receive logs via Kafka using two different methods: +- [loki.source.kafka](https://grafana.com/docs/alloy/latest/reference/components/loki.source.kafka): reads messages from Kafka using a consumer group and forwards them to other `loki.*` components. +- [otelcol.receiver.kafka](https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.kafka/): accepts telemetry data from a Kafka broker and forwards it to other `otelcol.*` components. + + + +## Dependencies + +Before you begin, ensure you have the following to run the demo: + +- Docker +- Docker Compose + +{{< admonition type="tip" >}} +Alternatively, you can try out this example in our interactive learning environment: [Sending Logs to Loki via Kafka using Alloy](https://killercoda.com/grafana-labs/course/loki/alloy-kafka-logs). + +It's a fully configured environment with all the dependencies already installed. + +![Interactive](/media/docs/loki/loki-ile.svg) + +Provide feedback, report bugs, and raise issues in the [Grafana Killercoda repository](https://github.com/grafana/killercoda). +{{< /admonition >}} + + + +## Scenario +In this scenario, we have a microservices application called the Carnivorous Greenhouse. This application consists of the following services: +- **User Service:** Manages user data and authentication for the application. Such as creating users and logging in. +- **Plant Service:** Manages the creation of new plants and updates other services when a new plant is created. +- **Simulation Service:** Generates sensor data for each plant. +- **Websocket Service:** Manages the websocket connections for the application. +- **Bug Service:** A service that when enabled, randomly causes services to fail and generate additional logs. +- **Main App:** The main application that ties all the services together. +- **Database:** A database that stores user and plant data. + +Each service generates logs that are sent to Alloy via Kafka. In this example, they are sent on two different topics: +- `loki`: This sends a structured log formatted message (json). +- `otlp`: This sends a serialized OpenTelemetry log message. + +You would not typically do this within your own application, but for the purposes of this example we wanted to show how Alloy can handle different types of log messages over Kafka. + + + + + +## Step 1: Environment setup + +In this step, we will set up our environment by cloning the repository that contains our demo application and spinning up our observability stack using Docker Compose. + +1. To get started, clone the repository that contains our demo application: + + ```bash + git clone -b microservice-kafka https://github.com/grafana/loki-fundamentals.git + ``` + +1. Next we will spin up our observability stack using Docker Compose: + + + ```bash + docker compose -f loki-fundamentals/docker-compose.yml up -d + ``` + + + {{< docs/ignore >}} + + + ```bash + docker-compose -f loki-fundamentals/docker-compose.yml up -d + ``` + + + {{< /docs/ignore >}} + + This will spin up the following services: + ```console + ✔ Container loki-fundamentals-grafana-1 Started + ✔ Container loki-fundamentals-loki-1 Started + ✔ Container loki-fundamentals-alloy-1 Started + ✔ Container loki-fundamentals-zookeeper-1 Started + ✔ Container loki-fundamentals-kafka-1 Started + ``` + +We will be access two UI interfaces: +- Alloy at [http://localhost:12345](http://localhost:12345) +- Grafana at [http://localhost:3000](http://localhost:3000) + + + + +## Step 2: Configure Alloy to ingest raw Kafka logs + +In this first step, we will configure Alloy to ingest raw Kafka logs. To do this, we will update the `config.alloy` file to include the Kafka logs configuration. + +### Open your Code Editor and Locate the `config.alloy` file + +Grafana Alloy requires a configuration file to define the components and their relationships. The configuration file is written using Alloy configuration syntax. We will build the entire observability pipeline within this configuration file. To start, we will open the `config.alloy` file in the code editor: + +{{< docs/ignore >}} +**Note: Killercoda has an inbuilt Code editor which can be accessed via the `Editor` tab.** +1. Expand the `loki-fundamentals` directory in the file explorer of the `Editor` tab. +1. Locate the `config.alloy` file in the `loki-fundamentals` directory (Top level directory). +1. Click on the `config.alloy` file to open it in the code editor. +{{< /docs/ignore >}} + + +1. Open the `loki-fundamentals` directory in a code editor of your choice. +1. Locate the `config.alloy` file in the top level directory, `loki-fundamentals'. +1. Click on the `config.alloy` file to open it in the code editor. + + +You will copy all three of the following configuration snippets into the `config.alloy` file. + +### Source logs from kafka + +First, we will configure the Loki Kafka source. `loki.source.kafka` reads messages from Kafka using a consumer group and forwards them to other `loki.*` components. + +The component starts a new Kafka consumer group for the given arguments and fans out incoming entries to the list of receivers in `forward_to`. + +Add the following configuration to the `config.alloy` file: +```alloy +loki.source.kafka "raw" { + brokers = ["kafka:9092"] + topics = ["loki"] + forward_to = [loki.write.http.receiver] + relabel_rules = loki.relabel.kafka.rules + version = "2.0.0" + labels = {service_name = "raw_kafka"} +} +``` + +In this configuration: +- `brokers`: The Kafka brokers to connect to. +- `topics`: The Kafka topics to consume. In this case, we are consuming the `loki` topic. +- `forward_to`: The list of receivers to forward the logs to. In this case, we are forwarding the logs to the `loki.write.http.receiver`. +- `relabel_rules`: The relabel rules to apply to the incoming logs. This can be used to generate labels from the temporary internal labels that are added by the Kafka source. +- `version`: The Kafka protocol version to use. +- `labels`: The labels to add to the incoming logs. In this case, we are adding a `service_name` label with the value `raw_kafka`. This will be used to identify the logs from the raw Kafka source in the Log Explorer App in Grafana. + +For more information on the `loki.source.kafka` configuration, see the [Loki Kafka Source documentation](https://grafana.com/docs/alloy/latest/reference/components/loki.source.kafka/). + +### Create a dynamic relabel based on Kafka topic + +Next, we will configure the Loki relabel rules. The `loki.relabel` component rewrites the label set of each log entry passed to its receiver by applying one or more relabeling rules and forwards the results to the list of receivers in the component’s arguments. In our case we are directly calling the rule from the `loki.source.kafka` component. + +Now add the following configuration to the `config.alloy` file: +```alloy +loki.relabel "kafka" { + forward_to = [loki.write.http.receiver] + rule { + source_labels = ["__meta_kafka_topic"] + target_label = "topic" + } +} +``` + +In this configuration: +- `forward_to`: The list of receivers to forward the logs to. In this case, we are forwarding the logs to the `loki.write.http.receiver`. Though in this case, we are directly calling the rule from the `loki.source.kafka` component. So `forward_to` is being used as a placeholder as it is required by the `loki.relabel` component. +- `rule`: The relabeling rule to apply to the incoming logs. In this case, we are renaming the `__meta_kafka_topic` label to `topic`. + +For more information on the `loki.relabel` configuration, see the [Loki Relabel documentation](https://grafana.com/docs/alloy/latest/reference/components/loki.relabel/). + +### Write logs to Loki + +Lastly, we will configure the Loki write component. `loki.write` receives log entries from other loki components and sends them over the network using the Loki logproto format. + +And finally, add the following configuration to the `config.alloy` file: +```alloy +loki.write "http" { + endpoint { + url = "http://loki:3100/loki/api/v1/push" + } +} +``` + +In this configuration: +- `endpoint`: The endpoint to send the logs to. In this case, we are sending the logs to the Loki HTTP endpoint. + +For more information on the `loki.write` configuration, see the [Loki Write documentation](https://grafana.com/docs/alloy/latest/reference/components/loki.write/). + +### Reload the Alloy configuration to check the changes + +Once added, save the file. Then run the following command to request Alloy to reload the configuration: + + +```bash +curl -X POST http://localhost:12345/-/reload +``` + + +The new configuration will be loaded. You can verify this by checking the Alloy UI: [http://localhost:12345](http://localhost:12345). + +## Stuck? Need help? + +If you get stuck or need help creating the configuration, you can copy and replace the entire `config.alloy` using the completed configuration file: + + + +```bash +cp loki-fundamentals/completed/config-raw.alloy loki-fundamentals/config.alloy +curl -X POST http://localhost:12345/-/reload +``` + + + + + + +## Step 3: Configure Alloy to ingest OpenTelemetry logs via Kafka + +Next we will configure Alloy to also ingest OpenTelemetry logs via Kafka, we need to update the Alloy configuration file once again. We will add the new components to the `config.alloy` file along with the existing components. + +### Open your Code Editor and Locate the `config.alloy` file + +Like before, we generate our next pipeline configuration within the same `config.alloy` file. You will add the following configuration snippets to the file **in addition** to the existing configuration. Essentially, we are configuring two pipelines within the same Alloy configuration file. + + +### Source OpenTelemetry logs from Kafka + +First, we will configure the OpenTelemetry Kafaka receiver. `otelcol.receiver.kafka` accepts telemetry data from a Kafka broker and forwards it to other `otelcol.*` components. + +Now add the following configuration to the `config.alloy` file: +```alloy +otelcol.receiver.kafka "default" { + brokers = ["kafka:9092"] + protocol_version = "2.0.0" + topic = "otlp" + encoding = "otlp_proto" + + output { + logs = [otelcol.processor.batch.default.input] + } +} +``` + +In this configuration: +- `brokers`: The Kafka brokers to connect to. +- `protocol_version`: The Kafka protocol version to use. +- `topic`: The Kafka topic to consume. In this case, we are consuming the `otlp` topic. +- `encoding`: The encoding of the incoming logs. Which decodes messages as OTLP protobuf. +- `output`: The list of receivers to forward the logs to. In this case, we are forwarding the logs to the `otelcol.processor.batch.default.input`. + +For more information on the `otelcol.receiver.kafka` configuration, see the [OpenTelemetry Receiver Kafka documentation](https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.kafka/). + + +### Batch OpenTelemetry logs before sending + +Next, we will configure a OpenTelemetry processor. `otelcol.processor.batch` accepts telemetry data from other otelcol components and places them into batches. Batching improves the compression of data and reduces the number of outgoing network requests required to transmit data. This processor supports both size and time based batching. + +Now add the following configuration to the `config.alloy` file: +```alloy +otelcol.processor.batch "default" { + output { + logs = [otelcol.exporter.otlphttp.default.input] + } +} +``` + +In this configuration: +- `output`: The list of receivers to forward the logs to. In this case, we are forwarding the logs to the `otelcol.exporter.otlphttp.default.input`. + +For more information on the `otelcol.processor.batch` configuration, see the [OpenTelemetry Processor Batch documentation](https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.batch/). + +### Write OpenTelemetry logs to Loki + +Lastly, we will configure the OpenTelemetry exporter. `otelcol.exporter.otlphttp` accepts telemetry data from other otelcol components and writes them over the network using the OTLP HTTP protocol. We will use this exporter to send the logs to Loki's native OTLP endpoint. + +Finally, add the following configuration to the `config.alloy` file: +```alloy +otelcol.exporter.otlphttp "default" { + client { + endpoint = "http://loki:3100/otlp" + } +} +``` + +In this configuration: +- `client`: The client configuration for the exporter. In this case, we are sending the logs to the Loki OTLP endpoint. + +For more information on the `otelcol.exporter.otlphttp` configuration, see the [OpenTelemetry Exporter OTLP HTTP documentation](https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.otlphttp/). + +### Reload the Alloy configuration to check the changes + +Once added, save the file. Then run the following command to request Alloy to reload the configuration: + +```bash +curl -X POST http://localhost:12345/-/reload +``` + + +The new configuration will be loaded. You can verify this by checking the Alloy UI: [http://localhost:12345](http://localhost:12345). + +## Stuck? Need help (Full Configuration)? + +If you get stuck or need help creating the configuration, you can copy and replace the entire `config.alloy`. This differs from the previous `Stuck? Need help` section as we are replacing the entire configuration file with the completed configuration file. Rather than just adding the first Loki Raw Pipeline configuration. + + +```bash +cp loki-fundamentals/completed/config.alloy loki-fundamentals/config.alloy +curl -X POST http://localhost:12345/-/reload +``` + + + + + + +## Step 3: Start the Carnivorous Greenhouse + +In this step, we will start the Carnivorous Greenhouse application. To start the application, run the following command: + +{{< admonition type="note" >}} +This docker-compose file relies on the `loki-fundamentals_loki` docker network. If you have not started the observability stack, you will need to start it first. +{{< /admonition >}} + + +{{< docs/ignore >}} + +**Note: This docker-compose file relies on the `loki-fundamentals_loki` docker network. If you have not started the observability stack, you will need to start it first.** + +{{< /docs/ignore >}} + + +```bash +docker compose -f loki-fundamentals/greenhouse/docker-compose-micro.yml up -d --build +``` + + + +{{< docs/ignore >}} + + +```bash +docker-compose -f loki-fundamentals/greenhouse/docker-compose-micro.yml up -d --build +``` + + +{{< /docs/ignore >}} + +This will start the following services: +```console + ✔ Container greenhouse-db-1 Started + ✔ Container greenhouse-websocket_service-1 Started + ✔ Container greenhouse-bug_service-1 Started + ✔ Container greenhouse-user_service-1 Started + ✔ Container greenhouse-plant_service-1 Started + ✔ Container greenhouse-simulation_service-1 Started + ✔ Container greenhouse-main_app-1 Started +``` + +Once started, you can access the Carnivorous Greenhouse application at [http://localhost:5005](http://localhost:5005). Generate some logs by interacting with the application in the following ways: + +- Create a user +- Log in +- Create a few plants to monitor +- Enable bug mode to activate the bug service. This will cause services to fail and generate additional logs. + +Finally to view the logs in Loki, navigate to the Loki Logs Explore view in Grafana at [http://localhost:3000/a/grafana-lokiexplore-app/explore](http://localhost:3000/a/grafana-lokiexplore-app/explore). + + + + + + +## Summary + +In this example, we configured Alloy to ingest logs via Kafka. We configured Alloy to ingest logs in two different formats: raw logs and OpenTelemetry logs. Where to go next? + +{{< docs/ignore >}} + +### Back to Docs +Head back to where you started from to continue with the Loki documentation: [Loki documentation](https://grafana.com/docs/loki/latest/send-data/alloy) + +{{< /docs/ignore >}} + +## Further reading + +For more information on Grafana Alloy, refer to the following resources: +- [Grafana Alloy getting started examples](https://grafana.com/docs/alloy/latest/tutorials/) +- [Grafana Alloy common task examples](https://grafana.com/docs/alloy/latest/tasks/) +- [Grafana Alloy component reference](https://grafana.com/docs/alloy/latest/reference/components/) + +## Complete metrics, logs, traces, and profiling example + +If you would like to use a demo that includes Mimir, Loki, Tempo, and Grafana, you can use [Introduction to Metrics, Logs, Traces, and Profiling in Grafana](https://github.com/grafana/intro-to-mlt). `Intro-to-mltp` provides a self-contained environment for learning about Mimir, Loki, Tempo, and Grafana. + +The project includes detailed explanations of each component and annotated configurations for a single-instance deployment. Data from `intro-to-mltp` can also be pushed to Grafana Cloud. + + + \ No newline at end of file diff --git a/docs/sources/send-data/alloy/examples/alloy-otel-logs.md b/docs/sources/send-data/alloy/examples/alloy-otel-logs.md new file mode 100644 index 0000000000000..caf787fc07ecb --- /dev/null +++ b/docs/sources/send-data/alloy/examples/alloy-otel-logs.md @@ -0,0 +1,292 @@ +--- +title: Sending OpenTelemetry logs to Loki using Alloy +menuTitle: Sending OpenTelemetry logs to Loki using Alloy +description: Configuring Grafana Alloy to send OpenTelemetry logs to Loki. +weight: 250 +killercoda: + title: Sending OpenTelemetry logs to Loki using Alloy + description: Configuring Grafana Alloy to send OpenTelemetry logs to Loki. + backend: + imageid: ubuntu +--- + + + +# Sending OpenTelemetry logs to Loki using Alloy + +Alloy natively supports receiving logs in the OpenTelemetry format. This allows you to send logs from applications instrumented with OpenTelemetry to Alloy, which can then be sent to Loki for storage and visualization in Grafana. In this example, we will make use of 3 Alloy components to achieve this: +- **OpenTelemetry Receiver:** This component will receive logs in the OpenTelemetry format via HTTP and gRPC. +- **OpenTelemetry Processor:** This component will accept telemetry data from other `otelcol.*` components and place them into batches. Batching improves the compression of data and reduces the number of outgoing network requests required to transmit data. +- **OpenTelemetry Exporter:** This component will accept telemetry data from other `otelcol.*` components and write them over the network using the OTLP HTTP protocol. We will use this exporter to send the logs to Loki's native OTLP endpoint. + + + +## Dependencies + +Before you begin, ensure you have the following to run the demo: + +- Docker +- Docker Compose + +{{< admonition type="tip" >}} +Alternatively, you can try out this example in our interactive learning environment: [Sending Logs to Loki via Kafka using Alloy](https://killercoda.com/grafana-labs/course/loki/alloy-otel-logs). + +It's a fully configured environment with all the dependencies already installed. + +![Interactive](/media/docs/loki/loki-ile.svg) + +Provide feedback, report bugs, and raise issues in the [Grafana Killercoda repository](https://github.com/grafana/killercoda). +{{< /admonition >}} + + + +## Scenario + +In this scenario, we have a microservices application called the Carnivourse Greenhouse. This application consists of the following services: + +- **User Service:** Manages user data and authentication for the application. Such as creating users and logging in. +- **Plant Service:** Manages the creation of new plants and updates other services when a new plant is created. +- **Simulation Service:** Generates sensor data for each plant. +- **Websocket Service:** Manages the websocket connections for the application. +- **Bug Service:** A service that when enabled, randomly causes services to fail and generate additional logs. +- **Main App:** The main application that ties all the services together. +- **Database:** A database that stores user and plant data. + +Each service generates logs using the OpenTelemetry SDK and exports to Alloy in the OpenTelemetry format. Alloy then ingests the logs and sends them to Loki. We will configure Alloy to ingest OpenTelemetry logs, send them to Loki, and view the logs in Grafana. + + + + + +## Step 1: Environment setup + +In this step, we will set up our environment by cloning the repository that contains our demo application and spinning up our observability stack using Docker Compose. + +1. To get started, clone the repository that contains our demo application: + + ```bash + git clone -b microservice-otel https://github.com/grafana/loki-fundamentals.git + ``` + +1. Next we will spin up our observability stack using Docker Compose: + + + ```bash + docker compose -f loki-fundamentals/docker-compose.yml up -d + ``` + + + {{< docs/ignore >}} + + + ```bash + docker-compose -f loki-fundamentals/docker-compose.yml up -d + ``` + + + {{< /docs/ignore >}} + + This will spin up the following services: + ```console + ✔ Container loki-fundamentals-grafana-1 Started + ✔ Container loki-fundamentals-loki-1 Started + ✔ Container loki-fundamentals-alloy-1 Started + ``` + +We will be access two UI interfaces: +- Alloy at [http://localhost:12345](http://localhost:12345) +- Grafana at [http://localhost:3000](http://localhost:3000) + + + + +## Step 2: Configure Alloy to ingest OpenTelemetry logs + +To configure Alloy to ingest OpenTelemetry logs, we need to update the Alloy configuration file. To start, we will update the `config.alloy` file to include the OpenTelemetry logs configuration. + +### Open your Code Editor and Locate the `config.alloy` file + +Grafana Alloy requires a configuration file to define the components and their relationships. The configuration file is written using Alloy configuration syntax. We will build the entire observability pipeline within this configuration file. To start, we will open the `config.alloy` file in the code editor: + +{{< docs/ignore >}} +**Note: Killercoda has an inbuilt Code editor which can be accessed via the `Editor` tab.** +1. Expand the `loki-fundamentals` directory in the file explorer of the `Editor` tab. +1. Locate the `config.alloy` file in the top level directory, `loki-fundamentals'. +1. Click on the `config.alloy` file to open it in the code editor. +{{< /docs/ignore >}} + + +1. Open the `loki-fundamentals` directory in a code editor of your choice. +1. Locate the `config.alloy` file in the `loki-fundamentals` directory (Top level directory). +1. Click on the `config.alloy` file to open it in the code editor. + + +You will copy all three of the following configuration snippets into the `config.alloy` file. + +### Recive OpenTelemetry logs via gRPC and HTTP + +First, we will configure the OpenTelemetry receiver. `otelcol.receiver.otlp` accepts logs in the OpenTelemetry format via HTTP and gRPC. We will use this receiver to receive logs from the Carnivorous Greenhouse application. + +Now add the following configuration to the `config.alloy` file: +```alloy + otelcol.receiver.otlp "default" { + http {} + grpc {} + + output { + logs = [otelcol.processor.batch.default.input] + } + } +``` + +In this configuration: +- `http`: The HTTP configuration for the receiver. This configuration is used to receive logs in the OpenTelemetry format via HTTP. +- `grpc`: The gRPC configuration for the receiver. This configuration is used to receive logs in the OpenTelemetry format via gRPC. +- `output`: The list of processors to forward the logs to. In this case, we are forwarding the logs to the `otelcol.processor.batch.default.input`. + +For more information on the `otelcol.receiver.otlp` configuration, see the [OpenTelemetry Receiver OTLP documentation](https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.otlp/). + + +### Create batches of logs using a OpenTelemetry Processor + +Next, we will configure a OpenTelemetry processor. `otelcol.processor.batch` accepts telemetry data from other `otelcol` components and places them into batches. Batching improves the compression of data and reduces the number of outgoing network requests required to transmit data. This processor supports both size and time based batching. + +Now add the following configuration to the `config.alloy` file: +```alloy +otelcol.processor.batch "default" { + output { + logs = [otelcol.exporter.otlphttp.default.input] + } +} +``` + +In this configuration: +- `output`: The list of receivers to forward the logs to. In this case, we are forwarding the logs to the `otelcol.exporter.otlphttp.default.input`. + +For more information on the `otelcol.processor.batch` configuration, see the [OpenTelemetry Processor Batch documentation](https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.batch/). + +### Export logs to Loki using a OpenTelemetry Exporter + +Lastly, we will configure the OpenTelemetry exporter. `otelcol.exporter.otlphttp` accepts telemetry data from other `otelcol` components and writes them over the network using the OTLP HTTP protocol. We will use this exporter to send the logs to Loki's native OTLP endpoint. + +Now add the following configuration to the `config.alloy` file: +```alloy +otelcol.exporter.otlphttp "default" { + client { + endpoint = "http://loki:3100/otlp" + } +} +``` + +For more information on the `otelcol.exporter.otlphttp` configuration, see the [OpenTelemetry Exporter OTLP HTTP documentation](https://grafana.com/docs/alloy/latest/reference/components/otelcol.exporter.otlphttp/). + +### Reload the Alloy configuration + +Once added, save the file. Then run the following command to request Alloy to reload the configuration: + +```bash +curl -X POST http://localhost:12345/-/reload +``` + + +The new configuration will be loaded. You can verify this by checking the Alloy UI: [http://localhost:12345](http://localhost:12345). + +## Stuck? Need help? + +If you get stuck or need help creating the configuration, you can copy and replace the entire `config.alloy` using the completed configuration file: + + +```bash +cp loki-fundamentals/completed/config.alloy loki-fundamentals/config.alloy +curl -X POST http://localhost:12345/-/reload +``` + + + + + + +## Step 3: Start the Carnivorous Greenhouse + +In this step, we will start the Carnivorous Greenhouse application. To start the application, run the following command: + +{{< admonition type="note" >}} +This docker-compose file relies on the `loki-fundamentals_loki` docker network. If you have not started the observability stack, you will need to start it first. +{{< /admonition >}} + + +{{< docs/ignore >}} + +**Note: This docker-compose file relies on the `loki-fundamentals_loki` docker network. If you have not started the observability stack, you will need to start it first.** + +{{< /docs/ignore >}} + + +```bash +docker compose -f loki-fundamentals/greenhouse/docker-compose-micro.yml up -d --build +``` + + + +{{< docs/ignore >}} + + +```bash +docker-compose -f loki-fundamentals/greenhouse/docker-compose-micro.yml up -d --build +``` + + +{{< /docs/ignore >}} + +This will start the following services: +```bash + ✔ Container greenhouse-db-1 Started + ✔ Container greenhouse-websocket_service-1 Started + ✔ Container greenhouse-bug_service-1 Started + ✔ Container greenhouse-user_service-1 Started + ✔ Container greenhouse-plant_service-1 Started + ✔ Container greenhouse-simulation_service-1 Started + ✔ Container greenhouse-main_app-1 Started +``` + +Once started, you can access the Carnivorous Greenhouse application at [http://localhost:5005](http://localhost:5005). Generate some logs by interacting with the application in the following ways: + +- Create a user +- Log in +- Create a few plants to monitor +- Enable bug mode to activate the bug service. This will cause services to fail and generate additional logs. + +Finally to view the logs in Loki, navigate to the Loki Logs Explore view in Grafana at [http://localhost:3000/a/grafana-lokiexplore-app/explore](http://localhost:3000/a/grafana-lokiexplore-app/explore). + + + + + + +## Summary + +In this example, we configured Alloy to ingest OpenTelemetry logs and send them to Loki. This was a simple example to demonstrate how to send logs from an application instrumented with OpenTelemetry to Loki using Alloy. Where to go next? + +{{< docs/ignore >}} + +### Back to Docs +Head back to where you started from to continue with the Loki documentation: [Loki documentation](https://grafana.com/docs/loki/latest/send-data/alloy) + +{{< /docs/ignore >}} + + +## Further reading + +For more information on Grafana Alloy, refer to the following resources: +- [Grafana Alloy getting started examples](https://grafana.com/docs/alloy/latest/tutorials/) +- [Grafana Alloy common task examples](https://grafana.com/docs/alloy/latest/tasks/) +- [Grafana Alloy component reference](https://grafana.com/docs/alloy/latest/reference/components/) + +## Complete metrics, logs, traces, and profiling example + +If you would like to use a demo that includes Mimir, Loki, Tempo, and Grafana, you can use [Introduction to Metrics, Logs, Traces, and Profiling in Grafana](https://github.com/grafana/intro-to-mlt). `Intro-to-mltp` provides a self-contained environment for learning about Mimir, Loki, Tempo, and Grafana. + +The project includes detailed explanations of each component and annotated configurations for a single-instance deployment. Data from `intro-to-mltp` can also be pushed to Grafana Cloud. + + + \ No newline at end of file