Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add Loki to audit logs example #1573

Merged
merged 4 commits into from
May 2, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions examples/audit/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
loki/
31 changes: 16 additions & 15 deletions examples/audit/README.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,23 @@
# Audit Event Logging Example
<p align="center">
<img src="../images/logos/loki.svg" alt="Grafana Loki" width=250 height=250 />
</p>

**This feature is still under development, and this example is subject to change.**
# Audit Event Logging / Grafana Loki Example

This example shows how you can run Flipt with Audit Event logging enabled to a file on disk.
This example shows how you can run Flipt with audit event logging enabled to a file using the `log` audit sink.

This works by setting the two environment variables `FLIPT_AUDIT_SINKS_LOG_ENABLED` and `FLIPT_AUDIT_SINKS_LOG_FILE`:

**Note**: Support for audit events were added in [v1.21.0](https://github.com/flipt-io/flipt/releases/tag/v1.21.0) of Flipt.

```bash
FLIPT_AUDIT_SINKS_LOG_ENABLED=true
FLIPT_AUDIT_SINKS_LOG_FILE=/var/log/flipt/audit.log
```

The auditable events currently are CRUD (except for read) operations on `flags`, `variants`, `segments`, `constraints`, `rules`, `distributions`, and `namespaces`. If you do any of these operations through the API, it should emit an audit event log to the specified location.

Since docker containers are ephemeral and data within the container is lost when the container exits, we mount a local file on the host to the audit event log location in the container as a volume. You would have to create the file [first](https://github.com/moby/moby/issues/21612#issuecomment-202984678) before starting the container:

```bash
mkdir -p /tmp/flipt && touch /tmp/flipt/audit.log
```
The auditable events currently are `create`, `update`, and `delete` operations on `flags`, `variants`, `segments`, `constraints`, `rules`, `distributions`, `namespaces`, and `tokens`. If you do any of these operations through the API, Flipt will emit an audit event log to the specified location.

and `tail` the logs as you are making API requests to the Flipt server when the container is running.

```bash
tail -f /tmp/flipt/audit.log
```
In this example, we are using Grafana's [Loki](https://grafana.com/docs/loki/latest/) to scrape, aggregate, and label the audit logs on the container and index them for searching via the Grafana/Loki integration.

## Requirements

Expand All @@ -36,3 +30,10 @@ To run this example application you'll need:

1. Run `docker-compose up` from this directory
1. Open the Flipt UI (default: [http://localhost:8080](http://localhost:8080))
1. Create some sample data: Flags/Segments/etc.
1. Open the Grafana Explore UI (default: [http://localhost:3000/explore](http://localhost:3000/explore))
1. Enter a sample query in the 'code' input, ex: `{type="flag"} |= `` | json` and click 'Run Query'.
1. You should see a table of audit events for flags. You can change the `type` label to `segment`, `variant`, etc. to see other types of audit events and also change the `action` label as well to filter on event actions such as `created`, `updated`, and `deleted`.
1. The Loki docs have more information on how to query and filter the logs: [https://grafana.com/docs/loki/latest/logql/](https://grafana.com/docs/loki/latest/logql/)

!['Audit Events/Loki Example'](../images/loki.png)
80 changes: 78 additions & 2 deletions examples/audit/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,89 @@ version: "3"

services:
flipt:
build: ../../
image: flipt/flipt:latest
command: ["./flipt", "--force-migrate"]
volumes:
- /tmp/flipt/audit.log:/var/log/flipt/audit.log
- ./loki/:/var/log/
ports:
- "8080:8080"
environment:
- FLIPT_LOG_LEVEL=debug
- FLIPT_AUDIT_SINKS_LOG_ENABLED=true
- FLIPT_AUDIT_SINKS_LOG_FILE=/var/log/flipt/audit.log
- FLIPT_META_TELMETRY_ENABLED=false
networks:
- flipt_network

# Since the Loki containers are running as user 10001 and the mounted data volume is owned by root,
# Loki would not have permissions to create the directories.
# Therefore the init container changes permissions of the mounted directory.
init:
image: grafana/loki:2.7.3
user: root
entrypoint:
- "chown"
- "10001:10001"
- "/loki"
volumes:
- ./loki:/loki
networks:
- flipt_network

promtail:
image: grafana/promtail:2.7.4
volumes:
- ./loki/:/var/log/
- ./promtail.yml:/etc/promtail/promtail.yml
ports:
- "9080:9080"
command: -config.file=/etc/promtail/promtail.yml
networks:
- flipt_network

loki:
image: grafana/loki:2.7.3
volumes:
- ./loki/:/loki
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
depends_on:
- init
- promtail
networks:
- flipt_network

grafana:
environment:
- GF_PATHS_PROVISIONING=/etc/grafana/provisioning
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
entrypoint:
- sh
- -euc
- |
mkdir -p /etc/grafana/provisioning/datasources
cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
orgId: 1
url: http://loki:3100
uid: loki
basicAuth: false
isDefault: true
version: 1
editable: false
EOF
/run.sh
image: grafana/grafana:latest
ports:
- "3000:3000"
networks:
- flipt_network

networks:
flipt_network:
28 changes: 28 additions & 0 deletions examples/audit/promtail.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
server:
http_listen_port: 9080
grpc_listen_port: 0
log_level: "info"

positions:
filename: /tmp/positions.yaml

clients:
- url: http://loki:3100/loki/api/v1/push
tenant_id: docker

scrape_configs:
- job_name: flipt-audit-logs
static_configs:
- targets:
- localhost
labels:
job: flipt-audit-logs
__path__: /var/log/flipt/*.log
pipeline_stages:
- json:
expressions:
type: "type"
action: "action"
- labels:
type:
action:
1 change: 1 addition & 0 deletions examples/images/logos/loki.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/images/loki.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.