-
Notifications
You must be signed in to change notification settings - Fork 3.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Add new target manager(PubsubTarget) for promtail Lets you to scrape log entries from pubsub topic and send it to lok - Basic `Target` and `TargetManager` - Minimum config to make it work * make the receive loop on pubsub manager work * Add mmock pubsub server for testing * Remove debugging logs * Minor changes to messages pushing into loki * Basic formatter to convert GCP log entry to loki log entry * Reverting go.mod, go.sum and vendors to master branch * only pubsub dependencies * rebuild proto * Fix protobuf diffs * Remove debug printfs * Add basic formatter for GCP log entry - Create uniq label to create stream without out-of-order timestamp - extract instance id and timestamp from log entry * - Small fixes to formatter - Config changes on pubsub manager for testing * - Formatting changes - Rewrite timestamp to avoid out-of-order errors on loki - Minor config changes to Pubsub Target manager * Fix target formatter tests * PR remarks * Minor refactoring and following changes - Rename Pubsub* entities to Gcplog* - Rename pubsub package into gcplog - Add flag `KeepIncomingTimestamp` to toggle between either keeping incoming timestamps or rewriting with current timestamp - Tests for gcplog target - Docs and basic metrics to gcplog target * Add docs for cloud configure for gcplog. * Add docs for cloud configure for gcplog. * Review feedback * Checkin generated files * Making linters happy Co-authored-by: Edward Welch <edward.welch@grafana.com>
- Loading branch information
Showing
50 changed files
with
14,324 additions
and
63 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| server: | ||
| http_listen_port: 9080 | ||
| grpc_listen_port: 0 | ||
|
|
||
| positions: | ||
| filename: /tmp/positions.yaml | ||
|
|
||
| clients: | ||
| - url: http://localhost:3100/loki/api/v1/push | ||
|
|
||
| scrape_configs: | ||
| - job_name: pubsub-test | ||
| gcplog: | ||
| project_id: "grafanalabs-dev" | ||
| subscription: "dev-logs-pull" | ||
| use_incoming_timestamp: false # default rewrite timestamp. | ||
| labels: | ||
| job: pubsub-gcp |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| --- | ||
| title: Cloud setup for gcplog TargetManager | ||
| --- | ||
| This document explain how one can setup Google Cloud Platform to forward its cloud resource logs from a particular GCP project into Google Pubsub topic so that is available for Loki promtail to consume. | ||
|
|
||
| This document assumes, that reader have `gcloud` installed and have required permissions(as mentioned in #[Roles and Permission] section) | ||
|
|
||
| # Roles and Permission | ||
| User should have following roles to complete the setup. | ||
| - "roles/pubsub.editor" | ||
| - "roles/logging.configWriter" | ||
|
|
||
| # Setup Pubsub Topic | ||
| Google Pubsub Topic will act as the queue to persist log messages which then can be read from `promtail`. | ||
|
|
||
| ```bash | ||
| $ gcloud pubsub topics create $TOPIC_ID | ||
| ``` | ||
|
|
||
| e.g: | ||
| ```bash | ||
| $ gcloud pubsub topics create cloud-logs | ||
| ``` | ||
|
|
||
| # Setup Log Router | ||
| We create a log sink to forward cloud logs into pubsub topic created before | ||
|
|
||
| ```bash | ||
| $ gcloud beta logging sinks create $SINK_NAME $SINK_LOCATION $OPTIONAL_FLAGS | ||
| ``` | ||
|
|
||
| e.g: | ||
| ```bash | ||
| $ gcloud beta logging sinks create cloud-logs pubsub.googleapis.com/projects/my-project/topics/cloud-logs \ | ||
| --log-filter='resource.type=("gcs_bucket")' \ | ||
| --description="Cloud logs" | ||
| ``` | ||
|
|
||
| Above command also adds `log-filter` option which represents what type of logs should get into the destination `pubsub` topic. | ||
| For more information on adding `log-filter` refer this [document](https://cloud.google.com/logging/docs/export/configure_export_v2#creating_sink) | ||
|
|
||
| # Create Pubsub subscription for Loki | ||
| We create subscription for the pubsub topic we create above and `promtail` uses this subscription to consume log messages. | ||
|
|
||
| ```bash | ||
| $ gcloud pubsub subscriptions create cloud-logs --topic=$TOPIC_ID \ | ||
| --ack-deadline=$ACK_DEADLINE \ | ||
| --message-retention-duration=$RETENTION_DURATION \ | ||
| ``` | ||
|
|
||
| e.g: | ||
| ```bash | ||
| $ gcloud pubsub subscriptions create cloud-logs --topic=pubsub.googleapis.com/projects/my-project/topics/cloud-logs \ | ||
| --ack-deadline=10s \ | ||
| --message-retention-duration=7d \ | ||
| ``` | ||
|
|
||
| For more fine grained options, refer to the `gcloud pubsub subscriptions --help` | ||
|
|
||
| # ServiceAccount for Promtail | ||
| We need a service account with following permissions. | ||
| - pubsub.subscriber | ||
|
|
||
| This enables promtail to read log entries from the pubsub subscription created before. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| package gcplog | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "strings" | ||
| "time" | ||
|
|
||
| "cloud.google.com/go/pubsub" | ||
| json "github.com/json-iterator/go" | ||
| "github.com/prometheus/common/model" | ||
|
|
||
| "github.com/grafana/loki/pkg/logproto" | ||
| "github.com/grafana/loki/pkg/promtail/api" | ||
| ) | ||
|
|
||
| // LogEntry that will be written to the pubsub topic. | ||
| // According to the following spec. | ||
| // https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry | ||
| // nolint: golint | ||
| type GCPLogEntry struct { | ||
| LogName string `json:"logName"` | ||
| Resource struct { | ||
| Type string `json:"type"` | ||
| Labels map[string]string `json:"labels"` | ||
| } `json:"resource"` | ||
| Timestamp string `json:"timestamp"` | ||
|
|
||
| // The time the log entry was received by Logging. | ||
| // Its important that `Timestamp` is optional in GCE log entry. | ||
| ReceiveTimestamp string `json:"receiveTimestamp"` | ||
|
|
||
| TextPayload string `json:"textPayload"` | ||
|
|
||
| // NOTE(kavi): There are other fields on GCPLogEntry. but we need only need above fields for now | ||
| // anyway we will be sending the entire entry to Loki. | ||
| } | ||
|
|
||
| func format(m *pubsub.Message, other model.LabelSet, useIncomingTimestamp bool) (api.Entry, error) { | ||
| var ge GCPLogEntry | ||
|
|
||
| if err := json.Unmarshal(m.Data, &ge); err != nil { | ||
| return api.Entry{}, err | ||
| } | ||
|
|
||
| labels := model.LabelSet{ | ||
| "logName": model.LabelValue(ge.LogName), | ||
| "resourceType": model.LabelValue(ge.Resource.Type), | ||
| } | ||
| for k, v := range ge.Resource.Labels { | ||
| if !model.LabelName(k).IsValid() || !model.LabelValue(k).IsValid() { | ||
| continue | ||
| } | ||
| labels[model.LabelName(k)] = model.LabelValue(v) | ||
| } | ||
|
|
||
| // add labels from config as well. | ||
| labels = labels.Merge(other) | ||
|
|
||
| ts := time.Now() | ||
| line := string(m.Data) | ||
|
|
||
| if useIncomingTimestamp { | ||
| tt := ge.Timestamp | ||
| if tt == "" { | ||
| tt = ge.ReceiveTimestamp | ||
| } | ||
| var err error | ||
| ts, err = time.Parse(time.RFC3339, tt) | ||
| if err != nil { | ||
| return api.Entry{}, fmt.Errorf("invalid timestamp format: %w", err) | ||
| } | ||
|
|
||
| if ts.IsZero() { | ||
| return api.Entry{}, fmt.Errorf("no timestamp found in the log entry") | ||
| } | ||
| } | ||
|
|
||
| // Send only `ge.textPaylload` as log line if its present. | ||
| if strings.TrimSpace(ge.TextPayload) != "" { | ||
| line = ge.TextPayload | ||
| } | ||
|
|
||
| return api.Entry{ | ||
| Labels: labels, | ||
| Entry: logproto.Entry{ | ||
| Timestamp: ts, | ||
| Line: line, | ||
| }, | ||
| }, nil | ||
| } |
Oops, something went wrong.