Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Logcli remote storage. #1814

Merged
merged 4 commits into from
Mar 30, 2020
Merged

Logcli remote storage. #1814

merged 4 commits into from
Mar 30, 2020

Conversation

cyriltovena
Copy link
Contributor

This allows to query a remote storage (eg. bigtable/gcs) using the logcli. It requires to provide a loki configuration files.

example of usage:

logcli query --org-id=3927 --local-config=/Users/ctovena/local-gcs.yaml {container_name="ingress-nginx"}

Example of configuration:

chunk_store_config:
limits_config:
  enforce_metric_name: false
  ingestion_burst_size_mb: 20
  ingestion_rate_mb: 10
  ingestion_rate_strategy: global
  max_global_streams_per_user: 10000
  max_query_length: 12000h
  max_query_parallelism: 32
  max_streams_per_user: 0
  reject_old_samples: true
  reject_old_samples_max_age: 168h
schema_config:
  configs:
    - from: "2018-04-15"
      index:
        period: 168h
        prefix: loki_dev_index_
      object_store: gcs
      schema: v9
      store: bigtable
    - from: "2019-10-30"
      index:
        period: 168h
        prefix: loki_dev_index_
      object_store: gcs
      schema: v11
      store: bigtable
storage_config:
  bigtable:
    instance: dev-us-central1
    project: dev
  gcs:
    bucket_name:  dev
  max_chunk_batch_size: 50

This is very handy to debug store queries.

@cyriltovena
wip
e28f2c0 
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@cyriltovena
Start hooking logcli with local storage config.
e949182 
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@cyriltovena
Fixes org-id and documentation.
3534e56 
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
sandeepsukhani
sandeepsukhani approved these changes Mar 24, 2020
View reviewed changes
Copy link
Contributor

@sandeepsukhani sandeepsukhani left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A small suggestion otherwise it LGTM.

cmd/logcli/main.go Outdated
@@ -169,6 +169,7 @@ func newQuery(instant bool, cmd *kingpin.CmdClause) *query.Query {
cmd.Flag("exclude-label", "Exclude labels given the provided key during output.").StringsVar(&query.IgnoreLabelsKey)
cmd.Flag("include-label", "Include labels given the provided key during output.").StringsVar(&query.ShowLabelsKey)
cmd.Flag("labels-length", "Set a fixed padding to labels").Default("0").IntVar(&query.FixedLabelsLen)
cmd.Flag("local-config", "Execute the current query using a configured storage from a given Loki configuration file.").Default("").StringVar(&query.LocalConfig)
Copy link
Contributor

@sandeepsukhani sandeepsukhani Mar 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

local-config sounds a little confusing. Should we call it storage-config or store-config instead?
What do you think?

@cyriltovena
Copy link
Contributor Author

Good idea !

@cyriltovena
Review feedback.
e92dc4f 
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>
@cyriltovena cyriltovena merged commit ba51aad into grafana:master Mar 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sandeepsukhani sandeepsukhani sandeepsukhani approved these changes

Assignees
No one assigned
Labels
size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@cyriltovena @sandeepsukhani