Set Labels from Environment Variables

[base698]

Running promtail on a cluster, it would be handy to have information stored in labels from environment variables. Examples would be hostnames, regions, rack location, etc. Something that has to be specified during runtime.

[tomwilkie]

@base698 can you use config management to inject this into the config file? There is an external labels field for this kind of thing - see https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/promtail.libsonnet#L42

[base698]

I'm not familiar with libsonnet. Where is this referenced: $._config.promtail_config.external_labels ?

[yubozhao]

@base698 The reference for external_labels is at

loki/pkg/promtail/client/client.go

Line 56 in 40ae03b

ExternalLabels model.LabelSet `yaml:"external_labels,omitempty"`

You can pass in as '--client.external_labels=LABELS' or generate a dynamic promtail-config base on your labels.

I will put up an example of this later on

[almariah]

@yubozhao --client.external_labels is not supported yet

[yubozhao]

@almariah It is not implemented yet? That's too bad. I thought it is implemented with

loki/pkg/promtail/client/client.go

Line 241 in 439e7bb

ls = c.externalLabels.Merge(ls)

[almariah]

@yubozhao Yes externalLabels is merged but from config file. Look at:

loki/pkg/promtail/client/client.go

Lines 68 to 77 in 439e7bb

	func (c *Config) RegisterFlags(flags *flag.FlagSet) {
	flags.Var(&c.URL, "client.url", "URL of log server")
	flags.DurationVar(&c.BatchWait, "client.batch-wait", 1*time.Second, "Maximum wait period before sending batch.")
	flags.IntVar(&c.BatchSize, "client.batch-size-bytes", 100*1024, "Maximum batch size to accrue before sending. ")
	flag.IntVar(&c.BackoffConfig.MaxRetries, "client.max-retries", 5, "Maximum number of retires when sending batches.")
	flag.DurationVar(&c.BackoffConfig.MinBackoff, "client.min-backoff", 100*time.Millisecond, "Initial backoff time between retries.")
	flag.DurationVar(&c.BackoffConfig.MaxBackoff, "client.max-backoff", 5*time.Second, "Maximum backoff time between retries.")
	flag.DurationVar(&c.Timeout, "client.timeout", 10*time.Second, "Maximum time to wait for server to respond to a request")
	}

This is where parsing CLI are configured and '--client.external_labels=LABELS' not yet implemented.

[almariah]

@yubozhao I guess because parsing a map from CLI is not implemented in https://github.com/cortexproject/cortex/tree/master/pkg/util/flagext which is used in promtail for parsing flags

[yubozhao]

@almariah oh cool. Thanks for point it out for me! I haven't touch Loki for a long time, thanks for bring me up to speed. I remember you could pass variables as container arguments. Maybe that is changed now?