Implement metrics metadata API

[juliusv]

Currently multiple UI tools (such as PromLens and Grafana) surface metric metadata (HELP + TYPE information) to the user when building queries. This is very useful, but can only work when the PromQL-compatible endpoint also implements the metric metadata API endpoint (https://prometheus.io/docs/prometheus/latest/querying/api/#querying-metric-metadata). Thanos itself does not store this information for historical metrics yet, but it would be possible to obtain current metadata from Prometheus servers via a new feature in the Thanos Sidecar + Store protocol and then aggregate them into a response to the Query client. Would this be doable or am I missing something?

[GreenRover]

+1

[bwplotka]

Hey, sorry for the lag!

It's definitely quite easy to work to do, and something we also planned for Exemplars: #3435

Work that has to be done is really similar to what we did with @s-urbaniak when we enabled federated Rules APIs: c733564#diff-ae8f55a2dc006142e7688086d880b8aceaac7406e5f88243fb56215867448f3e The Rules API was the first one on top of StoreAPI so it requires more changes. From now one adding new API should be trivial.

For example Targets API is in review and close to being merged: #3350

It's worth noting there are ideas to improve the code and configuration aspect a little bit but it's not blocking.

Currently steps to add federated API to Querier:

Will use @2nick contribution as an example as it was perfect.

From very high level (you need to check PR for details)

1. Add endpoint flag e.g for targets

2. Add new DNS provider

3. Extend storeset

Currently, it tells Querier what endpoints have what APIs (exemplars, Store, Target, rules, metadata etc) (:

4. Add HTTP API to Querier

5. If you want to fetch something from Prometheus add client method (will be used by sidecar)

6. Add new gRPC service

e.g https://github.com/thanos-io/thanos/blob/d9796e8133310243bb65d38b73ac028bc92e1815/pkg/targets/targetspb/rpc.proto

/// Targets represents API that is responsible for gathering targets and their states.
service Targets {
    /// Targets has info for all targets.
    /// Returned targets are expected to include external labels.
    rpc Targets (TargetsRequest) returns (stream TargetsResponse);
}

message TargetsRequest {
    enum State {
        ANY = 0;
        /// This will make sure strings.ToLower(.String()) will match 'active' and 'dropped' values for
        /// Prometheus HTTP API.
        ACTIVE = 1;
        DROPPED = 2;
    }
    State state = 1;
    PartialResponseStrategy partial_response_strategy = 2;
}

message TargetsResponse {
    oneof result {
        /// structure with targets. It is up to server implementation to decide how many of those to put here within single frame.
        TargetDiscovery targets = 1;

        /// warning is considered an information piece in place of series for warning purposes.
        /// It is used to warn rule API users about suspicious cases or partial response (if enabled).
        string warning = 2;
    }
}

NOTE: Make sure types have [(gogoproto.jsontag) = "activeTargets"]; so we can use the same types for json marshalling on HTTP APIs

7. Add new package and implement gRPC Service

e.g for sidecar

e.g for merging results in querier:

7. Add unit tests
8. Add e2e tests

[juliusv]

@bwplotka Wow, thanks for providing such a detailed guide for implementing this change! Your definition of "trivial" may be different from mine, as it still seems like quite a large change in terms of lines of code and touched files (maybe several days worth of work for someone new to Thanos?). But at least it seems conceptually straightforward then what needs to be done, and with those detailed steps it's much easier for anyone to pick it up!

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[juliusv]

Regarding stalebot, I saw that @yeya24 created #3686 to address this a month ago, but there hasn't been a review yet.