Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

promxy in front of thanos query-frontend #577

Closed
mushrushu opened this issue Mar 23, 2023 · 4 comments · Fixed by #580
Closed

promxy in front of thanos query-frontend #577

mushrushu opened this issue Mar 23, 2023 · 4 comments · Fixed by #580

Comments

@mushrushu
Copy link

Hello!

I'm having issues using promxy in front of thanos query frontend.

We have a prometheus instance with fairly recent metrics, there is also a thanos instance provided by external team for long term metrics storage. We decided to use promxy as a glue for both instances to have a unified metrics datasource in Grafana.

I've set this configuration up but saw and error in promxy webui: "Warning: Error fetching metrics list: Unexpected response status when fetching metric names: Unprocessable Entity". I've tested configurations with single datasource and the error persisted with thanos only.

As far as I can tell, promxy tries a get request to /api/v1/label/name/values with parameters end=9223309901257974&start=-9223309901257974 to thanos and it gets "end timestamp must not be before start time" in return. However prometheus is doing just fine in such a scenario.

my promxy config is:

promxy:
  server_groups:
    - static_configs:
        - targets:
            - "thanos:9090"

Am I missing something in my config file? Is it even a valid scenario for promxy?

@jacksontj
Copy link
Owner

I'm actually a bit surprised you would see this on latest as the time shouldn't be sent down (fixed in #575 ). If you are still seeing this we'll have to dig in some why. I imagine this is an interplay between the prometheus API and Thanos though. prometheus (for some reason) has mintime as that massive negative number -- which some systems don't handle. Technically prometheus APIs should support that time range (even though it is a bit crazy); but as a workaround you could use an absolute time range with truncate to make sure the start time isn't so negative.

#575

@mushrushu
Copy link
Author

My bad, originally I was testing on v0.0.77, so the described behaviour was for that version. However, when a new version became available [v0.0.78] I tried it but was not thorough enough. I saw the same behaviour in ui and didn't do anything else.
Now I see that there is now start parameter but the end parameter is still there with the same value as before and the result is still the same. "end timestamp must not be before start time"

Do I understand correctly that with the current version both time parameters (i.e. start/end) should be gone?

Should relative_time_range have the same effect for the matter as absolute_time_range?

Will try absolute_time_range anyway and report back later. Thanks.

@mushrushu
Copy link
Author

mushrushu commented Mar 27, 2023

I've tried to use absolute time range but to no avail. There request parameters were still the same (i.e. end=9223309901257974) and I've still got the same error "end timestamp must not be before start time".

my promxy config was:

promxy:
  server_groups:
    - static_configs:
        - targets:
            - "thanos:9090"
      absolute_time_range:
        start: '2023-03-01T01:00:00Z'
        end: '2023-03-26T23:00:00Z'
        truncate: true 

jacksontj added a commit that referenced this issue Mar 29, 2023
Upstream prom's API "rounds" to millisecond precision so the min/max
we get from promql will only be at millisecond precision

Fixes #577
jacksontj added a commit that referenced this issue Mar 29, 2023
Upstream prom's API "rounds" to millisecond precision so the min/max
we get from promql will only be at millisecond precision

Fixes #577
@jacksontj
Copy link
Owner

I got some time today to dig into this -- and congrats you have found a prometheus quirk! Although prometheus has a defined min and max time which include nanosecond precision -- it effectively truncates down to a millisecond timestamp before passing down to the querier (surprisingly it rounds instead of truncates, but just odd -- prometheus/prometheus#4939).

So to fix this we can simply truncate the min/max time to millisecond precision and it works.

In addition, you uncovered a bug in the time filter logic -- it doesn't filter time for LabelNames or LabelValues (the filter feature was added prior to those APIs being exposed). I'll also have a fix for that as well!

jacksontj added a commit that referenced this issue Mar 29, 2023
jacksontj added a commit that referenced this issue Mar 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants