Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thanos with duplicate results more than 1 hour #6616

Closed
antikilahdjs opened this issue Aug 15, 2023 · 2 comments
Closed

Thanos with duplicate results more than 1 hour #6616

antikilahdjs opened this issue Aug 15, 2023 · 2 comments

Comments

@antikilahdjs
Copy link

antikilahdjs commented Aug 15, 2023

Name and Version

thanos:v0.31.0

What architecture are you using?

amd64

What steps will reproduce the bug?

1.Install the helm using the normal way
2.Configure S3 miniO as bucket
3.I have the Prometheus CRD and I configured the Thanos Sidecar
4.Everything works perfectly and the remote_write has been configured to sent all data to S3
5. I have the Compactor, Query,Receiver already configured

Are you using any custom parameters or values?

    query:
    ## @param query.enabled Set to true to enable Thanos Query component
    ##
    enabled: true
    ## @param query.logLevel Thanos Query log level
    ##
    logLevel: info
    ## @param query.logFormat Thanos Query log format
    ##
    logFormat: logfmt
    ## @param query.replicaLabel Replica indicator(s) along which data is de-duplicated
    ##
    replicaLabel: [prometheus_replica]
    ## Dynamically configure store APIs using DNS discovery
    ## @param query.dnsDiscovery.enabled Enable store APIs discovery via DNS
    ## @param query.dnsDiscovery.sidecarsService Sidecars service name to discover them using DNS discovery
    ## @param query.dnsDiscovery.sidecarsNamespace Sidecars namespace to discover them using DNS discovery
    ##
    dnsDiscovery:
      enabled: true
      sidecarsService: "prometheus-k8s"
      sidecarsNamespace: "kubesphere-monitoring-system"

What is the expected behavior?

If I search query before 5 minutes I got the correct result (dedup)

image

What do you see instead?

I have 2 prometheus as operator replicas with thanos then send data to my storage like miniO.

If search some query more than 1 or 2 hours the result is twice duplicated like below

image

It bring many results and the dudup is not working or I did something wrong.

Additional information

I have the thanos sidecar enabled in my Prometheus CRD and RemoteWrite to Storegateway.

Please help me to identify it or it is some bug in version that I am using.

My Promtheus wasnt configured the external label then the default label is prometheus_replica. I dont know if I put some desnecessary values into the helm
I saw this thread #6257 but keeps confuse to me what I need to do to fix it. I am tried downgraded from 0.30.2 but is the error issue and the query keeps returning duplicated and somethimes more than 4 times

image

image

Thankss

@GiedriusS
Copy link
Member

Hi! Thanks for reporting this. I think this was solved in #6257. Could you please try the newest main?

@antikilahdjs
Copy link
Author

Hi @GiedriusS I hope you doing great. I think this was fixed. Thank you so much for your attention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants