Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Context deadline exceeded error handling #345

Closed
Maros112358 opened this issue Oct 2, 2023 · 4 comments · Fixed by #350
Closed

Context deadline exceeded error handling #345

Maros112358 opened this issue Oct 2, 2023 · 4 comments · Fixed by #350
Labels
bug Something isn't working
Milestone

Comments

@Maros112358
Copy link

Describe the bug
When database timeouts, error log on line 40 does not show in the logs, even though error log on line 47 shows a list of errors, where one of them is context deadline exceeded.

To Reproduce
Steps to reproduce the behavior:

  1. Use the config below
  2. Go to /metrics
  3. Check logs
  4. See error

Expected behavior
I expected to see Timeout collecting metrics from target: ...

Configuration

jobs:
  - job_name: my_job
    collectors: [mssql_timeout]
    static_configs:
      - targets:
          'my_target': 'my_connection_string'
collectors:
  - collector_name: mssql_timeout
    metrics:
      - metric_name: timeout_metrics
        type: gauge
        help: 'wait for 3 minutes in MSSQL'
        values: [xxx]
        query: |
          WAITFOR DELAY '00:03:00';

Additional context
Logs I got when reproducing the error:

ts=2023-10-02T12:04:36.730Z caller=klog.go:108 level=warn func=Warningf msg="Starting SQL exporter (version=0.12.3, branch=HEAD, revision=d25cd797ff6b4daa53a5b9cc29c4aee3c998cb9f) (go=go1.20.8, platform=darwin/arm64, user=root@3c5818fec6b5, date=20230920-16:18:01, tags=netgo)"
ts=2023-10-02T12:04:36.737Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9399
ts=2023-10-02T12:04:36.737Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9399
ts=2023-10-02T12:04:49.060Z caller=klog.go:124 level=error func=Errorf msg="Error gathering metrics: [from Gatherer #1] [job=\"my_job\", target=\"my_target\", collector=\"mssql_timeout\", query=\"timeout_metrics\"] context deadline exceeded"
@Maros112358 Maros112358 added the bug Something isn't working label Oct 2, 2023
@burningalchemist
Copy link
Owner

Hey @Maros112358 thanks for reporting, I'll have a look. 👍

@Maros112358
Copy link
Author

Maros112358 commented Oct 2, 2023

I removed the if statement and experimented with the code, but the following code does not give any more information than Error gathering metrics:

				klog.Errorf("Timeout collecting metrics from target: %s", err.Error())

It may be the problem of the underlying driver.

@Maros112358
Copy link
Author

I just found out, if you try to reproduce the bug, the sql_exporter also caches result of a "timeout" query.

How to reproduce:
0, use the aforementioned config
1, set min_interval to 1m
2, go to /metrics
3, wait for timeout → scrape_duration 10 seconds + context deadline exceeded
4, refresh the page → scrape duration under 1 second + context deadline exceeded

Is this desired behaviour?

@burningalchemist
Copy link
Owner

Hey @Maros112358, thanks a lot for the test case, I'll reproduce and fix it on my end. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants