Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

0.0.75: result requested from sort() is wrong #490

Closed
LiuPacific opened this issue Jan 25, 2022 · 2 comments · Fixed by #492
Closed

0.0.75: result requested from sort() is wrong #490

LiuPacific opened this issue Jan 25, 2022 · 2 comments · Fixed by #492
Labels
bug

Comments

@LiuPacific
Copy link

LiuPacific commented Jan 25, 2022 •
edited
Loading

1. Describe the bug

While executing sort(xx), promxy will return ordered timeseries within the result from each server_group. However, the results across different datasources haven't been combined and sorted.

2. Version

0.0.75

3. Reproduce

preparation of data

http://127.0.0.1:8481/select/6/prometheus/api/v1/query?time=2022-01-24T18:50:00.0%2B08:00&query=up
server_group 0:

{
    "status": "success",
    "isPartial": false,
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {
                    "__name__": "up",
                    "group": "typing-node-exporter-group1",
                    "instance": "host.docker.internal:9100",
                    "job": "typing-job1",
                    "monitor": "typing-vmagent6"
                },
                "value": [
                    1643080779,
                    "1"
                ]
            }
        ]
    }
}

http://127.0.0.1:8481/select/5/prometheus/api/v1/query?time=2022-01-24T18:50:00.0%2B08:00&query=up
server_group 1:

{
    "status": "success",
    "isPartial": false,
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {
                    "__name__": "up",
                    "age": "20",
                    "city": "typing",
                    "group": "typing-node-exporter-group0",
                    "instance": "host.docker.internal:9100",
                    "job": "typing-job0",
                    "monitor": "typing-vmagent5"
                },
                "value": [
                    1643021400,
                    "1"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "age": "20",
                    "city": "typing",
                    "instance": "localhost:9090",
                    "job": "prometheus0"
                },
                "value": [
                    1643021400,
                    "1"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "age": "21",
                    "city": "typing",
                    "instance": "localhost:9090",
                    "job": "prometheus2",
                    "mark": "1"
                },
                "value": [
                    1643021400,
                    "1"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "age": "22",
                    "city": "typing",
                    "group": "typing-node-exporter-group0",
                    "instance": "host.docker.internal:9100",
                    "job": "typing-job3",
                    "mark": "2",
                    "monitor": "typing-vmagent5"
                },
                "value": [
                    1643021400,
                    "1"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "instance": "host.docker.internal:10000",
                    "job": "demo"
                },
                "value": [
                    1643021400,
                    "0"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "instance": "host.docker.internal:10001",
                    "job": "demo"
                },
                "value": [
                    1643021400,
                    "0"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "instance": "host.docker.internal:10002",
                    "job": "demo"
                },
                "value": [
                    1643021400,
                    "0"
                ]
            }
        ]
    }
}

execution

http://127.0.0.1:8082/api/v1/query?time=2022-01-24T18:50:00.0%2B08:00&query=sort(up)

The results below show us that the first value is 1 while the second is 0 and the fifth one is 1.

{
    "status": "success",
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {
                    "__name__": "up",
                    "group": "typing-node-exporter-group1",
                    "instance": "host.docker.internal:9100",
                    "job": "typing-job1",
                    "liu": "tp_server_group0",
                    "monitor": "typing-vmagent6"
                },
                "value": [
                    1643021400,
                    "1"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "instance": "host.docker.internal:10001",
                    "job": "demo",
                    "liu": "tp_server_group1"
                },
                "value": [
                    1643021400,
                    "0"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "instance": "host.docker.internal:10000",
                    "job": "demo",
                    "liu": "tp_server_group1"
                },
                "value": [
                    1643021400,
                    "0"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "instance": "host.docker.internal:10002",
                    "job": "demo",
                    "liu": "tp_server_group1"
                },
                "value": [
                    1643021400,
                    "0"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "age": "20",
                    "city": "typing",
                    "instance": "localhost:9090",
                    "job": "prometheus0",
                    "liu": "tp_server_group1"
                },
                "value": [
                    1643021400,
                    "1"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "age": "20",
                    "city": "typing",
                    "group": "typing-node-exporter-group0",
                    "instance": "host.docker.internal:9100",
                    "job": "typing-job0",
                    "liu": "tp_server_group1",
                    "monitor": "typing-vmagent5"
                },
                "value": [
                    1643021400,
                    "1"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "age": "22",
                    "city": "typing",
                    "group": "typing-node-exporter-group0",
                    "instance": "host.docker.internal:9100",
                    "job": "typing-job3",
                    "liu": "tp_server_group1",
                    "mark": "2",
                    "monitor": "typing-vmagent5"
                },
                "value": [
                    1643021400,
                    "1"
                ]
            },
            {
                "metric": {
                    "__name__": "up",
                    "age": "21",
                    "city": "typing",
                    "instance": "localhost:9090",
                    "job": "prometheus2",
                    "liu": "tp_server_group1",
                    "mark": "1"
                },
                "value": [
                    1643021400,
                    "1"
                ]
            }
        ]
    }
}

4. Direct Cause in terms of code

According to the logic on https://github.com/jacksontj/promxy/blob/v0.0.75/pkg/proxystorage/proxy.go#L479, functions of which type is Call are sent directly to downstream nodes, including sort(), of which the result need to be merged and sorted again.

5. Conclusion

The problem could also exist in other functions with similar features, which are based on whole data to evaluate, like quantile() considered before.
By far, I haven't found any other func with the feature.
@LiuPacific LiuPacific mentioned this issue Jan 25, 2022
Closed
jacksontj added a commit that referenced this issue Jan 31, 2022
@jacksontj
Re-sort sort function returns in NodeReplacer
b567660 
Fixes #490
@jacksontj jacksontj mentioned this issue Jan 31, 2022
Merged
@jacksontj
Copy link
Owner

I have a PR with a fix (local testing looks good) -- #492 would you mind testing that out?

@jacksontj jacksontj added the bug label Feb 1, 2022
jacksontj added a commit that referenced this issue Feb 3, 2022
@jacksontj
Re-sort sort function returns in NodeReplacer
2c2b15b 
Fixes #490
@LiuPacific
Copy link
Author

I have a PR with a fix (local testing looks good) -- #492 would you mind testing that out?

thanks, it works well as expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Re-sort sort function returns in NodeReplacer jacksontj/promxy
2 participants
@jacksontj @LiuPacific