Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does influxdb-java has limitation for the # of select queries in one query() call? #930

Open
strNewBee opened this issue May 6, 2023 · 2 comments

Comments

@strNewBee
Copy link

strNewBee commented May 6, 2023

Environment

java 1.8
SpringBoot 2.4.3
SpringCloud 3.0.5
Influxdb-java 2.20
Influxdb 1.8

Platform

client app: windows 10
influxdb server: centos 7

Question

My influxdb database stores the data from May 2nd to May 5th, and the query string is as below, which contains 5 select:

SELECT metricsA FROM measurementA WHERE time >= '2023-05-02T10:10:30.78+08:00' AND time < '2023-05-20T08:20:30.78+08:00' AND ((uniqId = 'cadd3c08082f07740c459d4366b91002' AND sign=~/.*/) OR (uniqId = 'c0c0250e096735acc6b6dc341b47504a' AND sign=~/1.1|1.2/)) GROUP BY uniqId,sign tz('Etc/GMT-8');
SELECT metricsB FROM measurementB WHERE time >= '2023-05-02T10:10:30.78+08:00' AND time < '2023-05-20T08:20:30.78+08:00' AND ((uniqId = 'cadd3c08082f07740c459d4366b91002' AND sign=~/.*/) OR (uniqId = 'c0c0250e096735acc6b6dc341b47504a' AND sign=~/1.1|1.2/)) GROUP BY uniqId,sign tz('Etc/GMT-8');
SELECT metricsC,metricsD FROM measurementC WHERE time >= '2023-05-02T10:10:30.78+08:00' AND time < '2023-05-20T08:20:30.78+08:00' AND ((uniqId = 'cadd3c08082f07740c459d4366b91002' AND sign=~/.*/) OR (uniqId = 'c0c0250e096735acc6b6dc341b47504a' AND sign=~/1.1|1.2/)) GROUP BY uniqId,sign tz('Etc/GMT-8');
SELECT metricsE FROM measurementD WHERE time >= '2023-05-02T10:10:30.78+08:00' AND time < '2023-05-20T08:20:30.78+08:00' AND ((uniqId = 'cadd3c08082f07740c459d4366b91002') OR (uniqId = 'c0c0250e096735acc6b6dc341b47504a')) GROUP BY uniqId tz('Etc/GMT-8');
SELECT metricsF FROM measurementE WHERE time >= '2023-05-02T10:10:30.78+08:00' AND time < '2023-05-20T08:20:30.78+08:00' AND ((uniqId = 'cadd3c08082f07740c459d4366b91002' AND sign=~/.*/) OR (uniqId = 'c0c0250e096735acc6b6dc341b47504a' AND sign=~/1.1|1.2/)) GROUP BY uniqId,sign tz('Etc/GMT-8');

I'm calling InfluxDB.query() to execute it, but only the first 3 select query returns QueryResult. The Json format of QueryResult is:

"result":[
    {
        "series": [
                    {
                        "name": "measurementA",
                        "tags": {
                            "sign": "1.1",
                            "uniqId": "c0c0250e096735acc6b6dc341b47504a",
                            "IP": "4.4.4.4"
                        },
                        "columns": [
                            "time",
                            "PSUIn"
                        ],
                        "values": [
                            [
                                "2023-05-03T08:04:00+08:00",
                                900.0
                            ],
                            ...
                        ]
                    }
          ]
    },
    {...},
    {...}
]

When I execute each select query seperately, they can all return QueryResults. And there is more, which is when I narrowed down the time range to { 2023-05-02T10:10:30.78+08:00 ---- 2023-05-04T10:10:30.78+08:00 } instead of { 2023-05-02T10:10:30.78+08:00 ---- 2023-05-20T08:20:30.78+08:00 }, it works just fine and gives all the results for 5 queries.

I had no clue why this happened. Is there any constrants for the number of select query in one query() call?

@strNewBee
Copy link
Author

Now i can only query 3 select in a batch, and I'm worrying about the extra I/O may slow it down. When I set batch up to more than 3, it still gets this problem by missing some queries.

public static QueryResult queryMetricsByIpRegexV2(String start, String end,
                                                    Map<QueryMapEntity, List<String>> tableMap,
                                                    Map<String, String> rs2ip,
                                                    Map<String, String> ip2regex)
    {
        /* basic where */
        StringBuilder whereBase = new StringBuilder();
        StringBuilder objBase = new StringBuilder();
        StringBuilder signBase = new StringBuilder();
        createTimeBaseRFC(start, end, whereBase);

        /* create a filter */
        objBase.append(" AND (").append(createSignFilterRegex(rs2ip, ip2regex, false)).append(")");
        signBase.append(" AND (").append(createSignFilterRegex(rs2ip, ip2regex, true)).append(")");

        /* generate queries */
        StringBuilder queryCmd = new StringBuilder();
        long queryCount = 0L;
        List<String> queryCmdList = new ArrayList<>();
        for(Map.Entry<QueryMapEntity, List<String>> pair: tableMap.entrySet()) {
            QueryMapEntity target = pair.getKey();
            Boolean is_sign = target.getIs_sign();
            SelectQueryImpl query = QueryBuilder.select(pair.getValue().toArray())
                    .from(target.getDbName(), target.getTableName())
                    .where(whereBase.toString() + (is_sign ? signBase.toString() : objBase.toString()))
                    .groupBy((Object[]) (is_sign ? groupBySign : groupByUniq))
                    .tz("Etc/GMT-8");
            queryCmd.append(query.getCommand());
            queryCount += 1;
            if(queryCount == CommonConstant.queryBatch){  // 3 in a batch by default
                queryCmdList.add(queryCmd.toString());
                queryCmd.setLength(0);
                queryCount = 0L;
            }
        }
        if(queryCount != 0)
            queryCmdList.add(queryCmd.toString());

        /* perform queries */
        QueryResult result = null;
        for(String queryStr: queryCmdList){
            log.info("influxdb query: {}", queryStr);
            if(result == null)
                result = InfluxDBUtil.query(queryStr);
            else{
                for(QueryResult.Result res: InfluxDBUtil.query(queryStr).getResults()){
                    result.getResults().add(res);
                }
            }
        }
        return result;
    }

@strNewBee strNewBee changed the title Does influxdb-java has limitation for the # of select query in one query() call? Does influxdb-java has limitation for the # of select queries in one query() call? May 6, 2023
@strNewBee
Copy link
Author

Grateful for any help. (T T)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant