Pagination of API responses

[yesoreyeram]

Available from: Infinity version 2.0.0
Supported types: JSON
Supported parsers: Backend

Pagination

When calling an API, if the underlying system have lots of results typically the server will send responses in multiple pages.

Pagination in Infinity

Introducing pagination of API response in Infinity 2.0. Prior to infinity version 2.0.0, you can only one page at a time. To call multiple pages and consolidate, you might be adding multiple queries manually and use merge transformation.

Types of Pagination

In the modern API world, pagination is implemented in different ways. Following are the common pagination mechanisms supported in infinity

• Offset based pagination
• Page number based pagination
• Cursor based pagination

Offset based pagination

TODO: yet to be documented

Page number based pagination

TODO: yet to be documented

Cursor based pagination

TODO: yet to be documented

Iterating URLs in a loop

Apart from the regular pagination methods mentioned above, From infinity 2.0, you have the ability to iterate the URLs based on comma separated values.

For example, we would like to call 3 different pokemons API items and show in a single result. We will be using following three URL.

• https://pokeapi.co/api/v2/pokemon/bulbasaur
• https://pokeapi.co/api/v2/pokemon/ivysaur
• https://pokeapi.co/api/v2/pokemon/venusaur

As you see, we have three different URLs with 3 different paths. So I am going to to parameterise this as follows.

• https://pokeapi.co/api/v2/pokemon/{{ pokemon_name }}

Now in the infinity pagination, I can instruct it to replace {{ pokemon_name }} part of the URL with bulbasaur,ivysaur,venusaur. Under the hood, infinity will split the values by , and then replace {{ pokemon_name }} with each item in the list and make the API call.

Below screenshot is an example of how to use it.

When used with dashboard variables, you can replace the hardcoded list of items to point a variable. Example: When you have a multi-value variable called pokemon_id, you can refer it as ${pokemon_id:csv} in the list value.

Limitation

Number of maximum pages per query is limited to 5. This is to avoid resource exhaustion and DDOSing of the target system.

[ColdFire87]

@yesoreyeram this looks pretty good and useful. Do you have any date in mind releasing this? Thanks!

[mrdeadlocked]

Is it possible to up the 5 page limit? My REST API currently has a limit of 500, I can set up pagination but I only see 2500 results of the 20,000 we have. Can I increase the limit on my end someplace from this 5 page limit?

[chi-miel]

Hello,
Hello, does this feature also allow pagination in a graphql api ?

In my case, i use body type GraphQL and i need to explicitly indicate page number like :

query {
  list_server(name_search:"server_name",page:1){
    name
  }
}

[yesoreyeram]

@chi-miel - yes you can use. use the replace URL in the field type of your page parameter and then use the corresponding page tokens in the graphql query/post body.

[yesoreyeram]

@rucciva - yes it does. Use the replace URL in the field type.

[ac3xx]

Is pagination via the next page cursor still unsupported? Hitting a similar issue with the GitLab GraphQL API currently.

[dharman0125]

The hard limit of 5 pages max is too low of a limit. Understand trying to protect the source however for large datasets more pages are required.

[yesoreyeram]

Then potentially you are using wrong tool for the job.
What's the use case ?

[dharman0125]

A source tool with a hard limit of 500 datapoints per query and pagenates and I need more then 2500 datapoints.

[vesenthil]

Hi ,
My api will have more then 60000 data .i want to use pagination to retrieve the data in single hit. Which type of pagination is suitable?

[vesenthil]

I used offset pagination and getting below error.

[yesoreyeram]

60000 records

It looks like Grafana infinity is not the right choice for querying such a huge dataset. Anything involve more than 5 API calls / 10000 records, I will consider as huge performance impact. Consider querying aggregated results instead.

[jpike88]

@yesoreyeram with all due respect, I know better what tools to use for the job, I get paid to be good at that.

Instead, I see people here doing the painstaking workaround of making A,B,C queries because you limited your tool's functionality.

If 1 page however is only 10/20 records, that's just 100 records, or 10 records depending on the volume of data being fetched.

In other words please loosen the restriction and let developers make their own decisions.

[CoolZeroNL]

I'm also facing the issue of the hard limit of 5 pages max is too low of a limit. The target API im using has a hard limit of 13 records (dont ask me why) and cant be changed by input. So in this case it will only pull 65 records what is to low because in my case there are more records to process.

[CoolZeroNL]

Regarding the way of paging is working. I have some trouble te find out how to parse nested data when they are been paged. There seems is a differand behaver when the data is collected when using paging, and the inline option.

So when the data is been loaded trough the api, and is paged, we have/see the data returned in the 5 rows.

The nested data we want is that in "hits" from the api, there are 13 items in it. So now i will tell the plugin that the root is: hits and this returned in a error:

logger=plugin.yesoreyeram-infinity-datasource t=2024-02-29T16:18:19.239291243Z level=info msg="performing QueryData in infinity datasource" summarizeExpression= endpoint=queryData dsUID=ac4b8026-c324-4671-8f40-bb410b5372bd filterExpression= source=url url="http://127.0.0.1:9002/" dsName=infinity-datasource-csrf-xsrf-proxy pluginID=yesoreyeram-infinity-datasource root_selector=hits settings.AuthenticationMethod=none settings.OAuth2Settings.OAuth2Type= parser=backend type=json uname=admin uql=parse-json
logger=plugin.yesoreyeram-infinity-datasource t=2024-02-29T16:18:19.655897329Z level=error msg="plugin process exited" plugin=/opt/grafana/plugins/yesoreyeram-infinity-datasource/gpx_infinity_linux_amd64 id=108341 error="exit status 2"
logger=context userId=1 orgId=1 uname=admin t=2024-02-29T16:18:19.655916109Z level=error msg="Internal server error" error="[plugin.downstreamError] client: failed to query data: Failed to query data: rpc error: code = Unavailable desc = error reading from server: EOF" remote_addr=127.0.0.1 traceID=
logger=context userId=1 orgId=1 uname=admin t=2024-02-29T16:18:19.655997872Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=127.0.0.1 time_ms=432 duration=432.325643ms size=116 referer="https://fdn/d/d94b141c-abd3-4a85-862b-559fd4618ab1/dashboard?editPanel=3&orgId=1&tab=query" handler=/api/ds/query

When i change it to: [hits] i will get the hits, but they will not be extracted as expected.

So i have striped down the json, create a array of 2 and use it as inline data:

[{
    "hits": [
        {
            "id": 1,
            "provider": "x",
            "description": "x",
            "published": 1,
            "properties": {
                "A": true,
                "B": true

            },
            "media": [
                {
                    "id": "123",
                    "formats": {
                        "xxs": "xxs.jpg",
                        "xs": "xs.jpg"
                    },
                    "orientation": "landscape"
                },
                {
                    "id": "456",
                    "formats": {
                        "xxs": "xxs.jpg",
                        "xs": "xs.jpg"
                    },
                    "orientation": "landscape"
                }
                
            ],
            "location": {
                "lat": "1.32",
                "lon": "1.01"
            },
            "address": "x",
            "agent": {
                "id": 1,
                "name": "x"
            },
            "city": "x",
            "features": [],
            "_id": "1",
            "priceInfo": {
                "price": "1.00"
            },
            "photo": {
                "id": "123",
                "formats": {
                    "xxs": "xxs.jpg",
                    "xs": "xs.jpg"
                },
                "type": 2,
                "orientation": "landscape"
            },
            "link": "x"
        },
        {
            "id": 2,
            "provider": "x",
            "description": "x",
            "published": 1,
            "properties": {
                "A": true,
                "B": true

            },
            "media": [
                {
                    "id": "123",
                    "formats": {
                        "xxs": "xxs.jpg",
                        "xs": "xs.jpg"
                    },
                    "orientation": "landscape"
                },
                {
                    "id": "456",
                    "formats": {
                        "xxs": "xxs.jpg",
                        "xs": "xs.jpg"
                    },
                    "orientation": "landscape"
                }
                
            ],
            "location": {
                "lat": "1.32",
                "lon": "1.01"
            },
            "address": "x",
            "agent": {
                "id": 1,
                "name": "x"
            },
            "city": "x",
            "features": [],
            "_id": "2",
            "priceInfo": {
                "price": "1.00"
            },
            "photo": {
                "id": "123",
                "formats": {
                    "xxs": "xxs.jpg",
                    "xs": "xs.jpg"
                },
                "type": 2,
                "orientation": "landscape"
            },
            "link": "x"
        }
    ],
    "center": {
        "lat": 1.87,
        "lon": 1.45
    },
    "total": 87,
    "pages": 7,
    "page": 1,
    "totalDisplay": 87,
    "description": ""
},
{
    "hits": [
        {
            "id": 3,
            "provider": "x",
            "description": "x",
            "published": 1,
            "properties": {
                "A": true,
                "B": true

            },
            "media": [
                {
                    "id": "123",
                    "formats": {
                        "xxs": "xxs.jpg",
                        "xs": "xs.jpg"
                    },
                    "orientation": "landscape"
                },
                {
                    "id": "456",
                    "formats": {
                        "xxs": "xxs.jpg",
                        "xs": "xs.jpg"
                    },
                    "orientation": "landscape"
                }
                
            ],
            "location": {
                "lat": "1.32",
                "lon": "1.01"
            },
            "address": "x",
            "agent": {
                "id": 1,
                "name": "x"
            },
            "city": "x",
            "features": [],
            "_id": "3",
            "priceInfo": {
                "price": "1.00"
            },
            "photo": {
                "id": "123",
                "formats": {
                    "xxs": "xxs.jpg",
                    "xs": "xs.jpg"
                },
                "type": 2,
                "orientation": "landscape"
            },
            "link": "x"
        },
        {
            "id": 4,
            "provider": "x",
            "description": "x",
            "published": 1,
            "properties": {
                "A": true,
                "B": true

            },
            "media": [
                {
                    "id": "123",
                    "formats": {
                        "xxs": "xxs.jpg",
                        "xs": "xs.jpg"
                    },
                    "orientation": "landscape"
                },
                {
                    "id": "456",
                    "formats": {
                        "xxs": "xxs.jpg",
                        "xs": "xs.jpg"
                    },
                    "orientation": "landscape"
                }
                
            ],
            "location": {
                "lat": "1.32",
                "lon": "1.01"
            },
            "address": "x",
            "agent": {
                "id": 1,
                "name": "x"
            },
            "city": "x",
            "features": [],
            "_id": "4",
            "priceInfo": {
                "price": "1.00"
            },
            "photo": {
                "id": "123",
                "formats": {
                    "xxs": "xxs.jpg",
                    "xs": "xs.jpg"
                },
                "type": 2,
                "orientation": "landscape"
            },
            "link": "x"
        }
    ],
    "center": {
        "lat": 1.87,
        "lon": 1.45
    },
    "total": 87,
    "pages": 7,
    "page": 2,
    "totalDisplay": 87,
    "description": ""
}]

Whenever i now apply the paring hits the field is as expected selected as root, and the 2 enties are been extracted and showed as seperated rows.

ps, after some hours debugging the json file , seems that there is a strange behaver of it and found that the items in hits are not correct been used as in when there is no number, the API return "" instead of null. Is this something for the pluging to catch an fix presend failing the query ?

{
    "hits": [
        {
            "anumber": "",
            "anumber_base": ""
        },
        {
            "anumber": "121",
            "anumber_base": 121
        }
    ]
}

[wastegum]

Hi, I'm having some trouble using the pagination feature with a json "next" cursor. My API response data is formatted as follows:

{
  "somekey": "somevalue",
  "results": [
    {
      "date": "some timestamp",
      "result": "data I care about"
    },
    {
      "date": "some other timestamp",
      "result": "more data I care about"
    }
  ],
  "_links": {
    "next": {
      "href": "https://myapiurl/myendpoint?cursor=foobar"
    }
  }
}

As you can see, the data I care about is nested inside an array called "results", and I have no problem extracting this data by adding "results[]" in the "Rows/Root" parsing field with the Backend parser.

The reference to the next page exists outside of the results[] array and the API is providing me with a complete URL instead of a "next" marker. It seems that the plugin should support this via URL replacement, but no matter what I try I can't get it to pull in additional pages.

Here is the latest config I have tried:

Any help would be appreciated because without this plugin I'll need to either go write a custom plugin for this data source, or resort to replicating the data elsewhere in a data warehouse which seems like a waste of time and effort.

[jkroepke]

There is an open PR to support this scenario: #784

[cm-slu]

Hello, your solution is great.
Is there a solution to have more than 5 pages?
in my case I have 16 pages but I can have even more

[bahadir-be]

agree!
Can you make more than 5 pages in the new version?
We are using Azure managed grafana solution and we can not edit source code of this plugin :/

[GeoHaz7]

@yesoreyeram hello this is amazing stuff thank you for your work, i have a question thought.

when using Iterating URLs in a loop is there a way to add the variable that i used to loop as a column to have that value in each row to know to which variable value it belongs to?

[benayyad12]

Hello,

I solved pagination problem by adding multiple queries A,B,C for example. in A i used per_page=100 and number page_number 1 , for B per_page=100 and page_number 2. Then I merge using the transformation Merge by Id. I got all elements of page one and two in the same dashboard.

This is without BACKEND parser, it works with UQL

Thank you
Abdessalam BENAYYAD

[florianraiber]

Hi, i am trying to use the pagination feature with Azure IoT-Hub. IoT-Hub API uses headers for a cursor:

x-ms-max-item-count request header to set the page size.
X-Ms-Continuation response/request header for the cursor.

I am using this configuration and it does not work:

Do i need to set something in "Extraction path"? What does it extract using "Headers" field type?

Thanks a lot.

[Tukkan]

Correct me if I'm wrong, but I don't really understand the 5 pages limitation. If the data is paginated, why is the number of pages a performance issue?
If number of items is 20, you're fetching always 20 items anyways and the total number of items is usually passed in response data or header.
Thanks

[yesoreyeram]

Hiya.. Sorry that you are having trouble with max pages. Can you provide your thoughts and usecases in reply to #601 (comment) ? The more usecases we get, we can expedite this feature.

[Walkablenormal]

This has been mentioned numerous times in this thread, but I also believe the '5 pages' limitation should be configurable. Even when examining Grafana's native APIs, such as the Grafana OnCall API, some routes return 50 records per page. The page size is not configurable. In my opinion, a limit of 250 records is far too conservative.

[yesoreyeram]

Hiya.. Sorry that you are having trouble with max pages. Can you provide your thoughts and usecases in reply to #601 (comment) ? The more usecases we get, we can expedite this feature.

[isaacsanders]

Check out the GitHub GraphQL API for an application in GraphQL for this. It might be nice to give the GraphQL mode access to pagination.

[jpike88]

My eventual solution:

Fivetran + neon postgresql database (both have a free tier)

Works wonders

[jrote1]

There are many cases where the API that's being called is an internal API so it should be down to the developer to choose what the max number of pages is, can the limit be removed from max pages?

[yesoreyeram]

Hiya.. Sorry that you are having trouble with max pages. Can you provide your thoughts and usecases in reply to #601 (comment) ? The more usecases we get, we can expedite this feature.

[yulipumal]

Also facing the 5 pages limitation problem on an internal API.

[yesoreyeram]

Hiya.. Sorry that you are having trouble with max pages. Can you provide your thoughts and usecases in reply to #601 (comment) ? The more usecases we get, we can expedite this feature.

[valentintraen]

Same problem with pagination. 5 is very very low

[yesoreyeram]

Hiya.. Sorry that you are having trouble with max pages. Can you provide your thoughts and usecases in reply to #601 (comment) ? The more usecases we get, we can expedite this feature.

[yesoreyeram]

Background and current status of pagination feature ( Nov 2024 )

Pagination feature was introduced in infinity backend parser an year ago as an experimental/beta/undocumented feature. As there are different types of pagination and much complexities behind handling them, still we aren't in a position to make it out of beta. The complexities include how to handle error in one of the pages, when to stop the pagination, how to handle partial data, how to handle no data vs error, how to stop racing condition, making parallel requests for paginated queries to improve the performance etc.

Though this feature is in beta, We love the enthusiasm of the community who tried the pagination feature and providing continuous feedback and exploring the potentials of the infinity plugin.

Maximum number of pages option & issues behind

When the pagination feature was introduced, I strongly believed that getting paginated results in grafana is a unusual/unconventional use case. With Grafana's use cases user should query aggregated results instead of pulling every data and post processing/filtering/aggregation in the grafana where possible. Also loading large number of results with pagination into grafana's memory will put additional stress to grafana server. If one of the poor query triggers heavy pagination, it will affect all other grafana users in that instance.

While considering the pagination, we do consider not only the requests coming from dashboards but also other cron jobs such as alerting, recorded queries etc. Increasing the max pagination will increase the grafana server memory/cpu in such cases as well.

I agree that there are workarounds that a user can trigger multiple paginated queries via options such as repeat panel etc. That still doesn't justify the needs to increase the max pagination size.

Path/Options forward

Though this is beta/experimental feature, we understand that pagination is important for many users and some use cases. Also removing the limitation on max pagination is also crucial for some users.

Note

We haven't ignored the option for max page customisation. Instead, we are exploring safer options such as setting the option via environment variable so that grafana admins can take informed decision about this configuration. This will also protect our grafana cloud and partner cloud hosted grafana environments.

Also we have prioritised other important features such as supporting oAuth scenarios, making the backend parser as default, improving the documentation etc to make this plugin better. Also we are running low in bandwidth to implement/enhance the pagination. But we are open for enhancement once we have enough use case information around this.

Looking for feedback & use-cases

The more use cases we can gather from the community, the better we are able to design and prioritise a solution for the pagination feature. Feel free to provide the use-cases and feedback in this discussion.

[yesoreyeram]

Thanks @jkroepke for your use-case information. The more use-cases we get, we can design this better to serve all.

[isaacsanders]

I use this plugin a lot with the GitHub GraphQL and REST APIs, so making pagination work with those would be a game changer

[valentintraen]

I use it with Rundeck API to retrieve informations that I can't get from database. For exemple, to get projects wich need to be synch with gitlab, I have to ask the api once for each Rundeck projects and I have many of them. I had to create 20 Grafana variables to get 5 distinct project each time and create 20 queries in my panel with a pagination of 5 for each.

[mattselkey]

I'm trying to use it to grab configuration properties from an internal cloud service (VMware cloud director), the data is used to correlate configuration changes in those properties vs deployment issues across Kubernetes monitored clusters. It's a really weird use case and the vendor provided tooling just doesn't surface the data that the vendor's api dopes, but that data is paginated.

Having it in Grafana where all the other infra is monitored would be amazing, at the moment I am using some hacky scripts which run some much more slowly than the api direct, it's seconds to many minutes difference. Thank you for considering the change!! 👏

[mjhita6299]

We are trying to get billing data of multiple aws accounts, azure subscriptions
Using inifinty source to avoid maintaining any in between proxy or db for data aggregation.
You should put warning to set the pagination, rest it should be upto user to what he wants to set it up too.
Im using api call in just one panel and rest of the panels use that one panel as source, so no calls for each panel.

[flux3all]

My use case: I need to create a "scorecard" or "status" visualization to show policy compliance for a set of devices - a kind of table with devices in the rows and "device policies" (rules that check aspects of a device configuration) in the columns (fields). If a device is compliant for a policy, the table cell should be green. If the device is non-compliant, the table cell should be red.

The data source for the required visualization is the configuration management application's REST API. The REST API request uses a device ID in the URL to get a response containing a list of non-compliant policies for that device. I have 50 devices (in a dashboard variable). But given the nature of the REST API endpoint and current Grafana/Infinity limitations, I will need to configure the Grafana panel with 50 separate hard-coded REST API queries and then use Transformations to merge the results together. This is not good. Maybe I am missing something??? If I was trying to get this data with an independent script, I would loop through the variable device list to construct the REST API query URLs, not hardcode anything. Anyway, I was stumped about how to do this with Infinity until I came across this pagination feature which got me excited until I realized it's still in beta (and has a limitation of 5 pages). If there's another way to do this, please let me know. Thanks.

[fatshark-pca]

I am using Grafana from a DevOps perspective:

• I perform a request towards a build project to get a historical list of builds
• Each build consists of parallel jobs. I want to visualize each job for each build, and I can only retrieve the job status by performing a request to that particular build

Build	Job 1	Job 2
# 1	✅	✅
# 2	✅	✅
...	...	...

Based on my personal need and from what I can read from other people's comments, it doesn't seem like pagination is the main need, but the ability to run multi-step API calls, with one step consisting of parallel/looping requests that is later merged/aggregated. It just so happens that the pagination functionality solves this problem, with the limitation of only being able to run 5 requests. For my use case the data returned is not a lot per request, and I would much rather have Grafana be responsible for the aggregation than having to also set up prometheus/database/whatever and add something additional I need to maintain.

[flux3all]

Yes, agreed, the requirement is looping not pagination. But Looping is included in this discussion titled "pagination" possibly because looping is categorized for implementation within the pagination feature. Look for this sub-header in the description at the top of this page:
Iterating URLs in a loop
"Apart from the regular pagination methods mentioned above, From infinity 2.0, you have the ability to iterate the URLs based on comma separated values."

[benayyad12]

Hello, I had the same problem I don’t know how I retrieve data for multiple workflow runs knowing that each one has a different id. I just know that variable can be put as part of URL using /$id_job as example. But How to loop over existing IDs of worlkflows no ideas. Good luck Abdessalam BENAYYAD

…

On Mon 10 Feb 2025 at 14:42, Paul Coada ***@***.***> wrote: I am using Grafana from a DevOps perspective: - I perform a request towards a build project to get a historical list of builds - Each build consists of parallel jobs. I want to visualize each job for each build, and I can only retrieve the job status by performing a request to that particular build Build Job 1 Job 2 # 1 ✅ ✅ # 2 ✅ ✅ ... ... ... Based on my personal need and from what I can read from other people's comments, it doesn't seem like *pagination* is the main need, but the ability to run multi-step API calls, with one step consisting of parallel/looping requests that is later merged/aggregated. It just so happens that the pagination functionality solves this problem, with the limitation of only being able to run 5 requests. For my use case the data returned is not a lot per request, and I would much rather have Grafana be responsible for the aggregation than having to also set up prometheus/database/whatever and add something additional I need to maintain. — Reply to this email directly, view it on GitHub <#601 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AS3UFM3TECSJASIPAOQXQ732PCUD3AVCNFSM6AAAAAAYM2NMZ2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMJRHE4DKNA> . You are receiving this because you commented.Message ID: <grafana/grafana-infinity-datasource/repo-discussions/601/comments/12119854 @github.com>

[FlightXCaptain]

This 5-page limit is severe making this product unusable. I have 13 pages of 100 in my data currently.

Using Loki connectors, I have 100s of thousands of lines of data going through hourly. yet this cannot even handle more than 5 pages.

Frustrating.

[yesoreyeram]

Hi @FlightXCaptain - Sorry for the delay reply and the frustartion you had.

Please check #601 (reply in thread)

[luboss]

I don't understand why a feature sabotages itself by this limitation to 5 requests in one shot. It doesn't make any sense as the developer of the dashboard is responsible for the utilization of the API endpoint and should do anything to prevent DDOS or resource overutilization which might still be actually possible for the dashboard user anyway.
It really makes this feature useless. 5 requests can be added even manually.

[yesoreyeram]

#601 (comment)

[Altair-Bueno]

Pagination should be a JavaScript/Lua/Starlark/anything scriptable so we can accommodate to whatever weird API we are dealing with. Hundreds of combinations from forms with little documentation isn't the way to go, IMHO.

And I fully support the others when they say that the feature sabotages itself. The current iteration of pagination isn't useful. As a final note, to the maintainer that keeps replying "you are using the wrong tool for the job" to every single one of us that complaints about the feature, they should take a look at Grafana and ask "what is Grafana even about" bc he is clearly lost on this one.

[yesoreyeram]

Hi Altair - Sorry for the delay replying. I understand the frustration you had.

But our stands are clear here. We can easily add support for pagination via javascript or any other scripting languages you add, but that will open up tons of security issues and concerns. We really would like to avoid that situation. backend is bit safer than the frontend.

Also I used to say you are using the wrong tool for the job only when someone trying to pull tons of data and try to parse/manipulate in grafana. Any small mistake done by one user can definitely affect all the other dashboards in the grafana. Also grafana is not originally designed as ETL tool. But slowly we are exploring the options to support these kind of loads. Even in other datasources, we have seen this issue where user try to query huge volume of data and cause slow loading of entire grafana dashboards and put heavy load on grafana server to parse those volume of data. So my advice is to use the query the aggregated data where possible. If the underlying API put you limitation on maximum results / pagniation you can pull, we believe the same reason applicable for grafana.

Also a gentle reminder that still the pagination feature is in beta. ( I know it is been an year. ). Due to the bandwidth and engineering efforts to make it secure and robust, it takes us some time.

Good luck.

[yesoreyeram]

To be clear, we are still exploring options to make the infinity pagination a better feature for everyone, you, me, grafana, our partners, clients and community.

and again sorry for the troubles you are having with infinity. Any constructive feedback and PRs welcome.

[yesoreyeram]

forms with little documentation

noted. Since this is beta and we want to explore with limited number of users to get feedback, we didn't spend much time in the documentation. But more documentation coming soon and for when the pagination goes GA from beta.

[VinayTalla20]

Hi @yesoreyeram , we are using offset pagination which is working absolutely fine, the response from our api is parsed with "Documents" key:

jsonData = {
"Documents": json.loads(DeserilizedJson)
}

but the results are displayed in a single row the paginatinated api call as below.

     required_data = db_collection.find({"lastUpdatedTime": {"$gte": start_time_obj, "$lte": end_time_obj}}).sort({ "lastUpdatedTime": -1}).skip(int(pagination_offset)).limit(int(pagination_limit))
     
  we observed that the response from an api call in visualized as a single row.   

previously before pagination our result used to be like below:

what we are missing here, can we achieve same for pagination thing

we aslo below issue:

Status: 500. Message: error while performing the infinity query. unable to merge fields due to different fields

[yesoreyeram]

Have you entered root selector? Some JSONata syntax like $.Documents or whatever your case needs.

[VinayTalla20]

we need to fetch MongoDB records from our existing python written api, where we are using skip and limit functions to fetch data from database using mongo driver.

the pagelimit is set to constant from grafan.ini config, can it be dynamic?

The records fetched from first api call returns array of json objects, my requireemnt is to show each json object in a new row. But we see is that is completed array is shown in row.

[yesoreyeram]

Have you entered root selector? Some JSONata syntax like $.Documents or whatever your case needs.

[VinayTalla20]

No, we did not used any Jsonata syntax

[yesoreyeram]

You should use JSONata in root selector as mentioned

[shahar-br]

Hello @yesoreyeram thanks for adding pagination!!
I'm looking to do some pagination for an api that supports up to 100 results per page. The pagination is done as a part of graphql query. I wasn't able to configure the pagination to work and the queries seems to always be sent with the same paramaters and not be overwritten with the pagination setup. Is this a supported scenario? Also is an override of the max pages available via an environment variable? Where can I find the docs to configure it? Thanks
See config here:

  "operationName": "GetEntities",
  "variables": {
    "filters": {
      "globalFilter": {
        "keyword": "",
        "entities": ["Alert"],
        "dateRange": {
          "from": "${__from:date:YYYY-MM-DD}",
          "to":   "${__to:date:YYYY-MM-DD}"
        }
      }
    },
    "pagination": {
      "from": 0,
      "size": 100
    },
    "sort": [ { "field": "createdOn", "order": "desc" } ]
  },
  "query": "query GetEntities($filters: SearchFilter, $pagination: PaginationSearchOffset, $sort: [Sort]) { searchV2(filters: $filters, pagination: $pagination, sort: $sort) { items { __typename ... on AlertSearch { id createdOn } } total: offset { total } } }"
}

[alecanteroinvgate]

Can you workaround this problem?
I'm with the same place :(

[shahar-br]

I couldn't find a workaround

[yesoreyeram]

#601 (reply in thread)

[danielsilvapereira]

Hi, I had the same problem, basically the pagination was separating my json into two different arrays, so I used a transformation combo to get around the problem

[yesoreyeram]

Just use root selector/JSONata query to make into rows. Then add pagination. That should work.

[ideasman69]

This is a great addition! Any chance you have some advice on how to use pagination with a data source like the below?

I can select a page number, and have a maximum of 100 for per_page:

The response includes "next_page", "page", "per_page", "pev_page", "total_pages" and "total_values" - with the actual data i want in a final "values" field:

Any advice or recommendations would be appreciated!

[yesoreyeram]

Hiya.. Thanks for reporting the issue you have. Unfortunately there was a bug. Fixed by #1284. With this bug, you can use tokens such as ${__pagination.foo} or ${pagination.page} or ${pagination.limit} in your URL body.

documents will be updated once the pagination feature is GA.

[kcsuden]

*Reposting from different PR thread for more visibility

Does any of this functionality work yet? I am using JSONata to try to grab the pagination cursor from the hubspot api like so

However, the pagination logic doesn't work. For Replace URL, the requests are being made, but they are clearly just using the original query, since all the subsequent outputs are the same.

For Query Param, the query parameter is being added into the content of the response, but the content of the response is not being used for the next query url