Result paging

[jorgebay]

The native protocol v2 allows for paging the result of queries. For that, the QUERY and EXECUTE messages have a page size value.

[DanielSmedegaardBuus]

This would be so awesome :)

[salex89]

What is the status of this feature? Does node-cassandra-cql support automatic paging in a similar way like the Java driver or not?

[jorgebay]

It does not, yet. You can follow the status on this issue.
I implemented most of the functionality but it is not ready.

[sarus]

👍 On this feature. Out of curiosity will this allow client (browser) side paging.

If a user via a web browser is paging through a result set will this keep the connection open for the duration of the browser session or can the connection be returned to the pool without losing the current "page".

Thank you for the great library so far. Has been working great so far.

[jorgebay]

@sarus Thanks! Result paging in Cassandra works with "state tokens". You get the paging state, you can store it and then request more rows for the same query and the provided.

This can later be abstracted to large result set / stream / whatever that keeps querying for new data when you consumed all the rows on the previous page.

[sarus]

That sounds great. I'm not a Cassandra expert by any means but if there's anything I can do to help with the paging (even if it's just testing and writing tests) please let me know. It's definitely a feature we're really looking forward to having.

[salex89]

I'm not sure that Cassandra result set paging is going to relieve you massively from the client side paging logic. You'll have to do your client paging still and maybe send individual queries for every page because one page can be observed for indefinitely long and one query (paged or not) gets all of the results ready (correct me if I'm wrong). So one open connection is not that optimal, I would say. I believe Cassandra paging is mostly applicable for huge data sets which should be streamed completely consumed and not throw a memory exception in the process.

I'm better in Java than in node.js so I maybe influenced with my research there but this is what I concluded, because I had a big data set streaming problem when first commented here.

P.S I would like to take part in development o node-cassandra-cql as soon as I have the time, and get a bit more familiar with the binary interface since I never used it.

[sarus]

@salex89 Your point is what I was thinking as well. At first when I read the release notes on the pagination I immediately thought "Perfect for user paging via a browser!". When looking at how you actually use it in Java though it seemed like it was meant more for sending huge amounts of data back without running out of memory as you mentioned.

@jorgebay Once you have the paging state stored can you close that connection and then issue a query for the next page of data at an undetermined time later? If so, how does Cassandra clean up the paging state that it must be saving somewhere? How does it know when it no longer needs that state information?

Thanks!