"query_devices_by_filter_scroll" method

[halloffame0793]

Hi All, I am currently using the "query_devices_by_filter_scroll" method to pull in a very large amount of hosts. In the API and falconPy docs I am seeing that this method mainly functions off a large alphanumeric offset pointer which expires after 2 minutes.

What is unclear to me is what is actually expiring. While using this method I am seeing the same offset pointer being used but in the json response I can see that the "expires_at" epoch time value continuing to be updated per batch of machines that is being returned. I have also observed my query continues to return batches of machines for much longer than 2 minutes.

With that being said - I would like some clarity on if the offset pointer is supposed to expire after 2 minutes or is there a 2 minute time limit per batch or machines. Any assistance would be appreciated . Thanks

[jshcodes]

Hi @halloffame0793 -

My understanding is this expiration is referring to the offset pointer, and not the specific recordset batch.

Thank you for the question! 😄

[halloffame0793]

Thanks @jshcodes, that was what I thought too but I'm not sure how its possible that while looping though the same offset pointer is returning batches of hosts for 4-5 minutes.

Please keep in mind I am returning 100,000s to a million host records in 1 query.

[jshcodes]

Does your loop take longer to consume the 5000 records returned than the two minutes before the pointer expires? If not, I think you're OK as the batch would be calculated after the first query processed, and then it's just taking your position and calculating the next. (Or are you seeing errors before you move on to the next iteration?)

[halloffame0793]

@jshcodes not sure if I completely understand your question but batches of 5000 machines are being returned every second or so due to while loop I am creating. Each time the query runs in the while loop the offset value is passed to it for pagination purposes. This query runs for much longer than the 2 minute expiration decribed in the docs. Where the errors would occur is when the offset pointer expires as then you could not pass it into the next interation of the query.

[jshcodes]

I think I see what you're saying.

From what I've been told it expires if you don't use it at all for 2 minutes (this part should be easy to prove out), so if you make another request, it's "refreshed" and would be giving you close to the full two minutes on it's return. So as long as your loop is making requests, you should be getting back another batch. I need to recreate this to confirm though, it's an interesting question. I can definitely simulate the two minute delay (or four / five in this case) in a loop, but am not sure when exactly the two minute timer starts for the returning token. (This part I need to think about a way to simulate.)

[halloffame0793]

Thanks! Let me know what you find. I would be very interesting if you have different results than I did.