[low code connectors] fix bug where headers were not passed to cursor interpolation

[brianjlai]

What

We were not correctly passing headers to the cursor value so when we tried to perform interpolation to get values from headers it would fail.

How

Pass the headers to interpolation like we do for the stop condition. I also added a little bit of code to make it easier to extract next urls in an indexable way. The existing implementation on requests.Response returns a giant string of links which would be a pain for developers to extract. Relying on the existing Response.links() helper method and storing it back onto the response is much more user friendly. +tests

Recommended reading order

1. cursor_pagination_strategy.py

[girarda]

looks good! Where do you think we should document this behavior?

[sherifnada]

Can you clarify how adding the pre-parsed headers field changes the interface for the developer? What’s the before and after?

[brianjlai]

Before:
headers.linkreturns a single string of all the links. For Greenhouse for example:

"<https://harvest.greenhouse.io/v1/applications?page=2&per_page=1>; rel="next",<https://harvest.greenhouse.io/v1/applications?page=8&per_page=1>; rel="last""

This is probably impossible for developers to parse without utility functions added in jinja for string manpulation and regexes. Basically they would need to replicate all the behavior of requests.Response.links() property method in their interpolated method.

After:
headers.linkretains a dictionary of urls indexed by keys. Usually next or last

YAML config:
cursor_value: "{{ headers.link.next.url }}"

[brianjlai]

@girarda
Thats a good question. I feel like maybe the best spot is to add a recipe to the cookbook saying that this is how you would perform cursor pagination when an API stores the next URL to paginate on in the headers. The response next url is probably more common but I bet more than just greenhouse does this type of thing

[sherifnada]

the best spot
would also recommend adding it to the different components inside a retriever e.g: paginators etc since that would be where a user consumes it.

[brianjlai]

@sherifnada that's a good point. Although I don't think the other pagination strategies need or use the response for anything. One area that I do think could get some value from receiving headers is in our error handlers. We do this for some error handling strategies, but I do think there is one scenario not covered in http_response_filter() where headers might be useful, I'll update to cover that as well.

[girarda]

should we update the "link" header here to be consistent with the cursor pagination strategy?