[EPIC] Support single use refresh tokens (connectors can update their configs during a sync)

[davinchia]

Tell us about the problem you're trying to solve

Today we do not handle the case where a new refresh token is returned when a new access token is granted.

This is because there doesn't exist a Protocol message that updates configs. This will probably involve adding a message similar to STATE but updates the config.

Describe the solution you’d like

We should provide a way to save a returned refresh token.

[santels]

To add to the context,

I'm currently working on a source connector that's implementing "short-lived access tokens and long-lived refresh tokens".

Based on the documentation of the source I'm working on, they return a new refresh_token after the renewal (of the access token) and they require the new refresh_token to be used in the succeeding token refresh calls.

From their docs:

Each time you perform a token refresh, you should save the new refresh token returned in the response

From this specification, I need to have the most updated refresh_token for the succeeding token refresh calls to be valid.

[avaidyanatha]

@sherifnada @davinchia

Currently, @santels is blocked on contributing their Xero connector because of this issue. How big of a change would this be to implement?

[davinchia]

To me, the task that takes more time here is extending the protocol. We have to make sure we do it correctly. Making the CDK modification should be a matter of prioritisation after. Do you agree @sherifnada ?

@avaidyanatha is there are preferred deadline for this or is it asap?

[wissevrowl]

I want to write two source connectors which both require the use of Authorization grant followed by rotating refresh tokens.

In the meantime I am thinking of using external storage (like azure blob or ftp) to store this azure token outside of the connector until this feature is complete. Or is there another way to more elegantly solve this?

[sherifnada]

@wissevrowl until we fix this issue that's probably your best bet

[akvadrako]

I have this problem with the Intuit Quickbooks source. The old refresh token is expired when a new one is granted (or used, possibly). The singer tap itself does handle it in a fashion ­– it overwrites the config.json file with the new token. However this doesn't work with the check command.

Is the protocol under discussion? It seems something simple like this would be sufficient?

{"type": "CONFIG", "config": {"refresh_token": "...", "access_token": "..."}}

[RickRen7575]

Has anyone come up with a way around this for now? @akvadrako ?

[akvadrako]

I spend a day on this and gave up. The singer tap could handle it, but the airbyte adapter would delete the updated config file right away, throwing away the refresh token. I tried to patch the tap and the connector, but there were plenty of issues with incompatible versions since none of the dependencies are versioned.

[RickRen7575]

@akvadrako so the solution was just to do this outside of Airbyte?

[akvadrako]

Yes, in the end we didn't use the airbyte connector.

[thomas-vl]

@sherifnada can you provide an ETA for this to become available? This is a serious blocker for a while now.

[RickRen7575]

Any update on this? How can we help?

[evantahler]

I happen to know that Hubspot has rather short-lived oAuth refresh tokens. Is what we do within that connector generalizable?

[evantahler]

@RickRen7575 this isn't yet prioritized... we will comment here when it is!

[RickRen7575]

Just checking in here!

[evantahler]

We are going to start looking into this in Q4! This is "Support single use refresh tokens" in our roadmap.

[supertopher]

I talked to @davinchia and @pmossman to add context on customers effected by this story
TCS believes this story will still be done this quarter.
turns out @evantahler 's team is on this one.. but we still are expecting this this quarter to the best of my knowledge

[RickRen7575]

That is so great to hear!

…

On Wed, Nov 9, 2022 at 2:44 PM Topher Lubaway ***@***.***> wrote: I talked to @davinchia <https://github.com/davinchia> and @pmossman <https://github.com/pmossman> to add context on customers effected by this story TCS believes this story will still be done this quarter. — Reply to this email directly, view it on GitHub <#3990 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCP57ST2QFIM7DWLTGZHTLWHP5KVANCNFSM46LXUMKA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- [image: photo] *Ricky Renner* Data Engineer, AMEND Consulting LLC (216) 906-5178 www.linkedin.com/in/ricky-renner

[evantahler]

Noting that this work will help troubleshoot:

• Bing Ads: https://github.com/airbytehq/oncall/issues/999
• Google Sheets
• Salesforce (maybe)

[RickRen7575]

So not Quickbooks Online?

…

On Wed, Nov 9, 2022 at 5:52 PM Evan Tahler ***@***.***> wrote: Noting that this work will help troubleshoot: - Bing Ads: airbytehq/oncall#999 <https://github.com/airbytehq/oncall/issues/999> - Google Sheets - Salesforce (maybe) — Reply to this email directly, view it on GitHub <#3990 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACCP57QH3B5YIEMOM5PEZODWHQTKVANCNFSM46LXUMKA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- [image: photo] *Ricky Renner* Data Engineer, AMEND Consulting LLC (216) 906-5178 www.linkedin.com/in/ricky-renner

[evantahler]

Noting that this work will help troubleshoot:

Those are additional issues we see that solving this will /also/ help!

[midavadim]

#8019
Source Typeform: support oauth:
ISSIE: #7814
PR: #8019
blocked since the connector uses single-use refresh token, which requires proper processing from platform side

[evantahler]

@pedroslopez I think that it is time to close this epic! The platform and CDK now fully support AirbyteControlMessages to allow connectors to change their configuration. Future work will be done in individual connectors.

[leo-schick]

@evantahler is that true as well for the low-code CDK? What authentication do I need to use there to support changing refresh tokens?

[girarda]

@leo-schick, this isn't available yet with the low-code CDK. You can track the issue here