fix(connector-builder): add flag to disable cache

[lazebnyi]

What

disable caching when running from the builder

How

add the disable_cache option to the ModelToComponentFactory and pass this value during the initialization of the requester

Review guide

1. model_to_component_factory.py
2. connector_builder_handler.py

User Impact

No

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 9, 2024 1:09pm

[maxi297]

I think this can be fine as a temporary fix but it will change the behavior of connectors between the Connector Builder and an actual run. This is an example where there is a cache issue and disabling the cache would make the Connector Builder have the right behavior for the customer during test reads but the sync would have missing records.

Would there be an easy way to call HttpClient.clear_cache when this is a test read instead?

[bnchrch]

This seems straightforward to me. Will defer to @girarda and @maxi297 if correct.

[maxi297]

I'm fine with this as a temporary fix. The longer term fix would be https://github.com/airbytehq/airbyte-internal-issues/issues/9650

[maxi297]

@lazebnyi Before merging, can we validate these cases? I fear we might still have caches even with your change

[lazebnyi]

@maxi297 we created the stream component with the updated stream configuration you mentioned , but we disabled caching right before creating the instance (line 836 - model_to_component_factory.py)
so, from my perspective, this approach seems fine

[maxi297]

This approach also seems fine to me but I would feel safer with a test to confirm that substreams are also affected by disabling the cache. This will also serve as a good safety net against regression

[lazebnyi]

/approve-regression-tests

Check job output.

✅ Approving regression tests

[lazebnyi]

local e2e tests passed successfully