SAT: optionally disable discovered catalog caching

[alafanechere]

What

I think the ongoing build failure of the source-tiktok-marketing connectors are due to a default caching of discovered catalog in SAT.

This connector catalog is dynamic according to connector configurations. Multiple acceptance tests are run with different configurations hence different catalogs. But some tests fail because they try to retrieve a stream defined in the configured catalog absent from the cached discovered catalog.

After investigation, there is nothing badly configured on TikTok's acceptance test config.
I figured a catalog discovered in the previous tests was reused and caused the error.

How

• Expose a new configuration field on SAT to disable discovered catalog caching.

[alafanechere]

/test connector=connectors/source-tiktok-marketing

🕑 connectors/source-tiktok-marketing https://github.com/airbytehq/airbyte/actions/runs/3549411368
✅ connectors/source-tiktok-marketing https://github.com/airbytehq/airbyte/actions/runs/3549411368
Python tests coverage:

Name                                  Stmts   Miss  Cover
---------------------------------------------------------
source_tiktok_marketing/__init__.py       2      0   100%
source_tiktok_marketing/source.py        61      6    90%
source_tiktok_marketing/streams.py      352     37    89%
---------------------------------------------------------
TOTAL                                   415     43    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     199     93    53%   35, 41-43, 48, 53, 59, 65, 71, 77-79, 98, 103-105, 111-113, 119-120, 125-126, 131, 137, 146-155, 161-166, 181, 205, 236, 242, 250-255, 263-268, 276-289, 294-300, 307-318, 325-341
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1567    350    78%

Build Passed

Test summary info:

All Passed

[github-actions]

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

• Run integration tests
• Bump connector or module version
• Add changelog
• Publish the new version

❌ Sources (64)

Connector	Version	Changelog	Publish
source-airtable	0.1.3	✅	✅
source-amazon-ads	0.1.25	✅	✅
source-amazon-seller-partner	0.2.28	✅	✅
source-amazon-sqs	0.1.0	✅	✅
source-amplitude	0.1.17	✅	✅
source-appsflyer	0.1.0	⚠ (doc not found)	⚠ (not in seed)
source-asana	0.1.5	✅	✅
source-azure-table	0.1.3	✅	✅
source-braintree	0.1.3	✅	✅
source-cart	0.2.0	❌ (changelog missing)	✅
source-chargebee	0.1.16	✅	❌ (diff seed version)
source-commercetools	0.1.0	✅	✅
source-confluence	0.1.1	✅	✅
source-datadog	0.1.0	✅	✅
source-delighted	0.1.4	✅	✅
source-drift	0.2.5	✅	✅
source-facebook-marketing	0.2.74	✅	✅
source-facebook-pages	0.1.6	✅	✅
source-freshcaller	0.1.0	✅	✅
source-freshsales	0.1.2	✅	✅
source-freshservice	0.1.1	✅	✅
source-github	0.3.8	✅	✅
source-gitlab	0.1.6	✅	✅
source-google-ads	0.2.5	✅	✅
source-google-search-console	0.1.18	✅	✅
source-greenhouse	0.3.0	✅	✅
source-harvest	0.1.11	✅	✅
source-instagram	1.0.0	✅	✅
source-iterable	0.1.21	✅	✅
source-klaviyo	0.1.10	✅	✅
source-lemlist	0.1.1	❌ (changelog missing)	❌ (diff seed version)
source-lever-hiring	0.1.3	✅	✅
source-linnworks	0.1.5	❌ (changelog missing)	✅
source-mailchimp	0.2.15	✅	✅
source-mailgun	0.1.0	⚠ (doc not found)	✅
source-monday	0.1.4	✅	✅
source-notion	0.1.10	✅	✅
source-okta	0.1.13	✅	✅
source-onesignal	0.1.2	✅	✅
source-openweather	0.1.6	✅	✅
source-outreach	0.1.2	✅	✅
source-pardot	0.1.0	⚠ (doc not found)	⚠ (not in seed)
source-paystack	0.1.1	✅	✅
source-pinterest	0.1.9	✅	✅
source-pipedrive	0.1.13	✅	✅
source-plaid	0.3.2	❌ (changelog missing)	✅
source-posthog	0.1.7	✅	✅
source-prestashop	0.3.0	⚠ (doc not found)	✅
source-quickbooks-singer	0.1.5	⚠ (doc not found)	✅
source-recharge	0.2.4	❌ (changelog missing)	✅
source-retently	0.1.2	✅	✅
source-salesforce	1.0.27	✅	✅
source-salesloft	0.1.3	✅	✅
source-sendgrid	0.2.16	✅	✅
source-sentry	0.1.7	✅	✅
source-strava	0.1.2	✅	✅
source-surveymonkey	0.1.13	✅	✅
source-tplcentral	0.1.1	✅	⚠ (not in seed)
source-twilio	0.1.13	✅	✅
source-weatherstack	0.1.0	✅	✅
source-youtube-analytics	0.1.3	✅	✅
source-zendesk-sunshine	0.1.1	✅	✅
source-zendesk-talk	0.1.5	✅	✅
source-zenloop	0.1.3	✅	✅

• See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

• See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version.

[evantahler]

Should the default be false to prevent these kind of errors? What's the slow-down introduced if we don't cache?

[alafanechere]

If we don't cache the discovered catalog, every test using it will trigger a discover command run as the scope of the discovered_catalog fixture is function by default. As this caching feature did not bother us until now, a specific edge case, I would suggest keeping it as a default.
We could try to make it smarter and leverage another fixture scope for more granular caching (e.g. class scope).

[evantahler]

👍

[pedroslopez]

Makes sense!

If the issue is that the cache is different per config, could there be a way to cache per-config rather than globally? This way it wouldn't have to be a user-defined thing they have to figure out how to set and the SAT would just be a bit smarter about how it caches the schemas.

[alafanechere]

+1 @pedroslopez . I created an issue to track this improvement: https://github.com/airbytehq/airbyte-internal-issues/issues/1141

[alafanechere]

/publish connector=bases/source-acceptance-test auto-bump-version=false

🕑 Publishing the following connectors:
bases/source-acceptance-test
https://github.com/airbytehq/airbyte/actions/runs/3574012824

Connector	Did it publish?	Were definitions generated?
bases/source-acceptance-test	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️