snowflake-destination: allow buffer to be increased till 50

[subodh1810]

See this slack thread https://airbytehq-team.slack.com/archives/C046KAVD4NR/p1680158784059009

[subodh1810]

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/4565103407
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/4565103407
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1461    632    57%

Build Passed

Test summary info:

All Passed

[github-actions]

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

• Run integration tests
• Bump connector or module version
• Add changelog
• Publish the new version

✅ Sources (0)

Connector	Version	Changelog	Publish

• See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (1)

Connector	Version	Changelog	Publish
destination-snowflake	0.4.60	✅	✅

• See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

• base-normalization

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version.

[edgao]

we should probably prioritize decoupling buffer flushes from checkpoints at some point, but this is a good workaround for now

[ryankfu]

Non-blocking but maybe a ticket should be filed to add this warning as well here

airbyte/airbyte-integrations/connectors/destination-redshift/src/main/java/io/airbyte/integrations/destination/redshift/RedshiftStagingS3Destination.java

Lines 146 to 152 in 0b156a8

	if (numberOfFileBuffers > FileBuffer.SOFT_CAP_CONCURRENT_STREAM_IN_BUFFER) {
	LOGGER.warn("""
	Increasing the number of file buffers past {} can lead to increased performance but
	leads to increased memory usage. If the number of file buffers exceeds the number
	of streams {} this will create more buffers than necessary, leading to nonexistent gains
	""", FileBuffer.SOFT_CAP_CONCURRENT_STREAM_IN_BUFFER, catalog.getStreams().size());
	}

It's to safeguard users from running into OOM issues by logging the the increased memory size that comes with more FileBuffers

[ryankfu]

LGTM, non-blocking comment about a LOGGER.warn and maybe add a test but this can be separate

[subodh1810]

/publish connector=connectors/destination-snowflake

🕑 Publishing the following connectors:
connectors/destination-snowflake
https://github.com/airbytehq/airbyte/actions/runs/4567631522

Connector	Did it publish?	Were definitions generated?
connectors/destination-snowflake	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[dimoschi]

How can we use that feature to increase the buffer size?

[subodh1810]

@dimoschi upgrade snowflake destination to the latest version. Once done go to the snowflake connector setup page and in the UI you will see this option, you can increase it till 50.

[dimoschi]

@dimoschi upgrade snowflake destination to the latest version. Once done go to the snowflake connector setup page and in the UI you will see this option, you can increase it till 50.

Thanks