Increase Database Source SELECT Batch Size

[evantahler]

#12400 added dynamic batch sizes to our SQL sources to prevent OOM problems. However, I believe that we are currently creating batch sizes that are too small for optimal speed. This is validated in 2 ways:

• We have shown that Airbyte's speed increases with wider rows
• We are not seeing OOM errors on our database sources any more, and we are not utilizing more than 60% of available memory in our source containers

This PR does 2 things:

1. Bumps the total allocation of the source's available memory for the JDBC buffer from 50% to 60%
2. Rather than assuming the worst-case UTF8 byte-size for RECORD strings (4-byte/char), we move to 3-byte/char which is likely more common - assuming most chars fit in the ASCII space.

[evantahler]

@edgao I'm network-bound at home testing this out... what's the process for getting a connector tested on cloud or a dev server?

[edgao]

I think roughly this? https://internal-docs.airbyte.io/Things-To-Know/Cloud-Development-Environments - you'd need to push your connector image to dockerhub under some special tag + update the cloud connector version mask to point at that tag

but probably worth checking with the cloud folks for more details, I actually haven't done this before either :/

[davinchia]

curious, why not go higher?

if primary memory usage is due to this buffer, I think we can go as high as 80%. Usual java overhead is ~~10% so a 20% buffer seems good enough.

[evantahler]

I think we can keep ramping this up... but in an environment where I can't robustly test this, I want to go step-by-step

[evantahler]

I've done some local testing and thing seem ok... @edgao / @davinchia please let me know what I should do to be confident that this is safe to merge in

[github-actions]

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

• Run integration tests
• Bump connector or module version
• Add changelog
• Publish the new version

⚠ Sources (5)

Connector	Version	Changelog	Publish
source-alloydb	1.0.17	✅	✅
source-alloydb-strict-encrypt	1.0.17	✅	⚠ (not in seed)
source-mysql	1.0.14	✅	✅
source-mysql-strict-encrypt	1.0.14	✅	⚠ (not in seed)
source-postgres-strict-encrypt	1.0.28	✅	⚠ (not in seed)

• See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

• See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version.

[evantahler]

/publish connector=connectors/source-mysql

🕑 Publishing the following connectors:
connectors/source-mysql
https://github.com/airbytehq/airbyte/actions/runs/3568069054

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[evantahler]

/publish connector=connectors/source-mysql-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-mysql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/3568069503

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[evantahler]

/publish connector=connectors/source-postgres

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/3568194289

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

[evantahler]

/publish connector=connectors/source-postgres-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/3568196333

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️