-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Property to limit test parallelisation #23870
Property to limit test parallelisation #23870
Conversation
… in order to reduce parallelization. It's done by creating a `gradle.properties` file in the connector root folder. Extra: removes test concatenation for performance tests.
Affected Connector ReportNOTE
|
Connector | Version | Changelog | Publish |
---|
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Destinations (1)
Connector | Version | Changelog | Publish |
---|---|---|---|
destination-snowflake |
0.4.49 |
✅ | ✅ |
- See "Actionable Items" below for how to resolve warnings and errors.
👀 Other Modules (1)
- base-normalization
Actionable Items
(click to expand)
Category | Status | Actionable Item |
---|---|---|
Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml ), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
/test connector=connectors/destination-snowflake
Build FailedTest summary info:
|
Merging as the errors in the |
* Adds new functionality on integration test pipeline and test pipeline in order to reduce parallelization. It's done by creating a `gradle.properties` file in the connector root folder. Extra: removes test concatenation for performance tests. * rename parameter * fix names * Reduce threads
* Adds new functionality on integration test pipeline and test pipeline in order to reduce parallelization. It's done by creating a `gradle.properties` file in the connector root folder. Extra: removes test concatenation for performance tests. * rename parameter * fix names * Reduce threads
What
After adding parallelisation in the test pipelines, some of the connectors started to fail (specially the ones that uses shared resource)
PR
How
Both
test
andintegrationTest
has been modified to include a property ingradle.properties
file in the connector folder so we keep parallelisation by default with maximum threads possible but at the same time we let connector owners to configure the number of threads.That is really useful for connectors that shares resources and are limited (like Redshift or Snowflake that are limited by number of connections).
To limit the parallelism, is it enough to create
gradle.properties
file and include this line on it:Where X is the desired parallelisation.
Also, this PR removes the concatenation of test for performance tests, so are not run multiple times.
Example with a
println
when I had set up 3 threads, now should show 2 threads (removed as we don't want to have prints ):Recommended reading order
No recommendation
🚨 User Impact 🚨
The users that have issues with the test should create a
gradle.properties
file with the limitation desired.