Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

java CDK, source-postgres: refactor tests to spin up less testcontainers #31818

Closed
postamar opened this issue Oct 25, 2023 · 0 comments · Fixed by #32314
Closed

java CDK, source-postgres: refactor tests to spin up less testcontainers #31818

postamar opened this issue Oct 25, 2023 · 0 comments · Fixed by #32314
Assignees
@postamar
Labels
team/db-dw-sources Backlog for Database and Data Warehouse Sources team

Comments

@postamar
Copy link
Contributor

Topic

No response

Relevant information

As evidenced in the recent profiling work I did on source-postgres (see #31298 (comment) ) a lot of the time is spent spinning up testcontainers.

The problem is multifold:

  • the tests are grouped and run in classes which represent a specific mode of operation for the connector (usually reflected in the form of different connector configs). This is sensible, but for good performance we need to group them in terms of which database resource they need.
  • some testcontainers (looking at you, ssh bastion and PostgresUtils.getCertificates) are effectively rebuilt from scratch every time and would be better off packaged as a docker image on Docker Hub as part of the CDK build.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@postamar postamar

Labels
team/db-dw-sources Backlog for Database and Data Warehouse Sources team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

java CDK, source-postgres: parallelize tests airbytehq/airbyte
1 participant
@postamar