Destination-redshift: fixed integration tests - added tmp tables removal

[etsybaev]

What

Normalization integration tests create target schemas at destinations and never delete them. This behavior pollutes our DW accounts (Redshift, Snowflake, BigQuery) with over a thousand random schemas (integration_test_***). In case of Redshift, this causes the following problem:

 ***"type":"TRACE","trace":***"type":"ERROR","emitted_at":1.666222038961E12,"error":***"message":"Something went wrong in the connector. See the logs for more details.","internal_message":"java.sql.SQLException: [Amazon](500310) Invalid operation: 1040 \nDetails: Maximum tables limit exceeded. The maximum number of tables per cluster is 9900 for this instance type. The limit includes permanent and temporary tables. (pid:7865);","stack_trace":"java.sql.SQLException: [Amazon](500310) Invalid operation: 1040 \nDetails: Maximum tables limit exceeded. The maximum number of tables per cluster is 9900 for this instance type. The limit includes permanent and temporary tables. (pid:7865);\n\tat com.amazon.redshift.client.messages.inbound.ErrorResponse.toErrorException(Unknown Source)\n\tat com.amazon.redshift.client.PGMessagingContext.handleErrorResponse(Unknown Source)\n\tat com.amazon.redshift.client.PGMessagingContext.handleMessage(Unknown Source)\n\tat com.amazon.jdbc.communications.InboundMessagesPipeline.getNextMessageOfClass(Unknown Source)\n\tat com.amazon.redshift.client.PGMessagingContext.doMoveToNextClass(Unknown Source)\n\tat com.amazon.redshift.client.PGMessagingContext.getErrorResponse(Unknown Source)\n\tat com.amazon.redshift.client.PGClient.handleErrorsScenario3(Unknown Source)\n\tat com.amazon.redshift.client.PGClient.handleErrors(Unknown Source)\n\tat com.amazon.redshift.dataengine.PGQueryExecutor$CallableExecuteTask.call(Unknown Source)\n\tat com.amazon.redshift.dataengine.PGQueryExecutor$CallableExecuteTask.call(Unknown Source)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\nCaused by: com.amazon.support.exceptions.ErrorException: [Amazon](500310) Invalid operation: 1040 \nDetails: Maximum tables limit exceeded. The maximum number of tables per cluster is 9900 for this instance type. The limit includes permanent and temporary tables. (pid:7865);\n\t... 13 more\n","failure_type":"system_error"***

How

Added tmp data removing

🚨 User Impact 🚨

Only tests affected, no user impact

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub by running the /publish command described here
• After the connector is published, connector added to connector index as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed

[etsybaev]

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/4115534800
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/4115534800
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed