Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
A major update to build and run this framework to be pep8 compliant as much as possible and to modernise the testing framework.
Changes:
Manual QA steps
Performance Results:
Table A: 455.4k rows, 22.3 MB
Table B: 848.9k rows, 31.1 MB
MSGSPEC with Singer Decimal on runtime:
Start Time 2024-07-08T05:06:48
End Time 2024-07-08T05:13:27
Total Time: 6:39 (min/secs) - lag 1:19 (min/sec) before ingesting really kicks in.
ORJSON with Singer Decimal on runtime:
Start Time 2024-07-08T05:32:10
End Time 2024-07-08T05:39:40
Total Time: 7:30 (min/secs) - lag 1:34 (min/sec) before ingesting really kicks in.
MSGSPEC with Singer Decimal off runtime:
Start Time 2024-07-08T05:47:19
End Time 2024-07-08T05:53:53
Total Time: 6:34 (min/secs) - lag 0:51 (min/sec) before ingesting really kicks in.
ORJSON with Singer Decimal off runtime:
Start Time 2024-07-08T05:55:48
End Time 2024-07-08T06:03:18
Total Time: 7:30 (min/secs) - lag 1:13 (min/sec) before ingesting really kicks in.
Overall performance gain running from tap-oracle to target-snowflake (real world scenario). Close to one minute.
Risks
Rollback steps