Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/setup to pyproject #24

Merged
merged 6 commits into from
Jul 8, 2024
Merged

Conversation

s7clarke10
Copy link
Owner

@s7clarke10 s7clarke10 commented Jul 8, 2024

Description of change

A major update to build and run this framework to be pep8 compliant as much as possible and to modernise the testing framework.

Changes:

  1. Moving from setup.py to pyproject.toml and poetry for installation
  2. Introducing tox for platform independent testing of pep8 compliance.
  3. Introducing black for formatting
  4. Introducing isort for compliant and order includes
  5. Introducing flake8 for linting alongside pylint
  6. Introducing mypy for type checking
  7. Updating github actions to run tox tests rather than directly running pytests.
  8. Code formatting updates and changes to pass / met the standards above.
  9. Deprecating python 3.7

Manual QA steps

  • Ensuring all pytests pass
  • Ensuring all introduced tox python pep compliant tests pass
  • Running against tap-oracle which uses this variant of the singer-python framework
  • Ensure performance is good from ORJSON -> MSGSPEC

Performance Results:

  • Table A: 455.4k rows, 22.3 MB

  • Table B: 848.9k rows, 31.1 MB

  • MSGSPEC with Singer Decimal on runtime:

    Start Time 2024-07-08T05:06:48
    End Time 2024-07-08T05:13:27

    Total Time: 6:39 (min/secs) - lag 1:19 (min/sec) before ingesting really kicks in.

  • ORJSON with Singer Decimal on runtime:

    Start Time 2024-07-08T05:32:10
    End Time 2024-07-08T05:39:40

    Total Time: 7:30 (min/secs) - lag 1:34 (min/sec) before ingesting really kicks in.

  • MSGSPEC with Singer Decimal off runtime:

    Start Time 2024-07-08T05:47:19
    End Time 2024-07-08T05:53:53

    Total Time: 6:34 (min/secs) - lag 0:51 (min/sec) before ingesting really kicks in.

  • ORJSON with Singer Decimal off runtime:

    Start Time 2024-07-08T05:55:48
    End Time 2024-07-08T06:03:18

    Total Time: 7:30 (min/secs) - lag 1:13 (min/sec) before ingesting really kicks in.

  • Overall performance gain running from tap-oracle to target-snowflake (real world scenario). Close to one minute.

Risks

  • The move away from setup.py to pyproject.toml is not compliant with taps.

Rollback steps

  • revert this branch

Copy link

github-actions bot commented Jul 8, 2024

Code Coverage

Package Line Rate Branch Rate Health
. 78% 68%
Summary 78% (727 / 931) 68% (236 / 346)

Minimum allowed line rate is 60%

@s7clarke10 s7clarke10 merged commit 59efc29 into master Jul 8, 2024
8 checks passed
@s7clarke10 s7clarke10 deleted the feature/setup_to_pyproject branch July 8, 2024 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant