Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Attempt to bump python version to 3.11 #2417

Closed
wants to merge 77 commits into from
Closed

Attempt to bump python version to 3.11 #2417

wants to merge 77 commits into from

Commits on May 26, 2023

  1. update CI 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    8265db7 View commit details
    Browse the repository at this point in the history

  2. Release all the upper bounds 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    02ea106 View commit details
    Browse the repository at this point in the history

  3. bump more versions 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    c2d1bd5 View commit details
    Browse the repository at this point in the history

  4. update requirements 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    29e5cbf View commit details
    Browse the repository at this point in the history

  5. update requirements 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    34ce2bf View commit details
    Browse the repository at this point in the history

  6. add dependency 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    249fded View commit details
    Browse the repository at this point in the history

  7. update scipy 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    4e03227 View commit details
    Browse the repository at this point in the history

  8. fix typo 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    121a1be View commit details
    Browse the repository at this point in the history

  9. update 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    6aa27b0 View commit details
    Browse the repository at this point in the history

  10. revert all changes 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    6318f25 View commit details
    Browse the repository at this point in the history

  11. bump holoviews 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    aa53b24 View commit details
    Browse the repository at this point in the history

  12. release pyarrow 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    2ffb41c View commit details
    Browse the repository at this point in the history

  13. relax scipy 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    cca35a2 View commit details
    Browse the repository at this point in the history

  14. bump tensorflow 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    483cf0c View commit details
    Browse the repository at this point in the history

  15. give up tensorflow, it's not supported yet 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    11f4db5 View commit details
    Browse the repository at this point in the history

  16. bump lower bound of pyarrow as it caused build error 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    ed19a95 View commit details
    Browse the repository at this point in the history

  17. bump pyarrow lower bound 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    4b5ca3d View commit details
    Browse the repository at this point in the history

  18. bump spark version 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    14896f1 View commit details
    Browse the repository at this point in the history

  19. skip Spark install since it's not available in Python3.11 yet 

    Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
    @noklam @astrojuanlu
    noklam authored and astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    98919b9 View commit details
    Browse the repository at this point in the history

  20. Revert "skip Spark install since it's not available in Python3.11 yet" 

    This reverts commit 548965a.

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
    @astrojuanlu
    astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    b565b48 View commit details
    Browse the repository at this point in the history

  21. Revert "give up tensorflow, it's not supported yet" 

    This reverts commit 070cc7c.

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
    @astrojuanlu
    astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    b78525d View commit details
    Browse the repository at this point in the history

  22. Restore Python 3.11 in CI after sloppy rebase 

    Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
    @astrojuanlu
    astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    8c43052 View commit details
    Browse the repository at this point in the history

  23. Relax delta-spark upper bound 

    Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
    @astrojuanlu
    astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    07c03fb View commit details
    Browse the repository at this point in the history

  24. Relax scikit-learn upper bound 

    Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
    @astrojuanlu
    astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    ce6e4ab View commit details
    Browse the repository at this point in the history

  25. Update test_requirements.txt 

    Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
    @astrojuanlu
    astrojuanlu committed May 26, 2023
    Configuration menu
    Copy the full SHA
    ac21a73 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2023

  1. Merge branch 'main' into bump/python311

    @noklam
    noklam authored Jun 20, 2023
    Configuration menu
    Copy the full SHA
    b804451 View commit details
    Browse the repository at this point in the history

  2. Add numpy to avoid pyarrow installing invalid version 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    d0a8938 View commit details
    Browse the repository at this point in the history

  3. bump pyarrow pin 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    1afe08d View commit details
    Browse the repository at this point in the history

  4. remove pyarrow pin 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    af195f7 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2023

  1. Merge branch 'main' into bump/python311

    @SajidAlamQB
    SajidAlamQB authored Jun 22, 2023
    Configuration menu
    Copy the full SHA
    efac36c View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2023

  1. bump pyarrow to avoid numpy build failure 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    321e38a View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2023

  1. Merge branch 'main' into bump/python311

    @noklam
    noklam authored Jun 28, 2023
    Configuration menu
    Copy the full SHA
    8258a6a View commit details
    Browse the repository at this point in the history

  2. lint 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    2a3e094 View commit details
    Browse the repository at this point in the history

  3. fix linting, desc is deprecated for spark>=3.4 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    590ee10 View commit details
    Browse the repository at this point in the history

  4. bump ==3.10 -> >=3.10 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    64b130b View commit details
    Browse the repository at this point in the history

  5. consistent use of ' and " 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    65a1cd5 View commit details
    Browse the repository at this point in the history

  6. Pyspark breaks since 3.4 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    85fdba2 View commit details
    Browse the repository at this point in the history

  7. revert delta-spark, 2.0 break tests 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    677c29e View commit details
    Browse the repository at this point in the history

  8. update pyproject.toml to allow 3.11 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    6103bfb View commit details
    Browse the repository at this point in the history

  9. update pyproject python version 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    4437be6 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'main' into bump/python311

    @noklam
    noklam authored Jun 28, 2023
    Configuration menu
    Copy the full SHA
    db7cdb4 View commit details
    Browse the repository at this point in the history

  11. Python 3.11 need Pyspark 3.4 because of pickle error 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    fcbdde2 View commit details
    Browse the repository at this point in the history

  12. swithc docker to Py3.11 for testing pyspark issue 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    68bc69a View commit details
    Browse the repository at this point in the history

  13. Fix typo 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    d7cd2aa View commit details
    Browse the repository at this point in the history

  14. fix dockerfile 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    66a47dc View commit details
    Browse the repository at this point in the history

  15. comment out unnecessary dependency 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    9257968 View commit details
    Browse the repository at this point in the history

  16. comment more dependencies that cause error.. 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    30b6a13 View commit details
    Browse the repository at this point in the history

  17. add linting 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    f9a961f View commit details
    Browse the repository at this point in the history

  18. relax pandas-gbq pin for py311 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    869b3d1 View commit details
    Browse the repository at this point in the history

  19. mock importlib need to move to the end python/cpython#98771 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    754264c View commit details
    Browse the repository at this point in the history

  20. Fix caplog didn't capture the INFO level log, pytest default change 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    9900796 View commit details
    Browse the repository at this point in the history

  21. Fix test with missing credentials 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    b2724c7 View commit details
    Browse the repository at this point in the history

  22. fix import module order for tests 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    5e8b1ed View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2023

  1. Fix more CLI tests 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    2b1f92b View commit details
    Browse the repository at this point in the history

  2. test moto 4.1.12 

    Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
    @SajidAlamQB
    SajidAlamQB committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    5d9f026 View commit details
    Browse the repository at this point in the history

  3. make moto fixed for 3.10 

    Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
    @SajidAlamQB
    SajidAlamQB committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    d855426 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2023

  1. fix spark compatibility issue 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    bc6f09c View commit details
    Browse the repository at this point in the history

  2. Fix Spark error handling in exists() so it will be compatible with py… 

    …spark3.4

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    3843caf View commit details
    Browse the repository at this point in the history

  3. make delta compatible in the same fashion as spark 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    da40d8c View commit details
    Browse the repository at this point in the history

  4. refactor the test to pass coverage 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    ba47ba7 View commit details
    Browse the repository at this point in the history

  5. fix pyspark test 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    3997aa8 View commit details
    Browse the repository at this point in the history

  6. fix test that are too strict 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    7806129 View commit details
    Browse the repository at this point in the history

  7. fix delta 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    66d3966 View commit details
    Browse the repository at this point in the history

  8. refactor jdbc dataset tests with pytest and bug fix 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    761d008 View commit details
    Browse the repository at this point in the history

  9. fix issues with spark test for different version, introduce semver 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    e8479db View commit details
    Browse the repository at this point in the history

  10. Merge branch 'main' into bump/python311

    @noklam
    noklam authored Jul 3, 2023
    Configuration menu
    Copy the full SHA
    3e86cd1 View commit details
    Browse the repository at this point in the history

  11. fix linter 

    Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
    @noklam
    noklam committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    bdea52d View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2023

  1. Merge branch 'main' into bump/python311

    @SajidAlamQB
    SajidAlamQB authored Jul 14, 2023
    Configuration menu
    Copy the full SHA
    e622c8c View commit details
    Browse the repository at this point in the history

  2. add test for covering getOrCreate 

    Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
    @SajidAlamQB
    SajidAlamQB committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    18eaaed View commit details
    Browse the repository at this point in the history

  3. more specific with test 

    Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
    @SajidAlamQB
    SajidAlamQB committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    7af8483 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2023

  1. add 3.11 support to metadata 

    Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
    @SajidAlamQB
    SajidAlamQB committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    009742a View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into bump/python311

    @SajidAlamQB
    SajidAlamQB authored Jul 17, 2023
    Configuration menu
    Copy the full SHA
    f4e0839 View commit details
    Browse the repository at this point in the history

  3. check getOrCreate() method is not None 

    Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
    @SajidAlamQB
    SajidAlamQB committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    3435950 View commit details
    Browse the repository at this point in the history

  4. Update deltatable_dataset.py 

    Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
    @SajidAlamQB
    SajidAlamQB committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    82b80fc View commit details
    Browse the repository at this point in the history

  5. revert deltatable change 

    Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
    @SajidAlamQB
    SajidAlamQB committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    8d51f8b View commit details
    Browse the repository at this point in the history

  6. Update test_spark_jdbc_dataset.py 

    Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
    @SajidAlamQB
    SajidAlamQB committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    7ccad55 View commit details
    Browse the repository at this point in the history

  7. fix deltatable 

    Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
    @SajidAlamQB
    SajidAlamQB committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    70f12be View commit details
    Browse the repository at this point in the history