Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Attempt to bump python version to 3.11 #2417

Closed
wants to merge 77 commits into from
Closed

Attempt to bump python version to 3.11 #2417

wants to merge 77 commits into from

Conversation

noklam
Copy link
Contributor

@noklam noklam commented Mar 13, 2023 •
edited
Loading

Signed-off-by: Nok Chan nok.lam.chan@quantumblack.com

NOTE: Kedro datasets are moving from kedro.extras.datasets to a separate kedro-datasets package in
kedro-plugins repository. Any changes to the dataset implementations
should be done by opening a pull request in that repository.

Description

See what's failing in CI and estimate the effort of actually supporting newer version.

Investigation of #2048 #2270

Development notes

Checklist

  • Read the contributing guidelines
  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added a description of this change in the RELEASE.md file
  • Added tests to cover my changes

@noklam noklam changed the title Attempt to bump python version Attempt to bump python version to 3.11 Mar 17, 2023
@noklam
Copy link
Contributor Author

noklam commented Mar 21, 2023

Note:
I need to manually trigger a "Nightly build" on CI to build Python images for 3.11

@noklam noklam mentioned this pull request Mar 22, 2023
Open
@noklam
Copy link
Contributor Author

noklam commented Mar 22, 2023

Notes: (will keep this updated)
Challenges

  • Example 1 - Tensorflow haven't supported Python 3.11
    • Do we skip all the related datasets on a specific Python version? If so, how?
    • Follow up - how do we skip tensorflow install only in Python 3.11?

@noklam
Copy link
Contributor Author

noklam commented Mar 22, 2023 •
edited
Loading

Notes:

=========================== short test summary info ============================
SKIPPED [1] tests/extras/datasets/video/test_video_objects.py:125: Can't deal with videos with missing time info
ERROR tests/extras/datasets/dask/test_parquet_dataset.py
ERROR tests/extras/datasets/matplotlib/test_matplotlib_writer.py
ERROR tests/extras/datasets/spark/test_spark_dataset.py
ERROR tests/extras/datasets/video/test_video_dataset.py
ERROR tests/io/test_incremental_dataset.py
ERROR tests/io/test_partitioned_dataset.py
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_cant_pickle - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_read_existing_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_overwrite_empty_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_overwrite_not_empty_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_insert_not_empty_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_deltatable_dataset.py::TestDeltaTableDataSet::test_load - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_upsert_config_err - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_deltatable_dataset.py::TestDeltaTableDataSet::test_save - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_upsert_empty_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_deltatable_dataset.py::TestDeltaTableDataSet::test_exists - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_upsert_not_empty_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_invalid_pk_provided - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_save_and_load[None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_invalid_write_mode_provided - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_load_missing_model[None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_exists[None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_hdf5_save_format - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_unused_subclass_model_hdf5_save_format - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_protocol_usage[s3://bucket/test_tf-S3FileSystem] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_protocol_usage[file:///tmp/test_tf-LocalFileSystem] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_protocol_usage[/tmp/test_tf-LocalFileSystem] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_protocol_usage[gcs://bucket/test_tf-GCSFileSystem] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_protocol_usage[https://example.com/test_tf-HTTPFileSystem] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_load_extra_params[None-None-load_args0] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_catalog_release - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_fs_args[fs_args0] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_exists_with_exception[None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDataset::test_save_and_overwrite_existing_model[None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_save_and_load[2019-01-01T23.59.59.999Z-2019-01-01T23.59.59.999Z] - ModuleNotFoundError: No module named 'tensorflow'
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_save_and_load[None-None] - ModuleNotFoundError: No module named 'tensorflow'
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_hdf5_save_format[None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_prevent_overwrite[None-None] - ModuleNotFoundError: No module named 'tensorflow'
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_save_version_warning[2019-01-01T23.59.59.999Z-2019-01-02T00.00.00.000Z] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_http_filesystem_no_versioning - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_exists[None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_invalid_schema_insert - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_no_versions[None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_version_str_repr[None-None-None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_versioning_existing_dataset[None-None-None-None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/tensorflow/test_tensorflow_model_dataset.py::TestTensorFlowModelDatasetVersioned::test_save_and_load_with_device[None-None] - ImportError: cannot import name 'TensorFlowModelDataset' from 'kedro.extras.datasets.tensorflow' (/home/circleci/project/kedro/extras/datasets/tensorflow/__init__.py)
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_insert_to_non_existent_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_read_from_non_existent_table - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_spark_hive_dataset.py::TestSparkHiveDataSet::test_save_delta_format - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_memory_dataset.py::test_load_modify_original_data - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_memory_dataset.py::test_save_modify_original_data - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_memory_dataset.py::test_load_returns_same_spark_object - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
ERROR tests/extras/datasets/spark/test_memory_dataset.py::test_str_representation - _pickle.PicklingError: Could not serialize object: IndexError: tuple index out of range
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriter::test_save_data[None-None] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_save_data_None_None_0/test.png, protocol=file, save_args={'fmt': png}).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriter::test_exists[None-None] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_exists_None_None_0/test.png, protocol=file, save_args={'fmt': png}).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriterVersioned::test_prevent_overwrite[None-None] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_prevent_overwrite_None_No0/test.png, protocol=file, save_args={'fmt': png}, version=Version(load=None, save='2023-03-22T16.02.40.136Z')).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriterVersioned::test_save_version_warning[2019-01-02T00.00.00.000Z-2019-01-01T23.59.59.999Z] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_save_version_warning_20190/test.png, protocol=file, save_args={'fmt': png}, version=Version(load='2019-01-01T23.59.59.999Z', save='2019-01-02T00.00.00.000Z')).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriterVersioned::test_exists[None-None] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_exists_None_None_1/test.png, protocol=file, save_args={'fmt': png}, version=Version(load=None, save='2023-03-22T16.02.40.556Z')).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriterVersioned::test_save_data[None-None] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_save_data_None_None_1/test.png, protocol=file, save_args={'fmt': png}, version=Version(load=None, save='2023-03-22T16.02.40.740Z')).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/extras/datasets/holoviews/test_holoviews_writer.py::TestHoloviewsWriterVersioned::test_versioning_existing_dataset[None-None-None-None] - kedro.io.core.DataSetError: Failed while saving data to data set HoloviewsWriter(filepath=/tmp/pytest-of-circleci/pytest-0/popen-gw0/test_versioning_existing_datas0/test.png, protocol=file, save_args={'fmt': png}).
Could not find a 'matplotlib' renderer, available renderers are: .
FAILED tests/framework/cli/test_cli_hooks.py::TestKedroCLIHooks::test_kedro_cli_should_invoke_cli_hooks_from_plugin[-V-0] - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/framework/cli/test_cli.py::TestKedroCLI::test_project_commands_no_clipy - ModuleNotFoundError
FAILED tests/framework/cli/test_cli_hooks.py::TestKedroCLIHooks::test_kedro_cli_should_invoke_cli_hooks_from_plugin[info-2] - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/framework/cli/test_cli.py::TestKedroCLI::test_project_commands_invalid_clipy - AttributeError: None does not have the attribute '_is_project'
FAILED tests/framework/cli/test_cli_hooks.py::TestKedroCLIHooks::test_kedro_cli_should_invoke_cli_hooks_from_plugin[pipeline list-2] - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/framework/cli/test_cli.py::TestKedroCLI::test_project_commands_valid_clipy - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/framework/cli/test_cli.py::TestKedroCLI::test_kedro_cli_with_project - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/framework/cli/test_cli_hooks.py::TestKedroCLIHooks::test_kedro_cli_should_invoke_cli_hooks_from_plugin[starter-0] - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/io/test_data_catalog.py::TestDataCatalogVersioned::test_no_versions_with_cloud_protocol - AssertionError: Regex pattern did not match.
 Regex: "Did\\ not\\ find\\ any\\ versions\\ for\\ CSVDataSet\\(filepath=bucket/file\\.csv,\\ protocol=\\{\\},\\ save_args=\\{'index':\\ False\\},\\ version=Version\\(load=None,\\ save=None\\)\\)\\.\\ This\\ could\\ be\\ due\\ to\\ insufficient\\ permission\\."
 Input: "Failed while loading data from data set CSVDataSet(filepath=bucket/file.csv, protocol={}, save_args={'index': False}, version=Version(load=None, save=None)).\nUnable to locate credentials"
FAILED tests/tools/test_cli.py::TestCLITools::test_get_cli_structure_raw - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/tools/test_cli.py::TestCLITools::test_get_cli_structure_depth - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'
FAILED tests/tools/test_cli.py::TestCLITools::test_get_cli_structure_help - AttributeError: Module(cli=<Group Kedro>) does not have the attribute '_is_project'

@astrojuanlu astrojuanlu mentioned this pull request May 10, 2023
Closed
@astrojuanlu
Copy link
Member

Rebased this on top of main and attempted to include Spark and TensorFlow.

@astrojuanlu astrojuanlu force-pushed the bump/python311 branch 2 times, most recently from 2be0176 to edf3604 Compare May 11, 2023 08:30
@astrojuanlu
Copy link
Member

There seem to be some problems now with delta-spark:

  • Despite loosening the upper bound, pip-compile still errors on 3.11:
The conflict is caused by:
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.3.0 depends on pyspark<3.4.0 and >=3.3.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.2.0 depends on pyspark<3.4.0 and >=3.3.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.1.1 depends on pyspark<3.4.0 and >=3.3.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.1.0 depends on pyspark<3.4.0 and >=3.3.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.0.2 depends on pyspark<3.3.0 and >=3.2.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.0.1 depends on pyspark<3.3.0 and >=3.2.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 2.0.0 depends on pyspark<3.3.0 and >=3.2.0
    The user requested pyspark<4.0 and >=3.4
    delta-spark 1.2.1 depends on pyspark<3.3.0 and >=3.2.0

delta-spark merged support for Spark 3.4 one week ago delta-io/delta@5c3f4d3 so the next released version should be compatible.

  • In the other Python versions, it's leading to 2 tests failing: TestDeltaTableDataSet.test_save and TestDeltaTableDataSet.test_exists.

We're very close!

@astrojuanlu
Copy link
Member

https://pypi.org/project/delta-spark/2.4.0/

noklam added 18 commits May 26, 2023 14:29
@noklam @astrojuanlu
update CI
8265db7 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
Release all the upper bounds
02ea106 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
bump more versions
c2d1bd5 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
update requirements
29e5cbf 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
update requirements
34ce2bf 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
add dependency
249fded 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
update scipy
4e03227 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
fix typo
121a1be 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
update
6aa27b0 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
revert all changes
6318f25 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
bump holoviews
aa53b24 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
release pyarrow
2ffb41c 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
relax scipy
cca35a2 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
bump tensorflow
483cf0c 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
give up tensorflow, it's not supported yet
11f4db5 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
bump lower bound of pyarrow as it caused build error
ed19a95 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
bump pyarrow lower bound
4b5ca3d 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
@noklam @astrojuanlu
bump spark version
14896f1 
Signed-off-by: Nok <nok_lam_chan@mckinsey.com>
SajidAlamQB and others added 3 commits June 29, 2023 14:36
@SajidAlamQB
make moto fixed for 3.10
d855426 
Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
@noklam
fix spark compatibility issue
bc6f09c 
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
@noklam
Fix Spark error handling in exists() so it will be compatible with py…
3843caf 
…spark3.4

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
noklam
noklam commented Jul 3, 2023
View reviewed changes
kedro/extras/datasets/spark/spark_dataset.py Outdated
Comment on lines 404 to 409
if hasattr(exception, "desc"): # Pyspark <3.4
message = exception.desc
else:
message = exception.message

if "Path does not exist:" in message or "is not a Delta table" in message:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory we should freeze it, but this minor change would allow us release Python 3.11 without breaking change. We should also have this change in kedro-datasets

@noklam
make delta compatible in the same fashion as spark
da40d8c 
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
@noklam
refactor the test to pass coverage
ba47ba7 
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
@noklam
fix pyspark test
3997aa8 
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
@noklam
fix test that are too strict
7806129 
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
@noklam
fix delta
66d3966 
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
@noklam
refactor jdbc dataset tests with pytest and bug fix
761d008 
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
@noklam
fix issues with spark test for different version, introduce semver
e8479db 
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
@noklam
Merge branch 'main' into bump/python311
3e86cd1
@noklam
fix linter
bdea52d 
Signed-off-by: Nok <nok.lam.chan@quantumblack.com>
@noklam noklam mentioned this pull request Jul 5, 2023
Merged
5 tasks
@SajidAlamQB
Merge branch 'main' into bump/python311
e622c8c
@SajidAlamQB
add test for covering getOrCreate
18eaaed 
Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
@SajidAlamQB
more specific with test
7af8483 
Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
@SajidAlamQB
add 3.11 support to metadata
009742a 
Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
@SajidAlamQB
Merge branch 'main' into bump/python311
f4e0839
@SajidAlamQB
check getOrCreate() method is not None
3435950 
Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
@SajidAlamQB
Update deltatable_dataset.py
82b80fc 
Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
@SajidAlamQB
revert deltatable change
8d51f8b 
Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
@SajidAlamQB
Update test_spark_jdbc_dataset.py
7ccad55 
Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
@SajidAlamQB
fix deltatable
70f12be 
Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com>
@sbrugman
Copy link
Contributor

By updating the classifiers in pyproject.toml, pip will allow users to install from py3.11 after the release:

classifiers = [
    "Development Status :: 4 - Beta",
    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
]

@sbrugman
Copy link
Contributor

Any help needed here? Would be great to be able to bump versions.. Python 3.7 is also officially end-of-life and can be dropped

@astrojuanlu astrojuanlu mentioned this pull request Jul 27, 2023
Merged
5 tasks
@astrojuanlu
Copy link
Member

Superseded by #2851

@astrojuanlu astrojuanlu deleted the bump/python311 branch August 14, 2023 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@astrojuanlu astrojuanlu astrojuanlu left review comments

@merelcht merelcht Awaiting requested review from merelcht merelcht will be requested when the pull request is marked ready for review merelcht is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support python 3.11
4 participants
@noklam @astrojuanlu @sbrugman @SajidAlamQB