Merge branch 'main' into enh_athena2pyarrow

.bumpversion.toml

@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "3.8.0"
+current_version = "3.9.0"
 commit = false
 tag = false
 tag_name = "{new_version}"

README.md

@@ -94,27 +94,27 @@ FROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3
 ## At scale
 AWS SDK for pandas can also run your workflows at scale by leveraging [Modin](https://modin.readthedocs.io/en/stable/) and [Ray](https://www.ray.io/). Both projects aim to speed up data workloads by distributing processing over a cluster of workers.
 
-Read our [docs](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/scale.html) or head to our latest [tutorials](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials) to learn more.
+Read our [docs](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/scale.html) or head to our latest [tutorials](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials) to learn more.
 
 > ⚠️ **Ray is currently not available for Python 3.12. While AWS SDK for pandas supports Python 3.12, it cannot be used at scale.**
 
 ## [Read The Docs](https://aws-sdk-pandas.readthedocs.io/)
 
-- [**What is AWS SDK for pandas?**](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/about.html)
-- [**Install**](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/install.html)
- - [PyPi (pip)](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/install.html#pypi-pip)
- - [Conda](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/install.html#conda)
- - [AWS Lambda Layer](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/install.html#aws-lambda-layer)
- - [AWS Glue Python Shell Jobs](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/install.html#aws-glue-python-shell-jobs)
- - [AWS Glue PySpark Jobs](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/install.html#aws-glue-pyspark-jobs)
- - [Amazon SageMaker Notebook](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/install.html#amazon-sagemaker-notebook)
- - [Amazon SageMaker Notebook Lifecycle](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/install.html#amazon-sagemaker-notebook-lifecycle)
- - [EMR](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/install.html#emr)
- - [From source](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/install.html#from-source)
-- [**At scale**](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/scale.html)
- - [Getting Started](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/scale.html#getting-started)
- - [Supported APIs](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/scale.html#supported-apis)
- - [Resources](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/scale.html#resources)
+- [**What is AWS SDK for pandas?**](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/about.html)
+- [**Install**](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/install.html)
+ - [PyPi (pip)](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/install.html#pypi-pip)
+ - [Conda](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/install.html#conda)
+ - [AWS Lambda Layer](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/install.html#aws-lambda-layer)
+ - [AWS Glue Python Shell Jobs](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/install.html#aws-glue-python-shell-jobs)
+ - [AWS Glue PySpark Jobs](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/install.html#aws-glue-pyspark-jobs)
+ - [Amazon SageMaker Notebook](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/install.html#amazon-sagemaker-notebook)
+ - [Amazon SageMaker Notebook Lifecycle](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/install.html#amazon-sagemaker-notebook-lifecycle)
+ - [EMR](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/install.html#emr)
+ - [From source](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/install.html#from-source)
+- [**At scale**](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/scale.html)
+ - [Getting Started](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/scale.html#getting-started)
+ - [Supported APIs](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/scale.html#supported-apis)
+ - [Resources](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/scale.html#resources)
 - [**Tutorials**](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials)
  - [001 - Introduction](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/001%20-%20Introduction.ipynb)
  - [002 - Sessions](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/002%20-%20Sessions.ipynb)
@@ -155,30 +155,30 @@ Read our [docs](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/scale.html) or he
  - [039 - Athena Iceberg](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/039%20-%20Athena%20Iceberg.ipynb)
  - [040 - EMR Serverless](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/040%20-%20EMR%20Serverless.ipynb)
  - [041 - Apache Spark on Amazon Athena](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/041%20-%20Apache%20Spark%20on%20Amazon%20Athena.ipynb)
-- [**API Reference**](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html)
- - [Amazon S3](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#amazon-s3)
- - [AWS Glue Catalog](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#aws-glue-catalog)
- - [Amazon Athena](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#amazon-athena)
- - [Amazon Redshift](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#amazon-redshift)
- - [PostgreSQL](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#postgresql)
- - [MySQL](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#mysql)
- - [SQL Server](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#sqlserver)
- - [Oracle](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#oracle)
- - [Data API Redshift](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#data-api-redshift)
- - [Data API RDS](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#data-api-rds)
- - [OpenSearch](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#opensearch)
- - [AWS Glue Data Quality](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#aws-glue-data-quality)
- - [Amazon Neptune](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#amazon-neptune)
- - [DynamoDB](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#dynamodb)
- - [Amazon Timestream](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#amazon-timestream)
- - [Amazon EMR](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#amazon-emr)
- - [Amazon CloudWatch Logs](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#amazon-cloudwatch-logs)
- - [Amazon Chime](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#amazon-chime)
- - [Amazon QuickSight](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#amazon-quicksight)
- - [AWS STS](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#aws-sts)
- - [AWS Secrets Manager](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#aws-secrets-manager)
- - [Global Configurations](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#global-configurations)
- - [Distributed - Ray](https://aws-sdk-pandas.readthedocs.io/en/3.8.0/api.html#distributed-ray)
+- [**API Reference**](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html)
+ - [Amazon S3](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#amazon-s3)
+ - [AWS Glue Catalog](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#aws-glue-catalog)
+ - [Amazon Athena](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#amazon-athena)
+ - [Amazon Redshift](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#amazon-redshift)
+ - [PostgreSQL](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#postgresql)
+ - [MySQL](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#mysql)
+ - [SQL Server](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#sqlserver)
+ - [Oracle](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#oracle)
+ - [Data API Redshift](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#data-api-redshift)
+ - [Data API RDS](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#data-api-rds)
+ - [OpenSearch](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#opensearch)
+ - [AWS Glue Data Quality](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#aws-glue-data-quality)
+ - [Amazon Neptune](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#amazon-neptune)
+ - [DynamoDB](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#dynamodb)
+ - [Amazon Timestream](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#amazon-timestream)
+ - [Amazon EMR](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#amazon-emr)
+ - [Amazon CloudWatch Logs](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#amazon-cloudwatch-logs)
+ - [Amazon Chime](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#amazon-chime)
+ - [Amazon QuickSight](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#amazon-quicksight)
+ - [AWS STS](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#aws-sts)
+ - [AWS Secrets Manager](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#aws-secrets-manager)
+ - [Global Configurations](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#global-configurations)
+ - [Distributed - Ray](https://aws-sdk-pandas.readthedocs.io/en/3.9.0/api.html#distributed-ray)
 - [**License**](https://github.com/aws/aws-sdk-pandas/blob/main/LICENSE.txt)
 - [**Contributing**](https://github.com/aws/aws-sdk-pandas/blob/main/CONTRIBUTING.md)
 

VERSION

@@ -1 +1 @@
-3.8.0
+3.9.0

awswrangler/__metadata__.py

@@ -7,5 +7,5 @@
 
 __title__: str = "awswrangler"
 __description__: str = "Pandas on AWS."
-__version__: str = "3.8.0"
+__version__: str = "3.9.0"
 __license__: str = "Apache License 2.0"

awswrangler/_utils.py

@@ -856,7 +856,7 @@ def split_pandas_frame(df: pd.DataFrame, splits: int) -> list[pd.DataFrame]:
  total = len(df)
  each_section, extras = divmod(total, splits)
  section_sizes = [0] + extras * [each_section + 1] + (splits - extras) * [each_section]
- div_points = _nx.array(section_sizes, dtype=_nx.intp).cumsum()
+ div_points = _nx.array(section_sizes, dtype=_nx.intp).cumsum() # type: ignore[attr-defined]
 
  sub_dfs = []
  for i in range(splits):

awswrangler/athena/_read.py

@@ -794,11 +794,11 @@ def read_sql_query(
 
  **Related tutorial:**
 
- - `Amazon Athena <https://aws-sdk-pandas.readthedocs.io/en/3.8.0/
+ - `Amazon Athena <https://aws-sdk-pandas.readthedocs.io/en/3.9.0/
  tutorials/006%20-%20Amazon%20Athena.html>`_
- - `Athena Cache <https://aws-sdk-pandas.readthedocs.io/en/3.8.0/
+ - `Athena Cache <https://aws-sdk-pandas.readthedocs.io/en/3.9.0/
  tutorials/019%20-%20Athena%20Cache.html>`_
- - `Global Configurations <https://aws-sdk-pandas.readthedocs.io/en/3.8.0/
+ - `Global Configurations <https://aws-sdk-pandas.readthedocs.io/en/3.9.0/
  tutorials/021%20-%20Global%20Configurations.html>`_
 
  **There are three approaches available through ctas_approach and unload_approach parameters:**
@@ -862,7 +862,7 @@ def read_sql_query(
  /athena.html#Athena.Client.get_query_execution>`_ .
 
  For a practical example check out the
- `related tutorial <https://aws-sdk-pandas.readthedocs.io/en/3.8.0/
+ `related tutorial <https://aws-sdk-pandas.readthedocs.io/en/3.9.0/
  tutorials/024%20-%20Athena%20Query%20Metadata.html>`_!
 
 
@@ -1139,11 +1139,11 @@ def read_sql_table(
 
  **Related tutorial:**
 
- - `Amazon Athena <https://aws-sdk-pandas.readthedocs.io/en/3.8.0/
+ - `Amazon Athena <https://aws-sdk-pandas.readthedocs.io/en/3.9.0/
  tutorials/006%20-%20Amazon%20Athena.html>`_
- - `Athena Cache <https://aws-sdk-pandas.readthedocs.io/en/3.8.0/
+ - `Athena Cache <https://aws-sdk-pandas.readthedocs.io/en/3.9.0/
  tutorials/019%20-%20Athena%20Cache.html>`_
- - `Global Configurations <https://aws-sdk-pandas.readthedocs.io/en/3.8.0/
+ - `Global Configurations <https://aws-sdk-pandas.readthedocs.io/en/3.9.0/
  tutorials/021%20-%20Global%20Configurations.html>`_
 
  **There are three approaches available through ctas_approach and unload_approach parameters:**
@@ -1207,7 +1207,7 @@ def read_sql_table(
  /athena.html#Athena.Client.get_query_execution>`_ .
 
  For a practical example check out the
- `related tutorial <https://aws-sdk-pandas.readthedocs.io/en/3.8.0/
+ `related tutorial <https://aws-sdk-pandas.readthedocs.io/en/3.9.0/
  tutorials/024%20-%20Athena%20Query%20Metadata.html>`_!
 
 

awswrangler/catalog/_create.py

@@ -1115,7 +1115,7 @@ def create_csv_table(
  If True allows schema evolution (new or missing columns), otherwise a exception will be raised.
  (Only considered if dataset=True and mode in ("append", "overwrite_partitions"))
  Related tutorial:
- https://aws-sdk-pandas.readthedocs.io/en/3.8.0/tutorials/014%20-%20Schema%20Evolution.html
+ https://aws-sdk-pandas.readthedocs.io/en/3.9.0/tutorials/014%20-%20Schema%20Evolution.html
  sep : str
  String of length 1. Field delimiter for the output file.
  skip_header_line_count : Optional[int]
@@ -1300,7 +1300,7 @@ def create_json_table(
  If True allows schema evolution (new or missing columns), otherwise a exception will be raised.
  (Only considered if dataset=True and mode in ("append", "overwrite_partitions"))
  Related tutorial:
- https://aws-sdk-pandas.readthedocs.io/en/3.8.0/tutorials/014%20-%20Schema%20Evolution.html
+ https://aws-sdk-pandas.readthedocs.io/en/3.9.0/tutorials/014%20-%20Schema%20Evolution.html
  serde_library : Optional[str]
  Specifies the SerDe Serialization library which will be used. You need to provide the Class library name
  as a string.

awswrangler/s3/_read_orc.py

@@ -225,7 +225,7 @@ def read_orc(
  must return a bool, True to read the partition or False to ignore it.
  Ignored if `dataset=False`.
  E.g ``lambda x: True if x["year"] == "2020" and x["month"] == "1" else False``
- https://aws-sdk-pandas.readthedocs.io/en/3.8.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
+ https://aws-sdk-pandas.readthedocs.io/en/3.9.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
  columns : List[str], optional
  List of columns to read from the file(s).
  validate_schema : bool, default False
@@ -385,7 +385,7 @@ def read_orc_table(
  must return a bool, True to read the partition or False to ignore it.
  Ignored if `dataset=False`.
  E.g ``lambda x: True if x["year"] == "2020" and x["month"] == "1" else False``
- https://aws-sdk-pandas.readthedocs.io/en/3.8.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
+ https://aws-sdk-pandas.readthedocs.io/en/3.9.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
  columns : List[str], optional
  List of columns to read from the file(s).
  validate_schema : bool, default False

awswrangler/s3/_read_parquet.py

@@ -398,7 +398,7 @@ def read_parquet(
  must return a bool, True to read the partition or False to ignore it.
  Ignored if `dataset=False`.
  E.g ``lambda x: True if x["year"] == "2020" and x["month"] == "1" else False``
- https://aws-sdk-pandas.readthedocs.io/en/3.8.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
+ https://aws-sdk-pandas.readthedocs.io/en/3.9.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
  columns : List[str], optional
  List of columns to read from the file(s).
  validate_schema : bool, default False
@@ -640,7 +640,7 @@ def read_parquet_table(
  must return a bool, True to read the partition or False to ignore it.
  Ignored if `dataset=False`.
  E.g ``lambda x: True if x["year"] == "2020" and x["month"] == "1" else False``
- https://aws-sdk-pandas.readthedocs.io/en/3.8.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
+ https://aws-sdk-pandas.readthedocs.io/en/3.9.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
  columns : List[str], optional
  List of columns to read from the file(s).
  validate_schema : bool, default False

awswrangler/s3/_read_text.py

@@ -236,7 +236,7 @@ def read_csv(
  This function MUST return a bool, True to read the partition or False to ignore it.
  Ignored if `dataset=False`.
  E.g ``lambda x: True if x["year"] == "2020" and x["month"] == "1" else False``
- https://aws-sdk-pandas.readthedocs.io/en/3.8.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
+ https://aws-sdk-pandas.readthedocs.io/en/3.9.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
  s3_additional_kwargs: dict[str, Any], optional
  Forwarded to botocore requests.
  ray_args: typing.RaySettings, optional
@@ -398,7 +398,7 @@ def read_fwf(
  This function MUST return a bool, True to read the partition or False to ignore it.
  Ignored if `dataset=False`.
  E.g ``lambda x: True if x["year"] == "2020" and x["month"] == "1" else False``
- https://aws-sdk-pandas.readthedocs.io/en/3.8.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
+ https://aws-sdk-pandas.readthedocs.io/en/3.9.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
  s3_additional_kwargs: dict[str, Any], optional
  Forwarded to botocore requests.
  ray_args: typing.RaySettings, optional
@@ -567,7 +567,7 @@ def read_json(
  This function MUST return a bool, True to read the partition or False to ignore it.
  Ignored if `dataset=False`.
  E.g ``lambda x: True if x["year"] == "2020" and x["month"] == "1" else False``
- https://aws-sdk-pandas.readthedocs.io/en/3.8.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
+ https://aws-sdk-pandas.readthedocs.io/en/3.9.0/tutorials/023%20-%20Flexible%20Partitions%20Filter.html
  s3_additional_kwargs: dict[str, Any], optional
  Forwarded to botocore requests.
  ray_args: typing.RaySettings, optional

awswrangler/s3/_write_dataset.py

@@ -28,7 +28,7 @@ def _get_bucketing_series(df: pd.DataFrame, bucketing_info: typing.BucketingInfo
  axis="columns",
  )
  )
- return bucket_number_series.astype(np.array([pd.CategoricalDtype(range(bucketing_info[1]))]))
+ return bucket_number_series.astype(pd.CategoricalDtype(range(bucketing_info[1])))
 
 
 def _simulate_overflow(value: int, bits: int = 31, signed: bool = False) -> int:

awswrangler/s3/_write_orc.py

@@ -407,7 +407,7 @@ def to_orc(
  concurrent_partitioning: bool
  If True will increase the parallelism level during the partitions writing. It will decrease the
  writing time and increase the memory usage.
- https://aws-sdk-pandas.readthedocs.io/en/3.8.0/tutorials/022%20-%20Writing%20Partitions%20Concurrently.html
+ https://aws-sdk-pandas.readthedocs.io/en/3.9.0/tutorials/022%20-%20Writing%20Partitions%20Concurrently.html
  mode: str, optional
  ``append`` (Default), ``overwrite``, ``overwrite_partitions``. Only takes effect if dataset=True.
  catalog_versioning : bool
@@ -416,7 +416,7 @@ def to_orc(
  If True allows schema evolution (new or missing columns), otherwise a exception will be raised. True by default.
  (Only considered if dataset=True and mode in ("append", "overwrite_partitions"))
  Related tutorial:
- https://aws-sdk-pandas.readthedocs.io/en/3.8.0/tutorials/014%20-%20Schema%20Evolution.html
+ https://aws-sdk-pandas.readthedocs.io/en/3.9.0/tutorials/014%20-%20Schema%20Evolution.html
  database : str, optional
  Glue/Athena catalog: Database name.
  table : str, optional