Rework universal-pathlib in ObjectStoragePath

[gopidesupavan]

Body

#56270 (review)

Rework universal-pathlib usage, we are seeing latest version usage causing recursive error, it is due to accessing path in the __str__ function in the https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/io/path.py#L416

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[pmuntianu]

There is another issue with universal-pathlib 0.3.0:
We deploy our infra in k8s and use s3 as storage. We also run Astronomer Cosmos and DBT. All together it causes an error because ObjectStoragePath can not be used to create Path object.

dependencies:

python = ">=3.12,<3.13"
hiyapyco = "0.7.0"
psycopg2-binary = "2.9.10"
dbt-core = "1.9.6"
dbt-databricks = "1.9.5"
astronomer-cosmos = "1.10.1"
apache-airflow = {extras = ["statsd"], version = "2.10.3"}
apache-airflow-providers-amazon = "9.8.0"
apache-airflow-providers-databricks = "7.4.0"
apache-airflow-providers-openlineage = "2.3.0"
apache-airflow-providers-cncf-kubernetes = "10.3.1"
apache-airflow-providers-apprise = "2.1.0"
flask-appbuilder = {version = "4.5.3", extras = ["oauth"]}
flask-limiter = "3.9.2"
authlib = "1.6.0"
xmlsec = "1.3.14"

Error:

[2025-10-03, 11:25:57 UTC] {local.py:244} INFO - dbtRunner is available. Using dbtRunner for invoking dbt.
[2025-10-03, 11:25:57 UTC] {local.py:464} INFO - Cloning project to writable temp directory /tmp/tmp5pxgmqac from /opt/airflow/configuration/data-platform/data-assets/dbt
[2025-10-03, 11:25:57 UTC] {taskinstance.py:3311} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 767, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 733, in _execute_callable
    return ExecutionCallableRunner(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/cosmos/operators/base.py", line 310, in execute
    self.build_and_run_cmd(context=context, cmd_flags=self.add_cmd_flags())
  File "/home/airflow/.local/lib/python3.12/site-packages/cosmos/operators/local.py", line 842, in build_and_run_cmd
    result = self.run_command(
             ^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/cosmos/operators/local.py", line 601, in run_command
    self._clone_project(tmp_dir_path)
  File "/home/airflow/.local/lib/python3.12/site-packages/cosmos/operators/local.py", line 478, in _clone_project
    copy_manifest_file_if_exists(self.manifest_filepath, Path(tmp_dir_path))
  File "/home/airflow/.local/lib/python3.12/site-packages/cosmos/dbt/project.py", line 104, in copy_manifest_file_if_exists
    if source_manifest and Path(source_manifest).exists():
                           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/pathlib.py", line 1162, in __init__
    super().__init__(*args)
  File "/usr/local/lib/python3.12/pathlib.py", line 373, in __init__
    raise TypeError(
TypeError: argument should be a str or an os.PathLike object where __fspath__ returns a str, not 'ObjectStoragePath'

To fix the bug, we had to add this to our dependencies list:
universal-pathlib = "0.2.6" # upath 0.3.0 has a bug