Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions airflow-core/docs/core-concepts/objectstorage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ object you want to interact with. For example, to point to a bucket in s3, you w

.. code-block:: python

from airflow.io.path import ObjectStoragePath
from airflow.sdk import ObjectStoragePath

base = ObjectStoragePath("s3://aws_default@my-bucket/")

Expand Down Expand Up @@ -153,8 +153,8 @@ would do the following:

.. code-block:: python

from airflow.io.path import ObjectStoragePath
from airflow.io.store import attach
from airflow.sdk import ObjectStoragePath
from airflow.sdk.io import attach

from fsspec.implementations.dbfs import DBFSFileSystem

Expand All @@ -163,7 +163,7 @@ would do the following:


.. note::
To reuse the registration across tasks make sure to attach the backend at the top-level of your DAG.
To reuse the registration across tasks, make sure to attach the backend at the top-level of your DAG.
Otherwise, the backend will not be available across multiple tasks.


Expand All @@ -177,7 +177,7 @@ and builds upon `Universal Pathlib <https://github.com/fsspec/universal_pathlib>
the same API to interact with object storage as you would with a local filesystem. In this section we only list the
differences between the two APIs. Extended operations beyond the standard Path API, like copying and moving, are listed
in the next section. For details about each operation, like what arguments they take, see the documentation of
the :class:`~airflow.io.path.ObjectStoragePath` class.
the :class:`~airflow.sdk.definitions.path.ObjectStoragePath` class.


mkdir
Expand Down Expand Up @@ -321,7 +321,7 @@ are used to connect to s3 and a parquet file, indicated by a ``ObjectStoragePath
.. code-block:: python

import duckdb
from airflow.io.path import ObjectStoragePath
from airflow.sdk import ObjectStoragePath

path = ObjectStoragePath("s3://my-bucket/my-table.parquet", conn_id="aws_default")
conn = duckdb.connect(database=":memory:")
Expand Down
21 changes: 21 additions & 0 deletions airflow-core/newsfragments/45425.significant.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
The ``airflow.io`` class ``ObjectStoragePath`` and function ``attach`` are moved to ``airflow.sdk``.

* Types of change

* [x] Dag changes
* [ ] Config changes
* [ ] API changes
* [ ] CLI changes
* [ ] Behaviour changes
* [ ] Plugin changes
* [ ] Dependency changes
* [x] Code interface changes

* Migration rules needed

* ruff

* AIR302

* [ ] ``airflow.io.path.ObjectStoragePath`` → ``airflow.sdk.ObjectStoragePath``
* [ ] ``airflow.io.attach`` → ``airflow.sdk.io.attach``
1 change: 0 additions & 1 deletion airflow-core/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,6 @@ dependencies = [
# 0.115.10 fastapi was a bad release that broke our API's and static checks.
# Related fastapi issue here: https://github.com/fastapi/fastapi/discussions/13431
"fastapi[standard]>=0.115.0,!=0.115.10",
"fsspec>=2023.10.0",
"gitpython>=3.1.40",
"gunicorn>=20.1.0",
"httpx>=0.25.0",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,7 @@
import pendulum
import requests

from airflow.io.path import ObjectStoragePath
from airflow.sdk import dag, task
from airflow.sdk import ObjectStoragePath, dag, task

# [END import_module]

Expand Down
Loading