diff --git a/contributing-docs/quick-start-ide/contributors_quick_start_pycharm.rst b/contributing-docs/quick-start-ide/contributors_quick_start_pycharm.rst index 7d13dd1c8cd25..4a67670530737 100644 --- a/contributing-docs/quick-start-ide/contributors_quick_start_pycharm.rst +++ b/contributing-docs/quick-start-ide/contributors_quick_start_pycharm.rst @@ -29,7 +29,6 @@ Setup your project alt="Cloning github fork to Pycharm"> - 2. Paste the repository link in the URL field and submit. .. raw:: html @@ -39,52 +38,37 @@ Setup your project alt="Cloning github fork to Pycharm"> -3. Configure the source root directories well as for ``task-sdk`` and ``devel-common``. - You also have to set "source" and "tests" root directories for each provider you want to develop (!). +3. Synchronize local ``.venv`` virtualenv using uv - This is important in Airflow 3.0 we split ``task-sdk``, ``devel-common`` and each provider to be separate - distribution - each with separate ``pyproject.toml`` file, so you need to separately - add "src" and "tests" directories for each provider you develop to be respectively - "source roots" and "test roots". + .. code-block:: bash - This might be improve and might be better automated in the future, but for now you need to do it - for each provider separately. + $ uv sync - .. raw:: html +This will create ``.venv`` virtual environment in the project root directory and install all the dependencies of +airflow core. If you plan to work on providers, at this time you can install dependencies for all providers: -
- Adding Source Root directories to Pycharm -
+ .. code-block:: bash - You also need to add ``task-sdk`` sources (and ``devel-common`` in similar way). + $ uv sync --all-packages - .. raw:: html +Or for specific provider and it's cross-provider dependencies: -
- Adding Source Root directories to Pycharm -
- -4. Once step 3 is done it is recommended to invalidate caches and restart Pycharm. + .. code-block:: bash - .. raw:: html + $ uv sync --packages apache-airflow-provider-amazon -
- Invalidate caches and restart Pycharm -
+Next: Configure your IDEA project. -5. An alternative way to add source roots is to configure the ``airflow.iml`` file under ``.idea`` directory and update the - ``module.xml`` file: +3. The fastest way to add source roots is to configure the ``airflow.iml`` file under ``.idea`` directory and update the + ``module.xml`` file using the ``setup_idea.py`` script: To setup the source roots for all the modules that exist in the project, you can run the following command: This needs to done on the airflow repository root directory. It overwrites the existing ``.idea/airflow.iml`` and - ``.idea/modules.xml`` files. + ``.idea/modules.xml`` files if they exist. .. code-block:: bash - $ python setup_idea.py + $ uv run setup_idea.py Then Restart the PyCharm/IntelliJ IDEA. @@ -102,6 +86,52 @@ Setup your project alt="modules.xml"> +4. Alternatively, you can configure your project manually. Configure the source root directories well + as for ``airflow-core`` ``task-sdk``, ``airflow-ctl`` and ``devel-common``. You also have to set + "source" and "tests" root directories for each provider you want to develop (!). + + In Airflow 3.0 we split ``airflow-core``, ``task-sdk``, ``airflow-ctl``, ``devel-common``, + and each provider to be separate distribution - each with separate ``pyproject.toml`` file, + so you need to separately add ``src`` and ``tests`` directories for each provider you develop + to be respectively "source roots" and "test roots". + + .. raw:: html + +
+ Adding Source Root directories to Pycharm +
+ + You also need to add ``task-sdk`` sources (and ``devel-common`` in similar way). + + .. raw:: html + +
+ Adding Source Root directories to Pycharm +
+ +5. Once step 3 or 4 is done you should configure python interpreter for your PyCharm/IntelliJ to use + the virtualenv created by ``uv sync``. + + .. raw:: html + +
+ Configuring Python Interpreter +
+ +6. It is recommended to invalidate caches and restart PyCharm after setting up the project. + + .. raw:: html + +
+ Invalidate caches and restart Pycharm +
+ + + Setting up debugging #################### diff --git a/contributing-docs/quick-start-ide/images/pycharm_add_interpreter.png b/contributing-docs/quick-start-ide/images/pycharm_add_interpreter.png index 207bba32efc7d..70cc4f1097831 100644 Binary files a/contributing-docs/quick-start-ide/images/pycharm_add_interpreter.png and b/contributing-docs/quick-start-ide/images/pycharm_add_interpreter.png differ diff --git a/setup_idea.py b/setup_idea.py index 2f8e75d2b6b4b..2bbb2d23bac03 100644 --- a/setup_idea.py +++ b/setup_idea.py @@ -14,9 +14,18 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. +# /// script +# requires-python = ">=3.9" +# dependencies = [ +# "rich>=13.6.0", +# ] +# /// from __future__ import annotations -import os +from pathlib import Path + +from rich import print +from rich.prompt import Confirm iml_xml_template = """ @@ -57,43 +66,52 @@ source_root_module_patter: str = '' -source_root_modules: list[str] = ["airflow-core", "airflow-ctl", "dev/breeze", "task-sdk"] +source_root_modules: list[str] = ["airflow-core", "airflow-ctl", "task-sdk", "devel-common", "dev/breeze"] all_module_paths: list[str] = [] -ROOT_DIR: str = "./providers" +ROOT_AIRFLOW_FOLDER_PATH = Path(__file__).parent +IDEA_FOLDER_PATH = ROOT_AIRFLOW_FOLDER_PATH / ".idea" +AIRFLOW_IML_FILE = IDEA_FOLDER_PATH / "airflow.iml" +MODULES_XML_FILE = IDEA_FOLDER_PATH / "modules.xml" + +ROOT_PROVIDERS_FOLDER_PATH = ROOT_AIRFLOW_FOLDER_PATH / "providers" def setup_idea(): # Providers discovery - for dirpath, _, filenames in os.walk(ROOT_DIR): - if "pyproject.toml" in filenames: - relative_path = os.path.relpath(dirpath, ROOT_DIR) - source_root_modules.append(f"providers/{relative_path}") + for pyproject_toml_file in ROOT_PROVIDERS_FOLDER_PATH.rglob("pyproject.toml"): + relative_path = pyproject_toml_file.relative_to(ROOT_AIRFLOW_FOLDER_PATH).parent.as_posix() + source_root_modules.append(f"{relative_path}") source_root_modules.sort() for module in source_root_modules: + print(f"[green]Adding[/] module: [blue]{module}[/]") all_module_paths.append(source_root_module_patter.format(path=f"{module}/src", status="false")) - all_module_paths.append(source_root_module_patter.format(path=f"{module}/test", status="true")) - - source_root_module_path: str = "\n\t\t".join(all_module_paths) + all_module_paths.append(source_root_module_patter.format(path=f"{module}/tests", status="true")) - base_source_root_xml: str = iml_xml_template.format(SOURCE_ROOT_MODULE_PATH=source_root_module_path) + source_root_module_path = "\n\t\t".join(all_module_paths) - with open(".idea/airflow.iml", "w") as file: - file.write(base_source_root_xml) + base_source_root_xml = iml_xml_template.format(SOURCE_ROOT_MODULE_PATH=source_root_module_path) - with open(".idea/modules.xml", "w") as file: - file.write(module_xml_template) + IDEA_FOLDER_PATH.mkdir(exist_ok=True) + AIRFLOW_IML_FILE.write_text(base_source_root_xml) + MODULES_XML_FILE.write_text(module_xml_template) if __name__ == "__main__": - user_input = input( - "This script will overwrites the .idea/airflow.iml and .idea/modules.xml files. Press Enter Y/N to continue: " - ) - if user_input.lower() == "y": + print("\n[yellow]Warning!!![/] This script will update the PyCharm/IntelliJ IDEA configuration files:\n") + print(f"* {AIRFLOW_IML_FILE}") + print(f"* {MODULES_XML_FILE}\n") + should_continue = Confirm.ask("Overwrite the files?") + if should_continue: + print() setup_idea() - print("Updated airflow.iml and modules.xml files, Now restart the PyCharm/IntelliJ IDEA") + print("\n[green]Success\n") + print( + f"Updated {AIRFLOW_IML_FILE} and {MODULES_XML_FILE} files. " + f"Now restart the PyCharm/IntelliJ IDEA\n" + ) else: - print("Not updating airflow.iml and modules.xml files") - exit(0) + print("[yellow]Skipped\n") + print(f"Not updated {AIRFLOW_IML_FILE} and {MODULES_XML_FILE} files\n") diff --git a/tests/sdk/__init__.py b/tests/sdk/__init__.py deleted file mode 100644 index 13a83393a9124..0000000000000 --- a/tests/sdk/__init__.py +++ /dev/null @@ -1,16 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. diff --git a/tests/sdk/api/__init__.py b/tests/sdk/api/__init__.py deleted file mode 100644 index 13a83393a9124..0000000000000 --- a/tests/sdk/api/__init__.py +++ /dev/null @@ -1,16 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. diff --git a/tests/sdk/execution_time/__init__.py b/tests/sdk/execution_time/__init__.py deleted file mode 100644 index 13a83393a9124..0000000000000 --- a/tests/sdk/execution_time/__init__.py +++ /dev/null @@ -1,16 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License.