Airflow CLI extensions via plugins

[dwreeves]

Description

I'm not sure how big this is, but I imagine this is small enough to not require an AIP.

Example of what this could look like:

from airflow.plugins_manager import AirflowPlugin
from airflow.cli.cli_config import (
    ActionCommand,
    GroupCommand,
)


def function():
    print("Hello, world!")


class ExamplePlugin(AirflowPlugin):
    name = "example"
    cli_commands = [
        GroupCommand(
            name="example",
            help="Example command from plugin",
            description=(
                "This runs all example commands"
            ),
            subcommands=[
                ActionCommand(
                    name="subcommand",
                    help="Run a subcommand",
                    func=function,
                )
            ],
        ),
    ]

Use case/motivation

The main motivation that comes to mind is Cosmos: https://github.com/astronomer/astronomer-cosmos Cosmos is a library that runs dbt in Airflow. dbt is a command-line tool that orchestrates, manages, and runs SQL queries. It would be a big quality-of-life upgrade if Cosmos were able to execute dbt commands with something like: airflow cosmos --project foo dbt run --select bar.

Right now many users struggle to execute dbt via the command line when it's configured for Airflow. I personally use a weird setup to execute one-off runs of dbt models that I expect many users would not know how to do themselves.

Some advantages for Cosmos, and more generally:

• Would hook easily into config configured in the user's Airflow instance.
• Could be used to execute one-off commands. This is a popular use case for dbt; many more savvy users already have an unscheduled parameterized DAG devoted to executing manual dbt commands. There are, however, many other commands that users might want to run directly from the CLI. E.g. imagine if something like the maintenance DAGs repo had a CLI.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Generally - yep.

The main issue with it is that it has to be done in the way that does not introduce latency. We already do this for some providers (celery, kubernetes contribute their CLI) - but this is done only when they are used as executor - and because executor is configured in the config. Otherwise discovering and adding args should not be done before "providersmanager" is run - the problem is generally with discovery and walking through all the providers.

But maybe someone can find a good way to speed it up/pre-discover it as early as possible (when --help) and autocomplete is run.

[dwreeves]

Got it. Yep, that is tricky.

Just checking the code too, I was also not aware that plugins can be defined the airflow.apache_airflow_provider endpoint on top of the airflow.plugins entry point. 😭

[suii2210]

This would be a great DX improvement, especially for Cosmos where users want to run one-off commands without creating helper DAGs.
The concern about CLI startup time makes sense. A lazy-loading approach for plugin commands could help keep airflow --help and autocomplete fast.
Happy to help explore this if needed.

[potiuk]

We already have a PR for that that improves lazy loading and makes it possible to have plugins - it also introduces "cli" section in provider info, making it possible for any provider to contribute CLI. #59805

[suii2210]

Thanks for the clarification!
The provider metadata cli section and lazy-loading approach in #59805 makes sense and addresses the startup-time concern nicely.
Happy to help test this or contribute follow-up improvements/docs if needed.

[jason810496]

We already have a PR for that that improves lazy loading and makes it possible to have plugins - it also introduces "cli" section in provider info, making it possible for any provider to contribute CLI. #59805

Yes, exactly, that is also one of the reason why I want to introduce "cli" section in provider info, in order to unlock integration of provider-level and plugin-level cli with airflow cli.