Draft Common Message Queue

[vikramkoka]

Here is a very early draft PR to introduce and socialize the concept of a "common message queue" abstraction similar to the "Common SQL" and "Common IO" abstractions in Airflow.

This will be a provider package similar to those and is intended to be an abstraction over Apache Kafka, Amazon SQL, and Google PubSub to begin with. It can then be expanded to other messaging systems based on community adoption.

The initial goal with this is to provide a simple abstraction for integrating Event Driven Scheduling coming with Airflow 3 to message notification systems such as Kafka, currently being used to publish data availability.

At this stage, this is very much a WIP draft intended to solicit input from the community.

[vincbeck]

Implementation wise, here is my thinking. I am starting by MessageQueueTrigger.

Given msg_queue, MessageQueueTrigger needs to figure which hook it will use to poll/pop a message from the queue. Example: if msg_queue.starts_with("https://sqs."): hook = SqsHook(...). Then we can use the hook to retrieve the message. The hook will contain the logic for each provider (AWS, Google, Kafka, ...). This means, this new provider will have a dependency with all these providers. Do you think this is an issue? Did you have something else in mind?

[vikramkoka]

Implementation wise, here is my thinking. I am starting by MessageQueueTrigger.

Given msg_queue, MessageQueueTrigger needs to figure which hook it will use to poll/pop a message from the queue. Example: if msg_queue.starts_with("https://sqs."): hook = SqsHook(...). Then we can use the hook to retrieve the message. The hook will contain the logic for each provider (AWS, Google, Kafka, ...). This means, this new provider will have a dependency with all these providers. Do you think this is an issue? Did you have something else in mind?

You are right Vincent. I did think about the "Composition vs. Inheritance" approach tradeoff.

The composition style interface as defined here is easier for the DAG author, but more maintenance for us.
I talked about this with Ash and Jed as well and because of the underlying plumbing already present in Airflow for finding connections, et al, this seemed reasonable as an approach in order to make the end-user experience better.

[jscheffl]

In general looks good. Some more nit that I would have on the Python code/Interface but we can leave this until it is in real review.

Would be great to add an example DAG as well for the showcase.

[vincbeck]

I am iterating on that PR but the new provider is not recognized. I get:

ModuleNotFoundError: No module named 'airflow.providers.common.msgq'

With the new restructure, what is the process to add a new provider? Do I just need to create provider.yaml and pyproject.toml and it will be automatically detected/indexed? @potiuk

[vincbeck]

I updated the PR. I focused only on the trigger side. Please let me know if this is what you had in mind in terms of implementation regarding the trigger. I really see it as a proxy of the provider triggers. I could not test it because the new provider is not recognized but once that solved I should be able to test it.

[potiuk]

I am iterating on that PR but the new provider is not recognized. I get:

ModuleNotFoundError: No module named 'airflow.providers.common.msgq'

With the new restructure, what is the process to add a new provider? Do I just need to create provider.yaml and pyproject.toml and it will be automatically detected/indexed? @potiuk

You need to look at the main pyproject.toml and add the provider in the same way as others (there are few places). Then uv sync or image build should work after that.

And yes I updated https://github.com/apache/airflow/blob/main/providers/MANAGING_PROVIDERS_LIFECYCLE.rst#creating-a-new-community-provider - with the new structure and how to add a new provider, but that part is likely missing so after you figure it out, PRs there are most welcome.

BTW. It will likely slightly change in the future as we will move airflow-core and others, but still it would be great to keep it updated.

[potiuk]

Generally @vincbeck -> look at everything below [dependency-groups] in the root pyproject.toml - the provider should be added in all those places.

[vincbeck]

Thank you :D

[vincbeck]

I dont understand what Sphinx is complaining about: /opt/airflow/docs/apache-airflow-providers-common-messaging/_api/airflow/providers/common/messaging/providers/base_provider/index.rst: WARNING: document isn't included in any toctree

[potiuk]

I dont understand what Sphinx is complaining about: /opt/airflow/docs/apache-airflow-providers-common-messaging/_api/airflow/providers/common/messaging/providers/base_provider/index.rst: WARNING: document isn't included in any toctree

Sphinx is - as usual - speaking riddles :) . It means that there is index.rst file generated by autoapi (base_provider module) and that index is not mentioned anywhere. This means that you have to add it to some "table of content" file and refer to it - because otherwise that file is not reachable from anywhere. And it means some documentation is missing to explain what it is - usually a reference doc (see in other providers)

[potiuk]

I think it would be great to somehow explain that "toctree" better :)

[potiuk]

Likely some documentation about base_provider should be added here https://github.com/apache/airflow/blob/f4fd6fd5ae45cd20924149aa0201d2da08a63112/providers/common/messaging/docs/providers.rst

You need to extend base_provider ble ble ble....

[vincbeck]

Likely some documentation about base_provider should be added here https://github.com/apache/airflow/blob/f4fd6fd5ae45cd20924149aa0201d2da08a63112/providers/common/messaging/docs/providers.rst

You need to extend base_provider ble ble ble....

It is already there: https://github.com/apache/airflow/pull/46694/files#diff-f54feaaca8fd8ecfad946ef2cc5b389e082660ba53d305843bab44f5a014d582R36

[potiuk]

It is already there: https://github.com/apache/airflow/pull/46694/files#diff-f54feaaca8fd8ecfad946ef2cc5b389e082660ba53d305843bab44f5a014d582R36

So if the index to the "init.py" is not linked (and does not need to be linked) from anywhere - it should be excluded in "docs/conf.py" explicitly for provider package builds.

[potiuk]

And yes - I reverse engineered it having similar issues. Likely it should be done bettter, so we do not have to do it manually.

[vincbeck]

docs/conf.py

Yeah I did not want to do that but I think I'll do that, I really cannot find a solution. What I do not understand is there are a lot of modules in others providers that are not documented (for good reasons like the module utils in amazon) but Sphinx is not complaining about that. I do not know why it does like this new provider.

But anyway, thanks for your help and I'll add these 2 paths to exclude_patterns.

[potiuk]

Yeah I did not want to do that but I think I'll do that, I really cannot find a solution. What I do not understand is there are a lot of modules in others providers that are not documented (for good reasons like the module utils in amazon) but Sphinx is not complaining about that. I do not know why it does like this new provider.

They are likely referred to in class docstrings or others. The thing is that if your class or module is not referred ANYWHERE - the only way you can reach it is by direct URL. And this is what Sphinx complains about.

[vincbeck]

Yeah I did not want to do that but I think I'll do that, I really cannot find a solution. What I do not understand is there are a lot of modules in others providers that are not documented (for good reasons like the module utils in amazon) but Sphinx is not complaining about that. I do not know why it does like this new provider.

They are likely referred to in class docstrings or others. The thing is that if your class or module is not referred ANYWHERE - the only way you can reach it is by direct URL. And this is what Sphinx complains about.

It is probably that! Makes a bit more sense in all that Sphinx dialect :) Thanks

[vincbeck]

All green :) I also tested it manually and triggered few DAGs using MessageQueueTrigger and it works nicely. I really like the user experience, users do not have to worry about the actual implementation of the queue provider

[potiuk]

NICE!