Update airflow documentation for latest version of airflow

[stichbury]

Another task that comes out of kedro-org/kedro#2435

• kedro-airflow-k8s isn't available for Kedro >0.17 (but this is for GetInData to update)
• Our airflow/astronomer tutorial isn't for the latest Airflow versions

It would be sensible to run through the docs that we are responsible for to confirm they are still OK for the latest version, and meanwhile to caution the reader as to the limitations of version support as noted above.

[stichbury]

I'm not totally sure what version of Airflow the kedro-airflow plugin is tested with/documented against.

The test_requirements.txt file says apache-airflow < 3.0, but the latest version is 2.6, so 🤷

It would be helpful to know for each of the deployment docs (for future review/update) when they were last tested and against what. I think this should be a standard for every touchpoint in the docs going forward.

[stichbury]

Hi @sbrugman I wondered if you have any thoughts on the current state of the Kedro docs about Airflow as @astrojuanlu advised me that you've been working on PRs for kedro-airflow recently, so are presumably well-qualified to judge the current status/quality of the docs.

1. Are they working at all, or sadly out of date?
2. If you had to estimate the amount of work needed to update them to the latest version, as a T-shirt size, would it be Small, Medium, Large or other?

[sbrugman]

At this moment the kedro-airflow-k8s plugin is not maintained.
Users can use the official kedro-airflow plugin, with a custom template for users on kubernetes.
(There is no need for two plugins in practice)

[merelcht]

Closing in favour of #605