Airflow kubernetes cli command for generating pod templates #8009

Javier162380 · 2020-03-30T09:49:56Z

Adding a new command to the Airflow CLI.

airflow kubernetes pod-preview dag-id --sub-dir

The idea is to be able to generate pod templates from KubernetesPodOperator, so given a target dag it can generate all the k8s pod templates and printing in the CLI.

Example.

Just one consideration I am not connecting to any Kubernetes client so I am generating a default service account which means that you can not launch this pod in a Kubernetes cluster with a command like this.

kubectl apply -f file.yaml

Any feedback here is appreciated.

Also to be able to test this feature I need to add a new example-dag,

example_kubernetes_pod_operator.py

It is the same example given by Google in their composer tutorial (https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator), but adapting the imports to the new Kubernetes module import system.

Issue link: WILL BE INSERTED BY boring-cyborg

Make sure to mark the boxes below before creating PR: [x]

Description above provides context of the change
Unit tests coverage for changes (not needed for documentation changes)
Commits follow "How to write a good git commit message"
Relevant documentation is updated including usage instructions.
I will engage committers as explained in Contribution Workflow Example.

In the case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backward incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

boring-cyborg · 2020-03-30T09:50:00Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Be sure to read the Airflow Coding style.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/

potiuk · 2020-03-31T06:03:36Z

I think you will need to rebase it, we had some fixes in master!

Javier162380 · 2020-03-31T09:27:47Z

I think you will need to rebase it, we had some fixes in master!

ok this can explain why the travis test are failing?

potiuk · 2020-03-31T09:29:02Z

Some yes. Possibly not all.

…the cli

…on args to make more explanatory

…tributes method

Javier162380 · 2020-03-31T10:45:22Z

Some yes. Possibly not all.

ok I already rebase it let's hope this time tests pass

Javier162380 · 2020-04-02T14:39:39Z

the integration tests are still failing any clue @potiuk ?

mik-laj · 2020-04-20T16:58:56Z

@Javier162380 There seems to be a serialization error not related to this patch. I have prepared a small change that bypasses this problem.

curl https://termbin.com/9q5d | git am

From a9c150fe70b57e013d4bb106638caaf92a2e214b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Kamil=20Bregu=C5=82a?= <kamil.bregula@polidea.com>
Date: Mon, 20 Apr 2020 18:55:28 +0200
Subject: Fix serialization

---
 airflow/example_dags/example_kubernetes_pod_operator.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/airflow/example_dags/example_kubernetes_pod_operator.py b/airflow/example_dags/example_kubernetes_pod_operator.py
index f3b18cc13..74a7c4cab 100644
--- a/airflow/example_dags/example_kubernetes_pod_operator.py
+++ b/airflow/example_dags/example_kubernetes_pod_operator.py
@@ -20,8 +20,9 @@ import datetime
 from airflow import models
 from airflow.kubernetes import secret
 from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
+from airflow.utils.dates import days_ago
 
-YESTERDAY = datetime.datetime.now() - datetime.timedelta(days=1)
+YESTERDAY = days_ago(1)
 secret_env = [secret.Secret(
     deploy_type='env',
     deploy_target='SQL_CONN',
-- 
2.20.1

Javier162380 · 2020-04-21T08:28:15Z

@Javier162380 There seems to be a serialization error not related to this patch. I have prepared a small change that bypasses this problem.

curl https://termbin.com/9q5d | git am

From a9c150fe70b57e013d4bb106638caaf92a2e214b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Kamil=20Bregu=C5=82a?= <kamil.bregula@polidea.com>
Date: Mon, 20 Apr 2020 18:55:28 +0200
Subject: Fix serialization

---
 airflow/example_dags/example_kubernetes_pod_operator.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/airflow/example_dags/example_kubernetes_pod_operator.py b/airflow/example_dags/example_kubernetes_pod_operator.py
index f3b18cc13..74a7c4cab 100644
--- a/airflow/example_dags/example_kubernetes_pod_operator.py
+++ b/airflow/example_dags/example_kubernetes_pod_operator.py
@@ -20,8 +20,9 @@ import datetime
 from airflow import models
 from airflow.kubernetes import secret
 from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
+from airflow.utils.dates import days_ago
 
-YESTERDAY = datetime.datetime.now() - datetime.timedelta(days=1)
+YESTERDAY = days_ago(1)
 secret_env = [secret.Secret(
     deploy_type='env',
     deploy_target='SQL_CONN',
-- 
2.20.1

thanka! I don't was going wrong, to be honest

stale · 2020-06-05T09:06:12Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

turbaszek · 2020-10-10T12:14:23Z

@dimberman is this something that may help us for migration to Airflow 2.0?

dimberman · 2020-10-10T16:25:29Z

airflow/cli/cli_parser.py

+KUBERNETES_COMMANDS = (
+    ActionCommand(
+        name='pod-preview',
+        help='Generate a k8s template',


Suggested change

help='Generate a k8s template',

help='Generate a k8s template for KubernetesPodOperator tasks',

dimberman · 2020-10-10T16:27:55Z

airflow/cli/commands/kubernetes_command.py

+        pod_args['extract_xcom'] = kubernetes_tasks_args.get('do_xcom_push', None)
+
+    pod = pod_generator.PodGenerator(**(pod_args)).gen_pod()
+    pod = append_to_pod(pod,


Why not just use this existing function in the KubernetesPodOperator?

https://github.com/apache/airflow/blob/master/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L354

dimberman · 2020-10-12T00:20:31Z

@Javier162380 do you want to finish this? If not I can pick this up :)

Javier162380 · 2020-10-12T07:29:11Z

@Javier162380 do you want to finish this? If not I can pick this up :)

I can help you with! Need to reclone the repo it is quite oldie this PR

dimberman · 2021-02-05T18:52:31Z

Hi @Javier162380 are you interested in continuing with this task? Glad to help :).

ashb · 2021-02-08T20:13:17Z

@dimberman Didn't you add something like this already?

mik-laj · 2021-06-06T01:25:18Z

We already have airflow kubernetes airflow kubernetes generate-dag-yaml command. Closing...

boring-cyborg bot added area:CLI k8s labels Mar 30, 2020

Javier162380 changed the title ~~Cli/kubernetes~~ Airflow kubernetes cli command for generating pod templates Mar 30, 2020

potiuk requested a review from mik-laj March 30, 2020 10:36

Javier162380 added 14 commits March 31, 2020 12:38

adding kubernetes group command and kubernetes preview subcommand to …

ed03a0d

…the cli

adding kubernetes command module for generating kubernetes pod template

1affd1d

adding kubernetes test command

814b4f5

adding kubernetes pod operator example dag

fa0981b

erase unused import and add flake8 style recommendations

f72f30e

erasing unused imports and adding flake8 recommendations

91877f2

Add doctest to all the module methods

389a7d0

adding docstrings

b993858

adding empty space between imports

0dd7001

solve bug we were not rendering secrets and resources and type functi…

ff56972

…on args to make more explanatory

adding secret to the dag example

bcb81fa

Change call test to adapt to the new features

a5e0788

erasing typing an adding a more precise doc for get_kubernetes_pod_at…

5604726

…tributes method

formating test as one line was greatter than 110 chars

18349dc

Javier162380 force-pushed the cli/kubernetes branch from bf7ad91 to 18349dc Compare March 31, 2020 10:44

stale bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Jun 5, 2020

stale bot closed this Jun 12, 2020

mik-laj mentioned this pull request Jul 20, 2020

[AIRFLOW-5221] add host_aliases to KubernetesPodOperator #7365

Closed

6 tasks

kaxil reopened this Jul 20, 2020

stale bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Jul 20, 2020

kaxil added the pinned Protect from Stalebot auto closing label Jul 20, 2020

mik-laj mentioned this pull request Jul 23, 2020

[AIRFLOW-3126] Add option to specify additional K8s volumes #8150

Closed

5 tasks

dimberman requested changes Oct 10, 2020

View reviewed changes

mik-laj removed their request for review October 12, 2020 11:14

kaxil added provider:cncf-kubernetes Kubernetes provider related issues and removed area:k8s labels Nov 18, 2020

mik-laj closed this Jun 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Airflow kubernetes cli command for generating pod templates #8009

Airflow kubernetes cli command for generating pod templates #8009

Javier162380 commented Mar 30, 2020 •

edited

Loading

boring-cyborg bot commented Mar 30, 2020

potiuk commented Mar 31, 2020

Javier162380 commented Mar 31, 2020

potiuk commented Mar 31, 2020

Javier162380 commented Mar 31, 2020

Javier162380 commented Apr 2, 2020

mik-laj commented Apr 20, 2020

Javier162380 commented Apr 21, 2020

stale bot commented Jun 5, 2020

turbaszek commented Oct 10, 2020

dimberman Oct 10, 2020

dimberman Oct 10, 2020

dimberman commented Oct 12, 2020

Javier162380 commented Oct 12, 2020

dimberman commented Feb 5, 2021

ashb commented Feb 8, 2021

mik-laj commented Jun 6, 2021

	help='Generate a k8s template',
	help='Generate a k8s template for KubernetesPodOperator tasks',

Airflow kubernetes cli command for generating pod templates #8009

Airflow kubernetes cli command for generating pod templates #8009

Conversation

Javier162380 commented Mar 30, 2020 • edited Loading

boring-cyborg bot commented Mar 30, 2020

potiuk commented Mar 31, 2020

Javier162380 commented Mar 31, 2020

potiuk commented Mar 31, 2020

Javier162380 commented Mar 31, 2020

Javier162380 commented Apr 2, 2020

mik-laj commented Apr 20, 2020

Javier162380 commented Apr 21, 2020

stale bot commented Jun 5, 2020

turbaszek commented Oct 10, 2020

dimberman Oct 10, 2020

Choose a reason for hiding this comment

dimberman Oct 10, 2020

Choose a reason for hiding this comment

dimberman commented Oct 12, 2020

Javier162380 commented Oct 12, 2020

dimberman commented Feb 5, 2021

ashb commented Feb 8, 2021

mik-laj commented Jun 6, 2021

Javier162380 commented Mar 30, 2020 •

edited

Loading