KubernetesPodOperator Guide #13405
Closed
KubernetesPodOperator Guide #13405
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi @VBhojawala ! Just reviewing your doc and it looks good. FYI there's one focused on Amazon MWAA and EKS at https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-eks-example.html that might be worth a reference as it, too, simplifies the authorization process. |
|
Hi @dimberman, |
dimberman
requested changes
Jan 19, 2021
|
Hi @dimberman , |
dimberman
requested changes
Feb 4, 2021
|
Hi @dimberman, |
dimberman
requested changes
Mar 15, 2021
There was a problem hiding this comment.
Hi @VBhojawala I added some more descriptions and removed logs for most of the cases except special ones (e.g. showing how XCom works), please add these fixes and then I'll review this one more time. Also interested what @kaxil and @potiuk think about whether it's ok to keep code snippets in docs (I'm not super opinionated but I recall people telling me not to do that in the past)
Comment on lines
+88
to
+111
| - After executing / debugging example and checking the logs | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| {taskinstance.py:1230} INFO - Exporting the following env vars: | ||
| AIRFLOW_CTX_DAG_OWNER=airflow | ||
| AIRFLOW_CTX_DAG_ID=example_k8s_yaml_config | ||
| AIRFLOW_CTX_TASK_ID=k8s_pod_yaml_config_task | ||
| AIRFLOW_CTX_EXECUTION_DATE=2020-12-05T00:00:00+00:00 | ||
| AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-12-05T00:00:00+00:00 | ||
| {pod_launcher.py:176} INFO - Event: myapp-pod had an event of type Pending | ||
| {pod_launcher.py:113} WARNING - Pod not yet started: myapp-pod | ||
| {pod_launcher.py:176} INFO - Event: myapp-pod had an event of type Succeeded | ||
| {pod_launcher.py:289} INFO - Event with job id myapp-pod Succeeded | ||
|
|
||
| {pod_launcher.py:136} INFO - myapp-pod created from YAML pod template. | ||
|
|
||
| .. contents:: | ||
| :depth: 1 | ||
| :local: | ||
| {taskinstance.py:1136} INFO - Marking task as SUCCESS. dag_id=example_k8s_yaml_config, task_id= | ||
| k8s_pod_yaml_config_task, execution_date=20201205T000000, start_date=20201206T130803, end_date=20201206T130818 | ||
| {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check | ||
| {dagrun.py:447} INFO - Marking run <DagRun example_k8s_yaml_config @ 2020-12-05 00:00:00+00:00: | ||
| backfill__2020-12-05T00:00:00+00:00, externally triggered: False> successful | ||
| {backfill_job.py:377} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 1 | | ||
| running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0 |
There was a problem hiding this comment.
I'm not sure what the point of this section is? Are you showing what the logs look like? If so can you give instructions on how to find the logs (e.g. via the UI or via a k8s command)
Comment on lines
+229
to
+254
| - After executing / debugging example and checking the logs | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| {taskinstance.py:1230} INFO - Exporting the following env vars: | ||
| AIRFLOW_CTX_DAG_OWNER=airflow | ||
| AIRFLOW_CTX_DAG_ID=example_k8s_operator | ||
| AIRFLOW_CTX_TASK_ID=k8s_pod_operator_task | ||
| AIRFLOW_CTX_EXECUTION_DATE=2020-12-03T00:00:00+00:00 | ||
| AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-12-03T00:00:00+00:00 | ||
|
|
||
| {pod_launcher.py:176} INFO - Event: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c had an event | ||
| of type Pending | ||
| {pod_launcher.py:113} WARNING - Pod not yet started: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c | ||
| {pod_launcher.py:176} INFO - Event: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c had an event of type | ||
| Succeeded | ||
| {pod_launcher.py:289} INFO - Event with job id airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c Succeeded | ||
|
|
||
| {pod_launcher.py:136} INFO - Hello World from pod [airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c] | ||
|
|
||
| {pod_launcher.py:176} INFO - Event: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c had an event of | ||
| type Succeeded | ||
| {pod_launcher.py:289} INFO - Event with job id airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c Succeeded | ||
| {taskinstance.py:1136} INFO - Marking task as SUCCESS. dag_id=example_k8s_operator, task_id=k8s_pod_operator_task, | ||
| execution_date=20201203T000000, start_date=20201204T140331, end_date=20201204T140345 | ||
| ................................................................................................................ |
There was a problem hiding this comment.
Why did you add more log lines here?
Comment on lines
+706
to
+707
| - Example Dag demonstrating use of ``Secret`` class which internally configures ``Volume`` and ``VolumeMount`` for | ||
| given secret. |
There was a problem hiding this comment.
Suggested change
| - Example Dag demonstrating use of ``Secret`` class which internally configures ``Volume`` and ``VolumeMount`` for | |
| given secret. | |
| This example shows how we can use an Airflow ``Secret`` object to inject a secret into a pod as a VolumeMount |
Comment on lines
+741
to
+773
| - After executing / debugging example and checking the logs | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| {taskinstance.py:1230} INFO - Exporting the following env vars: | ||
| AIRFLOW_CTX_DAG_OWNER=airflow | ||
| AIRFLOW_CTX_DAG_ID=example_k8s_secret_volume | ||
| AIRFLOW_CTX_TASK_ID=k8s_pod_operator_secret_volume_task | ||
| AIRFLOW_CTX_EXECUTION_DATE=2020-12-06T00:00:00+00:00 | ||
| AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-12-06T00:00:00+00:00 | ||
| {pod_launcher.py:176} INFO - Event: airflow-pod-operator-secret-volume-c03db098442b45f2aeb58e2dbca8e78f had | ||
| an event of type Pending | ||
| {pod_launcher.py:113} WARNING - Pod not yet started: airflow-pod-operator-secret-volume- | ||
| c03db098442b45f2aeb58e2dbca8e78f | ||
| {pod_launcher.py:176} INFO - Event: airflow-pod-operator-secret-volume-c03db098442b45f2aeb58e2dbca8e78f | ||
| had an event of type Succeeded | ||
| {pod_launcher.py:289} INFO - Event with job id airflow-pod-operator-secret-volume-c03db098442b45f2aeb58e2dbca8e78f | ||
| Succeeded | ||
|
|
||
| {pod_launcher.py:136} INFO - Secret Directory Content | ||
| total 0 | ||
| lrwxrwxrwx 1 root root 13 Dec 7 15:17 DB_PWD -> ..data/DB_PWD | ||
| lrwxrwxrwx 1 root root 14 Dec 7 15:17 DB_USER -> ..data/DB_USER | ||
|
|
||
| {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check | ||
| {dagrun.py:444} INFO - Marking run <DagRun example_k8s_secret_volume | ||
| @ 2020-12-06 00:00:00+00:00: backfill__2020-12-06T00:00:00+00:00, externally triggered: False> successful | ||
| {backfill_job.py:377} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 1 | | ||
| running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0 | ||
| {backfill_job.py:830} INFO - Backfill done. Exiting. | ||
|
|
||
|
|
||
|
|
There was a problem hiding this comment.
Suggested change
| - After executing / debugging example and checking the logs | |
| .. code-block:: bash | |
| {taskinstance.py:1230} INFO - Exporting the following env vars: | |
| AIRFLOW_CTX_DAG_OWNER=airflow | |
| AIRFLOW_CTX_DAG_ID=example_k8s_secret_volume | |
| AIRFLOW_CTX_TASK_ID=k8s_pod_operator_secret_volume_task | |
| AIRFLOW_CTX_EXECUTION_DATE=2020-12-06T00:00:00+00:00 | |
| AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-12-06T00:00:00+00:00 | |
| {pod_launcher.py:176} INFO - Event: airflow-pod-operator-secret-volume-c03db098442b45f2aeb58e2dbca8e78f had | |
| an event of type Pending | |
| {pod_launcher.py:113} WARNING - Pod not yet started: airflow-pod-operator-secret-volume- | |
| c03db098442b45f2aeb58e2dbca8e78f | |
| {pod_launcher.py:176} INFO - Event: airflow-pod-operator-secret-volume-c03db098442b45f2aeb58e2dbca8e78f | |
| had an event of type Succeeded | |
| {pod_launcher.py:289} INFO - Event with job id airflow-pod-operator-secret-volume-c03db098442b45f2aeb58e2dbca8e78f | |
| Succeeded | |
| {pod_launcher.py:136} INFO - Secret Directory Content | |
| total 0 | |
| lrwxrwxrwx 1 root root 13 Dec 7 15:17 DB_PWD -> ..data/DB_PWD | |
| lrwxrwxrwx 1 root root 14 Dec 7 15:17 DB_USER -> ..data/DB_USER | |
| {taskinstance.py:1195} INFO - 0 downstream tasks scheduled from follow-on schedule check | |
| {dagrun.py:444} INFO - Marking run <DagRun example_k8s_secret_volume | |
| @ 2020-12-06 00:00:00+00:00: backfill__2020-12-06T00:00:00+00:00, externally triggered: False> successful | |
| {backfill_job.py:377} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 1 | | |
| running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0 | |
| {backfill_job.py:830} INFO - Backfill done. Exiting. |
Comment on lines
+814
to
+839
| - After executing / debugging example and checking the logs | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| INFO - Exporting the following env vars: | ||
| AIRFLOW_CTX_DAG_OWNER=airflow | ||
| AIRFLOW_CTX_DAG_ID=example_k8s_limit_resource | ||
| AIRFLOW_CTX_TASK_ID=k8s_pod_limit_resource_task | ||
| AIRFLOW_CTX_EXECUTION_DATE=2020-12-05T00:00:00+00:00 | ||
| AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-12-05T00:00:00+00:00 | ||
| {pod_launcher.py:176} INFO - Event: airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6 had an | ||
| event of type Pending | ||
| {pod_launcher.py:113} WARNING - Pod not yet started: airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6 | ||
| {pod_launcher.py:176} INFO - Event: airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6 had an | ||
| event of type Succeeded | ||
| {pod_launcher.py:289} INFO - Event with job id airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6 Succeeded | ||
|
|
||
| {pod_launcher.py:136} INFO - Hello World from pod [airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6] | ||
|
|
||
| {taskinstance.py:1136} INFO - Marking task as SUCCESS. dag_id=example_k8s_limit_resource,task_id= | ||
| k8s_pod_limit_resource_task, execution_date=20201205T000000, start_date=20201206T035456, end_date=20201206T035517 | ||
| {backfill_job.py:377} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 1 | | ||
| running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0 | ||
| {backfill_job.py:830} INFO - Backfill done. Exiting. | ||
|
|
||
|
|
There was a problem hiding this comment.
Suggested change
| - After executing / debugging example and checking the logs | |
| .. code-block:: bash | |
| INFO - Exporting the following env vars: | |
| AIRFLOW_CTX_DAG_OWNER=airflow | |
| AIRFLOW_CTX_DAG_ID=example_k8s_limit_resource | |
| AIRFLOW_CTX_TASK_ID=k8s_pod_limit_resource_task | |
| AIRFLOW_CTX_EXECUTION_DATE=2020-12-05T00:00:00+00:00 | |
| AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-12-05T00:00:00+00:00 | |
| {pod_launcher.py:176} INFO - Event: airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6 had an | |
| event of type Pending | |
| {pod_launcher.py:113} WARNING - Pod not yet started: airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6 | |
| {pod_launcher.py:176} INFO - Event: airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6 had an | |
| event of type Succeeded | |
| {pod_launcher.py:289} INFO - Event with job id airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6 Succeeded | |
| {pod_launcher.py:136} INFO - Hello World from pod [airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6] | |
| {taskinstance.py:1136} INFO - Marking task as SUCCESS. dag_id=example_k8s_limit_resource,task_id= | |
| k8s_pod_limit_resource_task, execution_date=20201205T000000, start_date=20201206T035456, end_date=20201206T035517 | |
| {backfill_job.py:377} INFO - [backfill progress] | finished run 1 of 1 | tasks waiting: 0 | succeeded: 1 | | |
| running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0 | |
| {backfill_job.py:830} INFO - Backfill done. Exiting. |
Comment on lines
+840
to
+895
| - Describing Pod configuration | ||
|
|
||
| .. code-block:: bash | ||
|
|
||
| $ kubectl describe pod airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6 | ||
|
|
||
| Name: airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6 | ||
| Namespace: default | ||
| Priority: 0 | ||
| Node: minikube/192.168.49.2 | ||
| Start Time: Sun, 06 Dec 2020 09:25:02 +0530 | ||
| Labels: airflow_version=2.0.0b2 | ||
| dag_id=example_k8s_limit_resource | ||
| execution_date=2020-12-05T0000000000-c71846343 | ||
| kubernetes_pod_operator=True | ||
| task_id=k8s_pod_limit_resource_task | ||
| try_number=1 | ||
| Annotations: <none> | ||
| Status: Succeeded | ||
| IP: 172.17.0.7 | ||
| IPs: | ||
| IP: 172.17.0.7 | ||
| Containers: | ||
| base: | ||
| Container ID: docker://819ae8713aefb51b4f9bbdd7567adf706ebcee402418e1c4ef358c0bab90bc8b | ||
| Image: alpine | ||
| Image ID: docker-pullable://alpine@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a | ||
| Port: <none> | ||
| Host Port: <none> | ||
| Command: | ||
| sh | ||
| -c | ||
| echo "Hello World from pod [$HOSTNAME]" | ||
| State: Terminated | ||
| Reason: Completed | ||
| Exit Code: 0 | ||
| Started: Sun, 06 Dec 2020 09:25:14 +0530 | ||
| Finished: Sun, 06 Dec 2020 09:25:14 +0530 | ||
| Ready: False | ||
| Restart Count: 0 | ||
|
|
||
| Limits: | ||
| cpu: 2 | ||
| memory: 200Mi | ||
| Requests: | ||
| cpu: 1 | ||
| memory: 100Mi | ||
|
|
||
| Environment: <none> | ||
| Mounts: | ||
| /var/run/secrets/kubernetes.io/serviceaccount from default-token-ltgdm (ro) | ||
| .................................................................................. | ||
|
|
||
|
|
||
|
|
||
|
|
There was a problem hiding this comment.
Suggested change
| - Describing Pod configuration | |
| .. code-block:: bash | |
| $ kubectl describe pod airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6 | |
| Name: airflow-pod-limit-resource-ac4c107dd10549c89a2015f976e729d6 | |
| Namespace: default | |
| Priority: 0 | |
| Node: minikube/192.168.49.2 | |
| Start Time: Sun, 06 Dec 2020 09:25:02 +0530 | |
| Labels: airflow_version=2.0.0b2 | |
| dag_id=example_k8s_limit_resource | |
| execution_date=2020-12-05T0000000000-c71846343 | |
| kubernetes_pod_operator=True | |
| task_id=k8s_pod_limit_resource_task | |
| try_number=1 | |
| Annotations: <none> | |
| Status: Succeeded | |
| IP: 172.17.0.7 | |
| IPs: | |
| IP: 172.17.0.7 | |
| Containers: | |
| base: | |
| Container ID: docker://819ae8713aefb51b4f9bbdd7567adf706ebcee402418e1c4ef358c0bab90bc8b | |
| Image: alpine | |
| Image ID: docker-pullable://alpine@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a | |
| Port: <none> | |
| Host Port: <none> | |
| Command: | |
| sh | |
| -c | |
| echo "Hello World from pod [$HOSTNAME]" | |
| State: Terminated | |
| Reason: Completed | |
| Exit Code: 0 | |
| Started: Sun, 06 Dec 2020 09:25:14 +0530 | |
| Finished: Sun, 06 Dec 2020 09:25:14 +0530 | |
| Ready: False | |
| Restart Count: 0 | |
| Limits: | |
| cpu: 2 | |
| memory: 200Mi | |
| Requests: | |
| cpu: 1 | |
| memory: 100Mi | |
| Environment: <none> | |
| Mounts: | |
| /var/run/secrets/kubernetes.io/serviceaccount from default-token-ltgdm (ro) | |
| .................................................................................. |
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
|
@VBhojawala can you take a look at the failing doc build? |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
bot
added
the
stale
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added KubernetesPodOperator guide in KubernetesPodOperator.rst
related: #8970