Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update main with 2.5.0 config values #28076

Closed
wants to merge 1 commit into from
Closed

Update main with 2.5.0 config values #28076

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 22 additions & 22 deletions airflow/config_templates/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2338,36 +2338,36 @@
- name: pod_template_file
description: |
Path to the YAML pod file that forms the basis for KubernetesExecutor workers.
version_added: 1.10.11
version_added: 2.5.0
type: string
example: ~
default: ""
see_also: ":ref:`concepts:pod_template_file`"
- name: worker_container_repository
description: |
The repository of the Kubernetes Image for the Worker to Run
version_added: ~
version_added: 2.5.0
type: string
example: ~
default: ""
- name: worker_container_tag
description: |
The tag of the Kubernetes Image for the Worker to Run
version_added: ~
version_added: 2.5.0
type: string
example: ~
default: ""
- name: namespace
description: |
The Kubernetes namespace where airflow workers should be created. Defaults to ``default``
version_added: ~
version_added: 2.5.0
type: string
example: ~
default: "default"
- name: delete_worker_pods
description: |
If True, all worker pods will be deleted upon termination
version_added: ~
version_added: 2.5.0
type: string
example: ~
default: "True"
Expand All @@ -2377,7 +2377,7 @@
failed worker pods will not be deleted so users can investigate them.
This only prevents removal of worker pods where the worker itself failed,
not when the task it ran failed.
version_added: 1.10.11
version_added: 2.5.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should change these values?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Many others below too. This one is just an example)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is not nice...section renaming

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I think this is the correct thing to do. We have moved the configurations to a new section. The new section was created in 2.5.0, therefore, it makes sense that all items in the section are also added in 2.5.0. This was the practice when we moved some items to a new 'database' section. See #22284. WDYT

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think so. This change will leave new-comers an impression that these configurations are really new since 2.5.0, while it's not the truth

Copy link
Contributor Author

@ephraimbuddy ephraimbuddy Dec 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AIRFLOW__KUBERNETES__POD_TEMPLATE_FILE was 1.10.11 while AIRFLOW__KUBERNETES_EXECUTOR__POD_TEMPLATE_FILE started existing in 2.5.0.
Makes me wonder if we should have a new field on the config file, version_renamed or something because, in some way, this was really added in 2.5.0 but practically works down to 1.10.11 with old section where it was originally added in the old section.
cc @ashb

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have been off for a while... @potiuk what do you think about this? I think the correctness depends on how one looks at it. The section/option is new in 2.5.0 while the old section/option is not new.

If we mark it 1.10.11, users would think it'll work in 1.10.11 but that's not true. It will only work in 1.10.11 with the old section name, not the new one.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's also misleading now.

We should add a feature IMHO (because as far as I know it does not exist now) to refer to which was deprecated/previous version of the config - and still keep the "from" there. It does not have to be detailed but it should explain that this section has been renamed from [kubernetes] in 2.5.0

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW. Glad you are back :). I hope you rested 👍 We missed you 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW. Glad you are back :). I hope you rested 👍 We missed you 😄

Thanks! Missed you guys too 😀

type: string
example: ~
default: "False"
Expand All @@ -2388,15 +2388,15 @@
per-heartbeat. It is HIGHLY recommended that users increase this
number to match the tolerance of their kubernetes cluster for
better performance.
version_added: 1.10.3
version_added: 2.5.0
type: string
example: ~
default: "1"
- name: multi_namespace_mode
description: |
Allows users to launch pods in multiple namespaces.
Will require creating a cluster-role for the scheduler
version_added: 1.10.12
version_added: 2.5.0
type: boolean
example: ~
default: "False"
Expand All @@ -2405,22 +2405,22 @@
Use the service account kubernetes gives to pods to connect to kubernetes cluster.
It's intended for clients that expect to be running inside a pod running on kubernetes.
It will raise an exception if called from a process not running in a kubernetes environment.
version_added: ~
version_added: 2.5.0
type: string
example: ~
default: "True"
- name: cluster_context
description: |
When running with in_cluster=False change the default cluster_context or config_file
options to Kubernetes client. Leave blank these to use default behaviour like ``kubectl`` has.
version_added: 1.10.3
version_added: 2.5.0
type: string
example: ~
default: ~
- name: config_file
description: |
Path to the kubernetes configfile to be used when ``in_cluster`` is set to False
version_added: 1.10.3
version_added: 2.5.0
type: string
example: ~
default: ~
Expand All @@ -2431,7 +2431,7 @@
List of supported params are similar for all core_v1_apis, hence a single config
variable for all apis. See:
https://raw.githubusercontent.com/kubernetes-client/python/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/api/core_v1_api.py
version_added: 1.10.4
version_added: 2.5.0
type: string
example: ~
default: ""
Expand All @@ -2442,31 +2442,31 @@
This should be an object and can contain any of the options listed in the ``v1DeleteOptions``
class defined here:
https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19
version_added: 1.10.12
version_added: 2.5.0
type: string
example: '{"grace_period_seconds": 10}'
default: ""
- name: enable_tcp_keepalive
description: |
Enables TCP keepalive mechanism. This prevents Kubernetes API requests to hang indefinitely
when idle connection is time-outed on services like cloud load balancers or firewalls.
version_added: 2.0.0
version_added: 2.5.0
type: boolean
example: ~
default: "True"
- name: tcp_keep_idle
description: |
When the `enable_tcp_keepalive` option is enabled, TCP probes a connection that has
been idle for `tcp_keep_idle` seconds.
version_added: 2.0.0
version_added: 2.5.0
type: integer
example: ~
default: "120"
- name: tcp_keep_intvl
description: |
When the `enable_tcp_keepalive` option is enabled, if Kubernetes API does not respond
to a keepalive probe, TCP retransmits the probe after `tcp_keep_intvl` seconds.
version_added: 2.0.0
version_added: 2.5.0
type: integer
example: ~
default: "30"
Expand All @@ -2475,43 +2475,43 @@
When the `enable_tcp_keepalive` option is enabled, if Kubernetes API does not respond
to a keepalive probe, TCP retransmits the probe `tcp_keep_cnt number` of times before
a connection is considered to be broken.
version_added: 2.0.0
version_added: 2.5.0
type: integer
example: ~
default: "6"
- name: verify_ssl
description: |
Set this to false to skip verifying SSL certificate of Kubernetes python client.
version_added: 2.1.0
version_added: 2.5.0
type: boolean
example: ~
default: "True"
- name: worker_pods_pending_timeout
description: |
How long in seconds a worker can be in Pending before it is considered a failure
version_added: 2.1.0
version_added: 2.5.0
type: integer
example: ~
default: "300"
- name: worker_pods_pending_timeout_check_interval
description: |
How often in seconds to check if Pending workers have exceeded their timeouts
version_added: 2.1.0
version_added: 2.5.0
type: integer
example: ~
default: "120"
- name: worker_pods_queued_check_interval
description: |
How often in seconds to check for task instances stuck in "queued" status without a pod
version_added: 2.2.0
version_added: 2.5.0
type: integer
example: ~
default: "60"
- name: worker_pods_pending_timeout_batch_size
description: |
How many pending pods to check for timeout violations in each check interval.
You may want this higher if you have a very large cluster and/or use ``multi_namespace_mode``.
version_added: 2.1.0
version_added: 2.5.0
type: integer
example: ~
default: "100"
Expand Down