Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support pod template for Spark 3.x applications #2141

Merged
merged 5 commits into from
Oct 24, 2024
Merged

Support pod template for Spark 3.x applications #2141

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5769519
Update API definition to support pod template
ChenYi015 Aug 22, 2024
2c37db1
Mark pod template field as schemaless
ChenYi015 Aug 27, 2024
1feff69
Add kubebuilder marker to preserve unknown fields
ChenYi015 Oct 21, 2024
f681007
Add example for using pod template
ChenYi015 Aug 22, 2024
157462d
Support pod template
ChenYi015 Aug 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ print-%: ; @echo $*=$($*)

.PHONY: manifests
manifests: controller-gen ## Generate CustomResourceDefinition, RBAC and WebhookConfiguration manifests.
$(CONTROLLER_GEN) crd rbac:roleName=spark-operator-controller webhook paths="./..." output:crd:artifacts:config=config/crd/bases
$(CONTROLLER_GEN) crd:generateEmbeddedObjectMeta=true rbac:roleName=spark-operator-controller webhook paths="./..." output:crd:artifacts:config=config/crd/bases

.PHONY: generate
generate: controller-gen ## Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.
Expand Down
8 changes: 8 additions & 0 deletions api/v1beta2/sparkapplication_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,14 @@ type Dependencies struct {
// SparkPodSpec defines common things that can be customized for a Spark driver or executor pod.
// TODO: investigate if we should use v1.PodSpec and limit what can be set instead.
type SparkPodSpec struct {
// Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
// Spark version >= 3.0.0 is required.
// Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
// +optional
// +kubebuilder:validation:Schemaless
// +kubebuilder:validation:Type:=object
// +kubebuilder:pruning:PreserveUnknownFields
Template *corev1.PodTemplateSpec `json:"template,omitempty"`
ChenYi015 marked this conversation as resolved.
Show resolved Hide resolved
// Cores maps to `spark.driver.cores` or `spark.executor.cores` for the driver and executors, respectively.
// +optional
// +kubebuilder:validation:Minimum=1
Expand Down
5 changes: 5 additions & 0 deletions api/v1beta2/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

31 changes: 31 additions & 0 deletions charts/spark-operator-chart/crds/sparkoperator.k8s.io_scheduledsparkapplications.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4755,6 +4755,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -9512,6 +9519,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -10351,6 +10365,23 @@ spec:
May contain labels and annotations that will be copied into the PVC
when creating it. No other fields are allowed and will be rejected during
validation.
properties:
annotations:
additionalProperties:
type: string
type: object
finalizers:
items:
type: string
type: array
labels:
additionalProperties:
type: string
type: object
name:
type: string
namespace:
type: string
type: object
spec:
description: |-
Expand Down
31 changes: 31 additions & 0 deletions charts/spark-operator-chart/crds/sparkoperator.k8s.io_sparkapplications.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4694,6 +4694,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -9421,6 +9428,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -10257,6 +10271,23 @@ spec:
May contain labels and annotations that will be copied into the PVC
when creating it. No other fields are allowed and will be rejected during
validation.
properties:
annotations:
additionalProperties:
type: string
type: object
finalizers:
items:
type: string
type: array
labels:
additionalProperties:
type: string
type: object
name:
type: string
namespace:
type: string
type: object
spec:
description: |-
Expand Down
31 changes: 31 additions & 0 deletions config/crd/bases/sparkoperator.k8s.io_scheduledsparkapplications.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4755,6 +4755,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -9512,6 +9519,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -10351,6 +10365,23 @@ spec:
May contain labels and annotations that will be copied into the PVC
when creating it. No other fields are allowed and will be rejected during
validation.
properties:
annotations:
additionalProperties:
type: string
type: object
finalizers:
items:
type: string
type: array
labels:
additionalProperties:
type: string
type: object
name:
type: string
namespace:
type: string
type: object
spec:
description: |-
Expand Down
31 changes: 31 additions & 0 deletions config/crd/bases/sparkoperator.k8s.io_sparkapplications.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4694,6 +4694,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -9421,6 +9428,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -10257,6 +10271,23 @@ spec:
May contain labels and annotations that will be copied into the PVC
when creating it. No other fields are allowed and will be rejected during
validation.
properties:
annotations:
additionalProperties:
type: string
type: object
finalizers:
items:
type: string
type: array
labels:
additionalProperties:
type: string
type: object
name:
type: string
namespace:
type: string
type: object
spec:
description: |-
Expand Down
16 changes: 16 additions & 0 deletions docs/api-docs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2828,6 +2828,22 @@ TODO: investigate if we should use v1.PodSpec and limit what can be set instead.
<tbody>
<tr>
<td>
<code>template</code><br/>
<em>
<a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podtemplatespec-v1-core">
Kubernetes core/v1.PodTemplateSpec
</a>
</em>
</td>
<td>
<em>(Optional)</em>
<p>Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version &gt;= 3.0.0 is required.
Ref: <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template">https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template</a>.</p>
</td>
</tr>
<tr>
<td>
<code>cores</code><br/>
<em>
int32
Expand Down
Loading