Skip to content

[SPARK-38135][K8S] Introduce job sheduling related configurations #35436

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Yikun wants to merge 1 commit into from
Closed

[SPARK-38135][K8S] Introduce job sheduling related configurations #35436

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import org.apache.spark.deploy.k8s.Constants._
import org.apache.spark.internal.Logging
import org.apache.spark.internal.config.{PYSPARK_DRIVER_PYTHON, PYSPARK_PYTHON}
import org.apache.spark.internal.config.ConfigBuilder
import org.apache.spark.network.util.ByteUnit

private[spark] object Config extends Logging {

Expand Down Expand Up @@ -675,6 +676,42 @@ private[spark] object Config extends Logging {
.checkValue(value => value > 0, "Maximum number of pending pods should be a positive integer")
.createWithDefault(Int.MaxValue)

val KUBERNETES_JOB_QUEUE = ConfigBuilder("spark.kubernetes.job.queue")
.doc("The name of the queue to which the job is submitted. This info " +
"will be stored in configuration and passed to specified feature step.")
.version("3.3.0")
.stringConf
.createWithDefault("default")

val KUBERNETES_JOB_MIN_CPU = ConfigBuilder("spark.kubernetes.job.minCPU")
.doc("The minimum CPU for running the job. This info " +
"will be stored in configuration and passed to specified feature step.")
.version("3.3.0")
.doubleConf
.createWithDefault(2.0)

val KUBERNETES_JOB_MIN_MEMORY = ConfigBuilder("spark.kubernetes.job.minMemory")
.doc("The minimum memory for running the job, in MiB unless otherwise specified. This info " +
"will be stored in configuration and passed to specified feature step.")
.version("3.3.0")
.bytesConf(ByteUnit.MiB)
.createWithDefaultString("3g")

val KUBERNETES_JOB_MIN_MEMBER = ConfigBuilder("spark.kubernetes.job.minMember")
.doc("The minimum number of pods running in a job. This info " +
"will be stored in configuration and passed to specified feature step.")
.version("3.3.0")
.intConf
.checkValue(value => value > 0, "The minimum number should be a positive integer")
.createWithDefault(1)
Copy link
Member

@martin-g martin-g Feb 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this mean that the driver and the executor(s) will be on the same pod ?

Copy link
Member Author

@Yikun Yikun Feb 8, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it's for driver, in the case of spot instance, we allow user only create driver pod first. So, it's equivalent to no limit by default.


val KUBERNETES_JOB_PRIORITY_CLASS_NAME = ConfigBuilder("spark.kubernetes.job.priorityClassName")
.doc("The priority of the running job. This info " +
"will be stored in configuration and passed to specified feature step.")
.version("3.3.0")
.stringConf
.createOptional

val KUBERNETES_DRIVER_LABEL_PREFIX = "spark.kubernetes.driver.label."
val KUBERNETES_DRIVER_ANNOTATION_PREFIX = "spark.kubernetes.driver.annotation."
val KUBERNETES_DRIVER_SERVICE_ANNOTATION_PREFIX = "spark.kubernetes.driver.service.annotation."
Expand Down