[SPARK-49033][CORE] Support server-side environmentVariables replacement in REST Submission API

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to support server-side environment variable replacement in REST Submission API.

• For example, ephemeral Spark clusters with server-side environment variables can provide backend-resource and information without touching client-side applications and configurations.

• The place holder pattern is {{SERVER_ENVIRONMENT_VARIABLE_NAME}} style like the following.

spark/docs/configuration.md

Line 694 in 163e512

<code>-verbose:gc -Xloggc:/tmp/{{APP_ID}}-{{EXECUTOR_ID}}.gc</code>

spark/core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala

Lines 233 to 234 in 163e512

	"org.apache.spark.deploy.worker.DriverWrapper",
	Seq("{{WORKER_URL}}", "{{USER_JAR}}", mainClass) ++ appArgs, // args to the DriverWrapper

Why are the changes needed?

A user can submits an environment variable holder like {{AWS_CA_BUNDLE}} and {{AWS_ENDPOINT_URL}} in order to use server-wide environment variables of Spark Master.

$ SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true" \
  AWS_ENDPOINT_URL=ENDPOINT_FOR_THIS_CLUSTER \
  sbin/start-master.sh

$ sbin/start-worker.sh spark://$(hostname):7077

curl -s -k -XPOST http://localhost:6066/v1/submissions/create \
  --header "Content-Type:application/json;charset=UTF-8" \
  --data '{
          "appResource": "",
          "sparkProperties": {
            "spark.master": "spark://localhost:7077",
            "spark.app.name": "",
            "spark.submit.deployMode": "cluster",
            "spark.jars": "/Users/dongjoon/APACHE/spark-merge/examples/target/scala-2.13/jars/spark-examples_2.13-4.0.0-SNAPSHOT.jar"
          },
          "clientSparkVersion": "",
          "mainClass": "org.apache.spark.examples.SparkPi",
          "environmentVariables": {
            "AWS_ACCESS_KEY_ID": "A",
            "AWS_SECRET_ACCESS_KEY": "B",
            "AWS_ENDPOINT_URL": "{{AWS_ENDPOINT_URL}}"
          },
          "action": "CreateSubmissionRequest",
          "appArgs": [ "10000" ]
  }'

• http://localhost:4040/environment/

Does this PR introduce any user-facing change?

No. This is a new feature and disabled by default via spark.master.rest.enabled (default: false)

How was this patch tested?

Pass the CIs with newly added test case.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Could you review this when you have some time, @viirya ?

[dongjoon-hyun]

Could you review this PR when you have some time, @yaooqinn ?

[viirya]

Hmm, where is this envVariablePattern used?

[dongjoon-hyun]

Oh, right. Now, it's no op. Let me clean up it.
I used it for Scala 2.12 patch.

[dongjoon-hyun]

Thank you, @viirya . It's removed.

[viirya]

Let's also update this comment?

[dongjoon-hyun]

Sure.

[dongjoon-hyun]

Thank you. Let me merge this because all CIs passed and the last two commits are only about comment and removal of unused lines. I checked the unit test result too manually.

[info] StandaloneRestSubmitSuite:
[info] - construct submit request (16 milliseconds)
[info] - create submission (479 milliseconds)
[info] - create submission with multiple masters (18 milliseconds)
[info] - create submission from main method (15 milliseconds)
[info] - kill submission (18 milliseconds)
[info] - request submission status (13 milliseconds)
[info] - create then kill (22 milliseconds)
[info] - create then request status (21 milliseconds)
[info] - create then kill then request status (27 milliseconds)
[info] - kill or request status before create (13 milliseconds)
[info] - SPARK-45819: clear (16 milliseconds)
[info] - SPARK-45843: killAll (16 milliseconds)
[info] - SPARK-46368: readyz with SC_OK (16 milliseconds)
[info] - SPARK-46368: readyz with SC_SERVICE_UNAVAILABLE (12 milliseconds)
[info] - good request paths (19 milliseconds)
[info] - good request paths, bad requests (22 milliseconds)
[info] - bad request paths (16 milliseconds)
[info] - server returns unknown fields (24 milliseconds)
[info] - client handles faulty server (23 milliseconds)
[info] - client does not send 'SPARK_ENV_LOADED' env var by default (0 milliseconds)
[info] - client does not send 'SPARK_HOME' env var by default (0 milliseconds)
[info] - client does not send 'SPARK_CONF_DIR' env var by default (1 millisecond)
[info] - SPARK-49033: Support server-side environment variable replacement in REST Submission API (2 milliseconds)
[info] - SPARK-45197: Make StandaloneRestServer add JavaModuleOptions to drivers (0 milliseconds)
[info] Run completed in 1 second, 584 milliseconds.
[info] Total number of tests run: 24
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 24, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 9 s, completed Jul 26, 2024, 9:49:45 PM

[yaooqinn]

Late LGTM, do we need a doc for this feature?

[dongjoon-hyun]

Thank you, @yaooqinn . Yes, I'll update it together. I have another PR at this area.

[yaooqinn]

@dongjoon-hyun, Thank you!