duplicate entries for key [name="job-files"]

[GentleGhostCoder]

Affected version

0.3.0

Current and expected behavior

I'm trying to run an example Python script print("hello world")
and get an error in the operator logs.
Maybe I forgot a prerequirements step? 🤔

---
apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
metadata:
  name: spark-example
  namespace: default
spec:
  version: "1.0"
  # everything under /jobs will be copied to /stackable/spark/jobs
  image: docker.stackable.tech/stackable/ny-tlc-report:0.1.0
  sparkImage: docker.stackable.tech/stackable/pyspark-k8s:3.3.0-stackable0.1.0
  sparkImagePullPolicy: IfNotPresent
  mode: cluster
  mainApplicationFile: local:///opt/hello_world.py
  deps:
    requirements: []
  # sparkConf: {}
  # volumes: 
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    # volumeMounts:
    #   - name: job-deps
    #     mountPath: /dependencies
  executor:
    cores: 1
    instances: 3
    memory: "512m"
    # volumeMounts:
    #   - name: job-deps
    #     mountPath: /dependencies

2022-08-02T07:34:19.419636Z  INFO app_controller:reconciling object{object.ref=SparkApplication.v1alpha1.spark.stackable.tech/spark-ny-image.spark object.reason=error policy requested retry}: stackable_spark_k8s_operator::spark_k8s_controller: Starting reconcile
2022-08-02T07:34:19.427931Z ERROR app_controller:reconciling object{object.ref=SparkApplication.v1alpha1.spark.stackable.tech/spark-ny-image.spark object.reason=error policy requested retry}: kube_client::client::builder: failed with status 500 Internal Server Error
2022-08-02T07:34:19.428001Z ERROR app_controller: stackable_operator::logging::controller: Failed to reconcile object controller.name="sparkapplications.spark.stackable.tech" error=reconciler for object SparkApplication.v1alpha1.spark.stackable.tech/spark-ny-image.spark failed error.sources=[failed to apply Job, Kubernetes reported error: ApiError: failed to create typed patch object (spark/spark-ny-image; batch/v1, Kind=Job): errors:
  .spec.template.spec.volumes: duplicate entries for key [name="job-files"]
  .spec.template.spec.containers[name="spark-submit"].volumeMounts: duplicate entries for key [mountPath="/stackable/spark/jobs"]:  (ErrorResponse { status: "Failure", message: "failed to create typed patch object (spark/spark-ny-image; batch/v1, Kind=Job): errors:\n  .spec.template.spec.volumes: duplicate entries for key [name=\"job-files\"]\n  .spec.template.spec.containers[name=\"spark-submit\"].volumeMounts: duplicate entries for key [mountPath=\"/stackable/spark/jobs\"]", reason: "", code: 500 }), ApiError: failed to create typed patch object (spark/spark-ny-image; batch/v1, Kind=Job): errors:
  .spec.template.spec.volumes: duplicate entries for key [name="job-files"]
  .spec.template.spec.containers[name="spark-submit"].volumeMounts: duplicate entries for key [mountPath="/stackable/spark/jobs"]:  (ErrorResponse { status: "Failure", message: "failed to create typed patch object (spark/spark-ny-image; batch/v1, Kind=Job): errors:\n  .spec.template.spec.volumes: duplicate entries for key [name=\"job-files\"]\n  .spec.template.spec.containers[name=\"spark-submit\"].volumeMounts: duplicate entries for key [mountPath=\"/stackable/spark/jobs\"]", reason: "", code: 500 }), failed to create typed patch object (spark/spark-ny-image; batch/v1, Kind=Job): errors:
  .spec.template.spec.volumes: duplicate entries for key [name="job-files"]
  .spec.template.spec.containers[name="spark-submit"].volumeMounts: duplicate entries for key [mountPath="/stackable/spark/jobs"]: ]

Possible solution

No response

Additional context

The aim is actually to get the following script to run.
https://github.com/commoncrawl/cc-webgraph/blob/main/src/script/hostgraph/build_hostgraph.sh
(The following scripts still need this -> https://github.com/commoncrawl/cc-pyspark)
It would be great if you could help me with this as well.

Environment

No response

Would you like to work on fixing this bug?

No response

[razvan]

Hey semmjon,
thank you for your report.
This error should have been fixed with #92.
Are you sure you are using the 0.3.0 version of the operator ?

[adwk67]

Hallo Semjon, here is a quick update:

• it looks as though fix PySpark tests #92 didn't actually make the release 0.3.0 so we are going to do a new release which includes it
• the documentation does not make it entirely clear that local when used in the path for mainApplicationFile means local-to-the-image i.e. a "local" path will be one that is valid in the image docker.stackable.tech/stackable/pyspark-k8s:3.3.0-stackable0.1.0. This is one reason why the user can supply a separate image used to copy over such resources on intialization before spark-submit is called.
• the common-crawl script you mention above states that it has only been tested against hadoop (+ yarn). Are you able to extract a hard-coded spark-submit command in the form that you require? Then we could take a look at how best to define a custom resource that will produce the right job definition for different inputs

[razvan]

We released version 0.4.0 of the operator where this bug has been fixed.

[GentleGhostCoder]

Many thanks for the fix. I will test it.
That with the common crawl script has done :) (no longer necessary)

[razvan]

Hey @semmjon , was your test successful ? Can we close this issue?

[GentleGhostCoder]

Yes thank you :-)