Kyuubi Image dependency on Spark Image

[fizaashraf37]

Is it possible to run kyuubi server on kubernetes without baking the spark image inside the kyuubi image? I cannot find any documentation about this. Neither it is mentioned anywhere that spark needs to be baked inside kyuubi image.
We want to keep the kyuubi image and our custom built spark image independent as we frequently need to update the spark image and we do not want the kyuubi image to be dependent on this. Our priority is to use the official kyuubi image without spark in this and use our custom spark image outside the kyuubi image to start spark apps in cluster mode using kyuubi.

[pan3793]

Kyuubi uses spark-submit to bootstrap the Spark application if necessary(you can find the whole spark-submit command in Kyuubi's log). So basically, if you write a shell script (or any executable program) which fully compatible with the spark-submit, and you can do anything you want under the hood, for example:

1. retrieve the spark image from an external config service, and bootstrap the application in your custom way
2. forward this command to another external service that can complete the Spark application bootstrap

[fizaashraf37]

We do not want to use any custom scripts. I think we can utilize kyuubi official image with spark to bootstrap spark applications and our custom spark image for spark applications. But we are facing one problem with this. When we use the --packages parameter in spark-submit command for required jars. These jars are not placed in spark/jars folder. Although these can be placed manually which means there is no permission issue on this folder.

[pan3793]

Using a different spark binary for spark-submit and image is feasible in some cases, but is not encouraged, adoption of this approach at your own risk

[fizaashraf37]

Right. What is the standard practice. Do kyuubi recommends to bake custom spark image inside the kyuubi image?

[pan3793]

most users use Spark image as the base image to build Kyuubi image. some advanced users use a fake spark-submit to download the latest spark binary from an external service(there are some strategies, e.g. choose a different version based on the tenant), then forward the bootstrap command to real spark-submit to achieve the hotfix ability without restarting Kyuubi Pod.

[fizaashraf37]

Alright. Thanks. I think we are good with base image in kyuubi at the moment.