[SUB-TASK][KPIP-2] Support Flink SQL Client startup options.

[a49a]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

The following issue only achieves FlinkSQLEngine process, but users have no way to set --jar and --library options in Kyuubi. We should achieve this in KyuubiServer process too.

#1685
SteNicholas@dd5a704

We can put --jar and --library options in kyuuubi-default.conf, and KyuubiServer process load these options.

How should we improve?

KyuubiServer process loads the Flink SQL Client startup options in kyuuubi-default.conf. And then KyuubiServer transforms the options to standard Flink SQL Client startup options and passes them to flink-sql-engine.sh when the flink-sql-engine.sh process will be started.

The kyuuubi-default.conf options like the following.

flink.client.option.jar foo.jar
flink.client.option.library /foo/bar

Now flink-sql-engine.sh startup options are nothing, so users can't set any options.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[a49a]

The FlinkSQLEngine process life is inconsistent with Session. One FlinkSQLEngine process corresponds to multiple sessions. So flink-sql-engine.sh can not use configuration of Session or ExecuteStatement.

[pan3793]

The FlinkSQLEngine process life is inconsistent with Session. One FlinkSQLEngine process corresponds to multiple sessions. So flink-sql-engine.sh can not use configuration of Session or ExecuteStatement.

From Flink document, Flink support ADD JAR and REMOVE JAR commands, what's the difference between use ADD JAR and inject jars on launching Flink engine?

[pan3793]

The kyuuubi-default.conf options like the following.

flink.client.option.jar foo.jar
flink.client.option.library /foo/bar

Give some comments based on my little understand of Flink, please correct me if I'm wrong.

There is no equality configuration in Flink of --jar and --library options defined by Flink CLI, if yes we'd better to introduce Kyuubi's style configurations instead of Flink's style to avoid potential conflict, e.g.

kyuubi.session.engine.flink.jar
kyuubi.session.engine.flink.library

[pan3793]

https://github.com/apache/incubator-kyuubi/blob/3eb1fa9e48b4c533f05b3719486df198c6e29751/externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkSQLEngine.scala#L79

BTW, I think we can modify this line to support overwriting Flink configuration by flink-sql-engine.sh, then we can achieve same experience like Spark engine, which means we can define Flink confs in JDBC URL, kyuubi-defaults.conf, flink-conf.yaml, and the former has higher priority.

[a49a]

https://github.com/apache/incubator-kyuubi/blob/3eb1fa9e48b4c533f05b3719486df198c6e29751/externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkSQLEngine.scala#L79

BTW, I think we can modify this line to support overwriting Flink configuration by flink-sql-engine.sh, then we can achieve same experience like Spark engine, which means we can define Flink confs in JDBC URL, kyuubi-defaults.conf, flink-conf.yaml, and the former has higher priority.

I spilled a new issue, I will achieve this first.

[a49a]

The kyuuubi-default.conf options like the following.

flink.client.option.jar foo.jar
flink.client.option.library /foo/bar

Give some comments based on my little understand of Flink, please correct me if I'm wrong.

There is no equality configuration in Flink of --jar and --library options defined by Flink CLI, if yes we'd better to introduce Kyuubi's style configurations instead of Flink's style to avoid potential conflict, e.g.

kyuubi.session.engine.flink.jar
kyuubi.session.engine.flink.library

I second with introducing Kyuubi's style configurations, but this has nothing to do with session.

Could we design prefix is kyuubi.engine.flink.?

kyuubi.engine.flink.jar
kyuubi.engine.flink.library

[a49a]

https://issues.apache.org/jira/browse/FLINK-27010