[Improvement]: Support terminal access hive table when switch to a catalog with HMS metastore .

[li36909]

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

use ArcticSparkSessionCatalog for terminal when user use hive type catalog, then at terminal we run sql for origin hive tables.

How should we improve?

there are few issues need to resolve here:

1. spark session catalog ask to use name 'spark_catalog', and at terminal we maybe need to support multiple hive catalog
2. at sql script, user maybe write the sql with the catalog name.

to resolve these two issue:
1、when setting spark conf befor start terminal session, we can replace the catalog name to spark_catalog.
2、we need to limit the user not contain the catalog name for current catalog at the sql script.

Are you willing to submit PR?
Yes I am willing to submit a PR!

Subtasks
No response

Code of Conduct
I agree to follow this project's Code of Conduct

[baiyangtx]

I suggest to change the title to

Support terminal access hive table when switch to a catalog with HMS metastore .

Because that is the real problem that we want to solve, and using ArcticSparkSessionCatalog is a feasible way to do this.

[baiyangtx]

I had thought about this feature, it's not easy to solve in mulit catalogs situation. The AMS creates the spark context in it's process and the spark engine could only create one spark context in each process, it's hard to create 2 different catalog using ArcticSparkSessionCatalog.

But on the other hand, 90% users only have one catalog, I think it is valuable to solve the feature with only one catalog.

I suggest to solve this issue in this way.

1. add a property to terminal configs to force use ArcticSparkSessionCatalog for hive.

ams.terminal.backend=local
ams.terminal.local.using-session-catalog-for-hive=true

or
ams.terminal.backend=kyuubi
ams.terminal.kyuubi.using-session-catalog-for-hive=true

2. the TerminalManager already passed the catalog.type to TerminalSessionFactory. so we can replace the ArcticSparkCatalog in the implement class.

Both KyuubiTerminalSessionFactory and LocalTerminalSessionFactory are using SparkContextUtil.getSparkConf to generate the spark configs. Replace the catalog name and implement class here.

3. in TerminalSession, ignore use {catalog} if current catalog is spark_catalog

[li36909]

thanks for your suggestions, I will try to fix it as you said