Connections and sessions managment in HA mode

[VladOrlov]

Kyuubi's doc HA contains this information:
With Kyuubi Hive JDBC Driver or vanilla Hive JDBC Driver, a client can specify service discovery mode in JDBC connection string, i.e. serviceDiscoveryMode=zooKeeper; and set zooKeeperNamespace=kyuubi;, then it can randomly pick one of the Kyuubi service uris from the specified ZooKeeper addresses in the /kyuubi path.

So each new connection from one user will be randomly redirected to available Kyuubi service and new session will be created?

Also we have connections issue in HA mode on Kubernetes. Zookeeeper contains pods' internal IP so clieant can't reach Kyuubi. Could you please explaine how correctly setup Kyuubi on Kubernetes in HA mode?

[pan3793]

So each new connection from one user will be randomly redirected to available Kyuubi service and new session will be created?

Yes.

Zookeeeper contains pods' internal IP so clieant can't reach Kyuubi.

Maybe #5015 can answer your question.

Further, you are not enforced to use ZK for Kyuubi Server HA, Ingress/LoadBalance are fine too. (Note that, ZK is still required for engine discovery, in this case, it's only used for Kyuubi internally, assume your Kyuubi server and Spark runs on the same K8s cluster, should not have network reachable issues.)

[hanna-liashchuk]

In this case, share level USER is "half" working? Since you have to be on the same machine to share the engine and zookeeper not necesarily will redirect you to the same server. Meaning - having 3 servers and 3 connects with share level user can result in 3 separate engines, right?

[hanna-liashchuk]

but what if kyuubi servers are in k8s and spark is in k8s. in this case an engine is registered in the same pod as server.
If there is a zookeeper (not embeded) and user connects with serviceDiscoveryMode=zooKeeper and share level USER, it will always go to the same server?

[pan3793]

in this case an engine is registered in the same pod as server.

It's only true in spark client mode. But it does not matter, even though the engine(Spark driver) runs in server1's pod, it still could be connected from server2 as long as the server2 can find the engine's connection info from zk,

it will always go to the same server

No, randomly pick one. But all Kyuubi servers could see the registered engine in ZK.

[hanna-liashchuk]

It's only true in spark client mode.

Is there another option for an interactive session?

Soo, let me wrap it up. A user will fo to Zookeeper, Zookeeper will send the user to a random Kyuubi server, the server will see that the user has share level USER and it will know the address of already available engine and send the user to that server?
i.g. Client -> Zookeeper -> Kyuubi Server 0 -> Kyuubi Server 1

[pan3793]

Is there another option for an interactive session?

Kyuubi supports all modes that Spark supported. including local, client, cluster.

Client -> Zookeeper -> Kyuubi Server 0 -> Kyuubi Server 1

No. Zookeeper is not only used for Server register, but also engine register, the key point here is the routing/communication mechanism between Kyuubi Server and Kyuubi Engine(Spark Driver)