Allow connecting to multiple kernel gateway servers

[AgrawalAmey]

When we connect to kernel gateway, we loose access to local kernels and we can connect to only one kernel gateway at a time. One possible way to address this could be adding something like a UnionKernelManager which will take a list of objects implementing MappingKernelManagers. In case of conflicts/duplicates, the we can solve the kernel manager to use based on priority order.

[AgrawalAmey]

@kevin-bates this functionality should help a lot in enterprise settings where the user would have access to multiple clusters with different privileges/resources.

[kevin-bates]

Thanks for opening this issue @AgrawalAmey. I would love to be able to support multiple gateways and local kernels simultaneously but I think you'd need to tie kernel management with discovery (kernelspec management) - which is exactly the premise behind kernel providers which we hope to get into the jupyter_server 2.x or 3.x release.

With kernel providers, we can support the various gateway kernels (Hadoop YARN, Kubernetes, Docker) directly via the jupyter server. The kernel management portion of EG essentially goes away (although one would still have the ability to proxy all kernel management to a single gateway server, just not multiples, if they needed that separation).

should help a lot in enterprise settings where the user would have access to multiple clusters with different privileges/resources.

If you are targeting on-prem clusters (e.g., Hadoop YARN), EG can support multiple clusters today via the process-proxy configuration settings (within the kernelspec). Not sure if that would help you or not.

[AgrawalAmey]

@kevin-bates thanks a lot for sharing the details about kernel providers. This looks very exciting. I will play around with the test package from the PR.