Cannot Connect to Jupyterhub Remote Kernel from Remote SSH Session

[matrix303]

Environment data

• VS Code version: 1.63.0-insider Commit: c1f2f40bcd81380bf2e286af64becddada57db88
• Jupyter Extension version (available under the Extensions sidebar): v2021.11.1001498571
• Python Extension version (available under the Extensions sidebar): v2021.11.1422169775
• OS (Windows | Mac | Linux distro) and version: Mac Monterery 12.0.1
• Python and/or Anaconda version: 3.9.6
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
• Jupyter server running: Local | Remote | N/A

Expected behaviour

Connect to Remote server via remote ssh --> Open NB on server -> connect to Jupyterhub Remote Session (running on same server)

Actual behaviour

Once opened NB on server, cannot connect to the jupyterhub remote session.

Steps to reproduce:

[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

Steps as indicated above

The token URL works if: instead of connecting to remove server via remote ssh, open local NB and just connect to remote kernel.

Also, once connected to remote ssh session, using the terminal, I have ensured I am able to PING the jupyterhub server.

Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

NOTE: REPLACED SENSITIVE SERVER AND USER DETAILS WITH !!SERVER!! and !!USER!!

Jupyter Extension Version: 2021.11.1001498571.
Python Extension Verison: 2021.11.1422169775.
Info 09:37:34: Experimentation service retrieved: [object Object]
User belongs to experiment group 'jupyterTestcf'
User belongs to experiment group 'jupyterEnhancedDataViewer'
Info 09:37:35: Checking for server existence.
Info 09:37:35: Checking for server usability.
Info 09:37:35: Starting notebook server.
Info 09:37:35: Connecting to server
Info 09:37:35: Connecting to process server
Info 09:37:35: Connecting server kernel https://!!SERVER!!/user/!!USER!!/
Connecting to Jupyter server at https://!!SERVER!!/user/!!USER!!/
Info 09:37:35: Creating server with settings : {"baseUrl":"https://!!SERVER!!/user/!!USER!!/","appUrl":"","wsUrl":"wss://!!SERVER!!/user/!!USER!!/","token":"!!TKN!!","appendToken":true,"init":{"cache":"no-store","credentials":"same-origin"}}
Info 09:37:35: Connection complete server
Info 09:37:35: Server started.
Info 09:37:35: Creating server with settings : {"baseUrl":"https://!!SERVER!!/user/!!USER!!/","appUrl":"","wsUrl":"wss://!!SERVER!!/user/!!USER!!/","token":"!!TKN!!","appendToken":true,"init":{"cache":"no-store","credentials":"same-origin"}}
Info 09:37:35: Creating server with settings : {"baseUrl":"https://!!SERVER!!/user/!!USER!!/","appUrl":"","wsUrl":"wss://!!SERVER!!/user/!!USER!!/","token":"!!TKN!!","appendToken":true,"init":{"cache":"no-store","credentials":"same-origin"}}
Info 09:37:35: Experiment status for python is {"enabled":true,"optInto":[],"optOutFrom":[]}
Info 09:37:35: Process Execution: > ~/envs/py39/bin/python -c "import notebook"
> ~/envs/py39/bin/python -c "import notebook"
Info 09:37:36: Process Execution: > ~/envs/py39/bin/python -c "import jupyter"
> ~/envs/py39/bin/python -c "import jupyter"
Info 09:37:36: Registering dummy command feature
Info 09:37:36: Process Execution: > ~/envs/py39/bin/python -m jupyter kernelspec --version
> ~/envs/py39/bin/python -m jupyter kernelspec --version
Error 09:37:55: SessionManager cannot enumerate kernelspecs. Returning default {}.
Error 09:37:55: SessionManager cannot enumerate kernelspecs. Returning default {}.
Info 09:39:42: Disposing session manager
Info 09:39:42: SessionManager - dispose contents manager
Info 09:39:42: ShutdownSessionAndConnection - dispose session manager
Info 09:39:42: Disposing session manager
Info 09:39:42: SessionManager - dispose contents manager
Info 09:39:42: ShutdownSessionAndConnection - dispose session manager
Info 09:39:42: Finished disposing jupyter session manager
Info 09:39:42: Finished disposing jupyter session manager
Error 09:39:42: Error loading notebook controllers [FetchError: request to https://!!SERVER!!/user/!!USER!!/api/kernels?1637764655336 failed, reason: connect ETIMEDOUT 132.219.136.9:443
	at ClientRequest.<anonymous> (/home/!!USER!!/.vscode-server-insiders/extensions/ms-toolsai.jupyter-2021.11.1001498571/out/client/extension.js:16:343264)
	at ClientRequest.emit (events.js:315:20)
	at TLSSocket.socketErrorListener (_http_client.js:469:9)
	at TLSSocket.emit (events.js:315:20)
	at emitErrorNT (internal/streams/destroy.js:106:8)
	at emitErrorCloseNT (internal/streams/destroy.js:74:3)
	at processTicksAndRejections (internal/process/task_queues.js:80:21)]
Error 09:39:42: Failed to find & set preferred controllers [FetchError: request to https://!!SERVER!!/user/!!USER!!/api/kernels?1637764655336 failed, reason: connect ETIMEDOUT 132.219.136.9:443
	at ClientRequest.<anonymous> (/home/!!USER!!/.vscode-server-insiders/extensions/ms-toolsai.jupyter-2021.11.1001498571/out/client/extension.js:16:343264)
	at ClientRequest.emit (events.js:315:20)
	at TLSSocket.socketErrorListener (_http_client.js:469:9)
	at TLSSocket.emit (events.js:315:20)
	at emitErrorNT (internal/streams/destroy.js:106:8)
	at emitErrorCloseNT (internal/streams/destroy.js:74:3)
	at processTicksAndRejections (internal/process/task_queues.js:80:21)]
Info 09:43:48: Notebook Closed ~/!!file.ipynb!!
Error 09:43:56: Failed to find & set preferred controllers [FetchError: request to https://!!SERVER!!/user/!!USER!!/api/kernels?1637764655336 failed, reason: connect ETIMEDOUT 132.219.136.9:443
	at ClientRequest.<anonymous> (/home/!!USER!!/.vscode-server-insiders/extensions/ms-toolsai.jupyter-2021.11.1001498571/out/client/extension.js:16:343264)
	at ClientRequest.emit (events.js:315:20)
	at TLSSocket.socketErrorListener (_http_client.js:469:9)
	at TLSSocket.emit (events.js:315:20)
	at emitErrorNT (internal/streams/destroy.js:106:8)
	at emitErrorCloseNT (internal/streams/destroy.js:74:3)
	at processTicksAndRejections (internal/process/task_queues.js:80:21)]
Info 09:43:56: Creating daemon pool for ~/envs/py39/bin/python with env variables count 159
Info 09:43:56: Process Execution: > ~/envs/py39/bin/python -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
> ~/envs/py39/bin/python -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
Info 09:43:56: Process Execution: > ~/envs/py39/bin/python -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
> ~/envs/py39/bin/python -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
Info 09:43:56: Process Execution: > ~/envs/py39/bin/python -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
> ~/envs/py39/bin/python -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v

[IanMatthewHuff]

@matrix303 Thanks for the report here. I believe this is a duplicate of this issue here:
#8134
Where we have a general connection issue to jupyter hub. I'm going to close this issue as it should be covered by the fix for the above issue when that happens.