[BUG] on Syndic Node

[satellite-no]

Description
Trying to setup Syndic Node on [AWS ](cpe:2.3:o:amazon:amazon_linux:2) but receiving error [ERROR ] Unable to connect pusher: Stream is closed after I accept the syndic key on the master.

Setup
Salt-master with several minions and s syndic node with 1 minion (for testing). All instances on latest rpm install 3005. All hosted in AWS with different VPC etc but able to communicated on 4505 & 4506 validated with nc.

Please be as specific as possible and give set-up details.

• VM running on a cloud service, please be explicit and add details
• classic packaging

Steps to Reproduce the behavior

• Setup syndic node on 3005 using AWS amazon_linux:2 following https://docs.saltproject.io/en/latest/topics/topology/syndic.html docs.

debug log

[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] stdout: +0000
[DEBUG   ] output: +0000
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pywintypes
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load smbprotocol
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pypsexec
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load winrm
[DEBUG   ] The functions from module 'config' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded config.merge
[DEBUG   ] SaltEvent PUB socket URI: /var/run/salt/minion/minion_event_6951c2af3b_pub.ipc
[DEBUG   ] SaltEvent PULL socket URI: /var/run/salt/minion/minion_event_6951c2af3b_pull.ipc
[ERROR   ] Unable to connect pusher: Stream is closed
Traceback (most recent call last):
  File "salt/utils/event.py", line 427, in connect_pull
    self.pusher.connect(timeout=timeout)
  File "salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "salt/ext/tornado/ioloop.py", line 459, in run_sync
    return future_cell[0].result()
  File "salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "salt/transport/ipc.py", line 345, in _connect
    yield self.stream.connect(sock_addr)
  File "salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
[DEBUG   ] Persisting schedule

Expected behavior

Syndic node connects like documented.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3005

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.9.13 (main, Aug 23 2022, 18:33:16)
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: amzn 2
        locale: utf-8
       machine: x86_64
       release: 5.10.135-122.509.amzn2.x86_64
        system: Linux
       version: Amazon Linux 2

Additional context
Add any other context about the problem here.

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[davama]

I got the same issue when i upgraded from 3004.2 to 3005.1

edit:
reverted a lower master to 3004.2 but kept the MoM at 3005.1 and syndic is talking to mom.

[viktordaniel]

Possible duplicate of #62577 ?

[davama]

Looks like PR exists for this

#63257