Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZMQ Error #140

Closed
skeltoh opened this issue Oct 28, 2021 · 13 comments
Closed

ZMQ Error #140

skeltoh opened this issue Oct 28, 2021 · 13 comments

Comments

@skeltoh
Copy link

skeltoh commented Oct 28, 2021

When I go to start an autopilot pilot, it starts and says:

fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git

Followed by the splash screen. Followed by:

Process Pilot_Station-2:
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/home/pi/git/autopilot/autopilot/networking/station.py", line 173, in run
self.listener.setsockopt_string(zmq.IDENTITY, self.id)
File "/home/pi/.venv/autopilot/lib/python3.7/site-packages/zmq/sugar/socket.py", line 340, in set_string
return self.set(option, optval.encode(encoding))
File "zmq/backend/cython/socket.pyx", line 420, in zmq.backend.cython.socket.Socket.set
File "zmq/backend/cython/socket.pyx", line 277, in zmq.backend.cython.socket._setsockopt
File "zmq/backend/cython/checkrc.pxd", line 28, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: Invalid argument

The process doesn't terminate, it stays there, but it doesn't connect to the terminal, which I'm assuming is due to the last error.

It looks like self.id is set to an empty string in station.py. Did I need to set the local Pilot IP somewhere?

@sneakers-the-rat
Copy link
Contributor

hmm, anything else in the traceback? from this I can't see what it might be, but I'll try and see if there's anyplace where the expectation of installing from git repo isn't handled

@sneakers-the-rat
Copy link
Contributor

wait is this on WSL? one thing is that pyzmq and libzmq can be picky about Unicode/bytes formatting and it has gone back and forth, if it's handled differently by windows that's interesting and can be worked around

@cxrodgers
Copy link
Contributor

That error happens when you call "git pull" or any git command from a directory that isn't a git repository. I wonder what is calling "git" and triggering that error. I wouldn't think the Pilot would do that. Maybe something about the environment?

Whatever it is, looks like it's going into REPO_DIR/.git instead of into REPO_DIR, where REPO_DIR is (probably) the autopilot repository. I bet it's some kind of path thing that windows handles differently, like backslashes or something like that.

The ZMQ error might be similar, like a difference in the way windows formats network resources.

Sorry I can't be of more help, good luck!

@skeltoh
Copy link
Author

skeltoh commented Oct 28, 2021

Thanks for your help! Just to clarify, this is all happening on a Pilot on a pi running raspbian, only my terminal is on WSL.

I'm going to try on a clean install. What config (version of raspbian, python, etc) is best tested?

On that note, whats the best tested setup for the terminal? Ill probably switch from WSL to a Linux vm if I run into any issues with the terminal.

@cxrodgers
Copy link
Contributor

For the Pi, I use the most recent release of Raspberry Pi OS Lite, and I believe that is the official recommendation. The python version should be 3.7.

I contributed to the installation docs a little while ago. https://docs.auto-pi-lot.com/en/latest/guide/installation.html
It should be possible to work through those instructions from a clean install, and get a working Pilot. But if that's not the case, or if anything is not clear, let me know!

@skeltoh
Copy link
Author

skeltoh commented Oct 28, 2021

I figured out the problem. I wasn't naming the pilot. I was able to fix the ZMQ error, but it looks like it's still not connecting.

I'm following those instructions, and installing with pip. For now, I'm only doing a few things on the setup:

Environment Configuration:
[ SUCCESS ] env_pilot
[ SUCCESS ] pigpiod
[ SUCCESS ] virtualenv detected, path: /home/pi/.venv/autopilot

Additional Messages: Launch file created at /home/pi/autopilot/launch_autopilot.sh

After running the launch script, here's what I run into:

jackd: no process found
I'm assuming this doesn't matter if I'm not using audio
pigpiod: no process found
Odd, considering success above
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/pi/.venv/autopilot/lib/python3.7/site-packages/autopilot/core/pilot.py", line 811, in
a = Pilot()
File "/home/pi/.venv/autopilot/lib/python3.7/site-packages/autopilot/core/pilot.py", line 143, in init
with open(os.path.join(os.path.dirname(os.path.dirname(file)), 'setup', 'welcome_msg.txt'), 'r') as welcome_f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/pi/.venv/autopilot/lib/python3.7/site-packages/autopilot/setup/welcome_msg.txt'

If I download the repo and manually cp autopilot/autopilot/setup/welcome_msg.txt /home/pi/.venv/autopilot/lib/python3.7/site-packages/autopilot/setup/welcome_msg.txt, the last error goes away, and I get:

jackd: no process found
pigpiod: no process found
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
(Splashscreen)
WARNING:networking.station_02.Pilot_Station.wheel-2:PUBLISH FAILED wheel-2_0 - ID: wheel-2_0; TO: T; SENDER: wheel-2; KEY: STATE; FLAGS: {}; VALUE: IDLE

@skeltoh
Copy link
Author

skeltoh commented Oct 28, 2021

Below is the complete log when I try to run the pilot. If I run this from the pilot:

telnet TERMINALIP 5560

It connects.
If I run this from the terminal:

telnet PILOTIP 5565

It also works.

DEBUG:core.pilot.Pilot.wheel-2:pilot networking initialized
DEBUG:core.pilot.Pilot.wheel-2:pullups and pulldowns set
DEBUG:networking.node.Net_Node._wheel-2:MESSAGE SENT - ID: _wheel-2_1; TO: T; SENDER: _wheel-2; KEY: HANDSHAKE; FLAGS: {}; VALUE: {'pilot': 'wheel-2', 'ip': '170.140.233.251', 'state': 'IDLE', 'prefs': {'AGENT': 'PILOT', 'AUDIOSERVER': False, 'AUTOPLUGIN': False, 'BASEDIR': '/home/pi/autopilot', 'CALIBRATIONDIR': '/home/pi/autopilot/calibration', 'CHILDID': [], 'CONFIG': '', 'DATADIR': '/home/pi/autopilot/data', 'FS': 192000, 'HARDWARE': {}, 'HASH': 'd352feef3b8aac8699fbc4c27270580d0a48ab0f', 'JACKDSTRING': 'jackd -P75 -p16 -t2000 -dalsa -dhw:sndrpihifiberry -P -rfs -n3 -s &', 'LINEAGE': 'NONE', 'LOGDIR': '/home/pi/autopilot/logs', 'LOGLEVEL': 'DEBUG', 'LOGNUM': 4, 'LOGSIZE': 5242880, 'MSGPORT': 5565, 'NAME': 'wheel-2', 'NCHANNELS': 1, 'OUTCHANNELS': '', 'PARENTID': '', 'PARENTIP': '', 'PARENTPORT': '', 'PIGPIOARGS': '-t 0 -l', 'PIGPIOMASK': '1111110000111111111111110000', 'PING_INTERVAL': 5, 'PLUGINDIR': '/home/pi/autopilot/plugins', 'PLUGIN_DB': '/home/pi/autopilot/plugin_db.json', 'PROTOCOLDIR': '/home/pi/autopilot/protocols', 'PULLDOWNS': '', 'PULLUPS': '', 'PUSHPORT': 5560, 'REPODIR': '/home/pi/source/autopilot', 'SOUNDDIR': '/home/pi/autopilot/sounds', 'TERMINALIP': '170.140.233.137', 'VENV': '/home/pi/.venv/autopilot', 'VIZDIR': '/home/pi/autopilot/viz'}}
DEBUG:core.pilot.Pilot.wheel-2:handshake sent
DEBUG:networking.station_02.Pilot_Station.wheel-2:RECEIVED: ID: _wheel-2_0; TO: wheel-2; SENDER: _wheel-2; KEY: STATE; FLAGS: {'NOLOG': True}; VALUE: IDLE
DEBUG:networking.station_02.Pilot_Station.wheel-2:MESSAGE PUSHED - ID: wheel-2_0; TO: T; SENDER: wheel-2; KEY: STATE; FLAGS: {}; VALUE: IDLE
DEBUG:networking.station_02.Pilot_Station.wheel-2:MESSAGE SENT - ID: wheel-2_1; TO: _wheel-2; SENDER: wheel-2; KEY: CONFIRM; FLAGS: {}; VALUE: _wheel-2_0
DEBUG:networking.station_02.Pilot_Station.wheel-2:FORWARDING (dealer): [b'_wheel-2', b'wheel-2', b'T', b'{"id": "_wheel-2_1", "to": "T", "sender": "_wheel-2", "key": "HANDSHAKE", "value": {"pilot": "wheel-2", "ip": "170.140.233.251", "state": "IDLE", "prefs": {"AGENT": "PILOT", "AUDIOSERVER": false, "AUTOPLUGIN": false, "BASEDIR": "/home/pi/autopilot", "CALIBRATIONDIR": "/home/pi/autopilot/calibration", "CHILDID": [], "CONFIG": "", "DATADIR": "/home/pi/autopilot/data", "FS": 192000, "HARDWARE": {}, "HASH": "d352feef3b8aac8699fbc4c27270580d0a48ab0f", "JACKDSTRING": "jackd -P75 -p16 -t2000 -dalsa -dhw:sndrpihifiberry -P -rfs -n3 -s &", "LINEAGE": "NONE", "LOGDIR": "/home/pi/autopilot/logs", "LOGLEVEL": "DEBUG", "LOGNUM": 4, "LOGSIZE": 5242880, "MSGPORT": 5565, "NAME": "wheel-2", "NCHANNELS": 1, "OUTCHANNELS": "", "PARENTID": "", "PARENTIP": "", "PARENTPORT": "", "PIGPIOARGS": "-t 0 -l", "PIGPIOMASK": "1111110000111111111111110000", "PING_INTERVAL": 5, "PLUGINDIR": "/home/pi/autopilot/plugins", "PLUGIN_DB": "/home/pi/autopilot/plugin_db.json", "PROTOCOLDIR": "/home/pi/autopilot/protocols", "PULLDOWNS": "", "PULLUPS": "", "PUSHPORT": 5560, "REPODIR": "/home/pi/source/autopilot", "SOUNDDIR": "/home/pi/autopilot/sounds", "TERMINALIP": "170.140.233.137", "VENV": "/home/pi/.venv/autopilot", "VIZDIR": "/home/pi/autopilot/viz"}}, "timestamp": "2021-10-28T16:52:36.403114", "flags": {}, "changed": false, "ttl": 2}']
DEBUG:networking.node.Net_Node._wheel-2:CONFIRMED MESSAGE _wheel-2_0
DEBUG:networking.node.Net_Node._wheel-2:RECEIVED: ID: wheel-2_1; TO: _wheel-2; SENDER: wheel-2; KEY: CONFIRM; FLAGS: {}; VALUE: _wheel-2_0
DEBUG:networking.station_02.Pilot_Station.wheel-2:REPUBLISH wheel-2_0 - ID: wheel-2_0; TO: T; SENDER: wheel-2; KEY: STATE; FLAGS: {}; VALUE: IDLE
DEBUG:networking.station_02.Pilot_Station.wheel-2:REPUBLISH wheel-2_0 - ID: wheel-2_0; TO: T; SENDER: wheel-2; KEY: STATE; FLAGS: {}; VALUE: IDLE
WARNING:networking.station_02.Pilot_Station.wheel-2:PUBLISH FAILED wheel-2_0 - ID: wheel-2_0; TO: T; SENDER: wheel-2; KEY: STATE; FLAGS: {}; VALUE: IDLE

@sneakers-the-rat
Copy link
Contributor

On that note, whats the best tested setup for the terminal? Ill probably switch from WSL to a Linux vm if I run into any issues with the terminal.

Supported systems are here! https://docs.auto-pi-lot.com/en/latest/guide/installation.html#supported-systems

I figured out the problem. I wasn't naming the pilot. I was able to fix the ZMQ error, but it looks like it's still not connecting.

Aha! OK added a check so that error message is a bit more informative - 35113b8

FileNotFoundError: [Errno 2] No such file or directory: '/home/pi/.venv/autopilot/lib/python3.7/site-packages/autopilot/setup/welcome_msg.txt'

this is embarassing... should be included in the sdist because it's in the MANIFEST.in but also added a quiet exception handler because it's really just a novelty thing :p - d547bcc

The rest of the problem looks like it could be a simple thing or a challenging thing. Due to some unfortunate decisions (that need to be unmade), Pilot agents by default send messages to another agent named 'T' for terminal:
https://github.com/wehr-lab/autopilot/blob/d352feef3b8aac8699fbc4c27270580d0a48ab0f/autopilot/networking/station.py#L1040

The Pilot_Station is forwarding the message from the Net_Node onto the terminal, and so it doesn't spend a lot of time serializing and unserializing message forwards it differently depending on who it's to. I can parse it for you if that would be helpful, but the relevant code is here: https://github.com/wehr-lab/autopilot/blob/d352feef3b8aac8699fbc4c27270580d0a48ab0f/autopilot/networking/station.py#L544-L570

tl;dr the terminal agent needs to be named 'T'

If that's not the problem we're probably in the wilde west of networking. From the configuration sent in the message, it looks like you're pushing a message to a non-local IPv4 address. That can be challenging depending on your network configuration -- eg. if you're in a university network, i've had real problems trying to get past the protected silos they put different IP ranges in, if you're behind a NAT same thing. I typically will use a switch or a router and have them directly connected with local IPs, but ofc arbitrary IPs should be supported -- it's just a little tricky!

One thing that would be helpful is copy/pasting prefs.json from the terminal and Pi :)

@skeltoh
Copy link
Author

skeltoh commented Oct 28, 2021

Thanks! As it turns out, my git error messages and the splash screen thing were due to REPODIR being set to the install location used by pip. If I clone the repository and set REPODIR to that (or install using pip -e) it seems to work fine.

I changed the terminal name to "T", unfortunately that doesn't seem to change the networking behavior.

They are already on the same switch, so I will trying using a separate router instead of the university network - but they are on the same IP range and there appear to be no limits in being able to ping/telnet/ssh/SMB etc each other including on the autopilot ports.

prefs.json from the Pi is:

{
"AGENT": "PILOT",
"AUDIOSERVER": false,
"AUTOPLUGIN": false,
"BASEDIR": "/home/pi/autopilot",
"CALIBRATIONDIR": "/home/pi/autopilot/calibration",
"CHILDID": [],
"CONFIG": "",
"DATADIR": "/home/pi/autopilot/data",
"FS": 192000,
"HARDWARE": {},
"HASH": "d352feef3b8aac8699fbc4c27270580d0a48ab0f",
"JACKDSTRING": "jackd -P75 -p16 -t2000 -dalsa -dhw:sndrpihifiberry -P -rfs -n3 -s &",
"LINEAGE": "NONE",
"LOGDIR": "/home/pi/autopilot/logs",
"LOGLEVEL": "DEBUG",
"LOGNUM": 4,
"LOGSIZE": 5242880,
"MSGPORT": 5565,
"NAME": "wheel-2",
"NCHANNELS": 1,
"OUTCHANNELS": "",
"PARENTID": "",
"PARENTIP": "",
"PARENTPORT": "",
"PIGPIOARGS": "-t 0 -l",
"PIGPIOMASK": "1111110000111111111111110000",
"PING_INTERVAL": 5,
"PLUGINDIR": "/home/pi/autopilot/plugins",
"PLUGIN_DB": "/home/pi/autopilot/plugin_db.json",
"PROTOCOLDIR": "/home/pi/autopilot/protocols",
"PULLDOWNS": "",
"PULLUPS": "",
"PUSHPORT": 5560,
"REPODIR": "/home/pi/source/autopilot",
"SOUNDDIR": "/home/pi/autopilot/sounds",
"TERMINALIP": "170.140.233.137",
"VENV": "/home/pi/.venv/autopilot",
"VIZDIR": "/home/pi/autopilot/viz"
}

and from the terminal is

{
"AGENT": "TERMINAL",
"AUTOPLUGIN": true,
"BASEDIR": "/home/lab/autopilot",
"CALIBRATIONDIR": "/home/lab/autopilot/calibration",
"CONFIG": "",
"DATADIR": "/home/lab/autopilot/data",
"DRAWFPS": 20,
"LOGDIR": "/home/lab/autopilot/logs",
"LOGLEVEL": "DEBUG",
"LOGNUM": 4,
"LOGSIZE": 5242880,
"MSGPORT": 5565,
"NAME": "T",
"PILOT_DB": "/home/lab/autopilot/pilot_db.json",
"PLUGINDIR": "/home/lab/autopilot/plugins",
"PLUGIN_DB": "/home/lab/autopilot/plugin_db.json",
"PROTOCOLDIR": "/home/lab/autopilot/protocols",
"PUSHPORT": 5560,
"REPODIR": "/home/lab/git/autopilot",
"SOUNDDIR": "/home/lab/autopilot/sounds",
"TERMINALIP": "170.140.233.137",
"TERMINAL_CUSTOM_SIZE": [
0,
0,
1000,
400
],
"TERMINAL_SETTINGS_FN": "/home/lab/autopilot/terminal.conf",
"TERMINAL_WINSIZE_BEHAVIOR": "remember",
"VENV": "/home/lab/mambaforge/envs/autopilot/",
"VIZDIR": "/home/lab/autopilot/viz"
}

@skeltoh
Copy link
Author

skeltoh commented Oct 28, 2021

I just realized - my PUSHPORT and MSGPORT need to be swapped for the terminal. It works now! Thanks again for your help!

@sneakers-the-rat
Copy link
Contributor

Ugh! yes all too common. One of the many reasons my blood pressure will finally descend a bit when we rebuild network discovery ;)

Closing for now, glad you got it figured out, and thanks @cxrodgers for your help as well :)

@cxrodgers
Copy link
Contributor

@sneakers-the-rat Should we swap the default settings of PUSHPORT and MSGPORT when the user is running the setup wizard for a Terminal? Would that be an easy workaround for this common issue?

@sneakers-the-rat
Copy link
Contributor

yes I think so. alternatively there's no real reason they can't be the same port. inverting the number would require an extension to the notion of constraints in the prefs defaults, to be able to change value depending on another prefs value, but that shouldn't be too hard

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants