Missing Meshtastic requirement

[metrafonic]

System Health details

..

Checklist

• I have enabled debug logging for my installation.
• I have filled out the issue template to the best of my ability.
• This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).
• This issue is not a duplicate issue of any previous issues..

Describe the issue

Installing via hacs, it seems like it doesn't install the Meshtastic python package

Reproduction steps

1.aaa
2. aa
3.
...

Debug logs

Logger: aiohttp.server
Source: /usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py:448
First occurred: 8:50:07 PM (1 occurrences)
Last logged: 8:50:07 PM

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 477, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 559, in _handle
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 92, in security_filter_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 210, in forwarded_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 26, in request_context_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 85, in ban_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 242, in auth_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/headers.py", line 32, in headers_middleware
    response = await handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/http.py", line 73, in handle
    result = await handler(request, **request.match_info)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/decorators.py", line 81, in with_admin
    return await func(self, request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 74, in wrapper
    return await method(view, request, data, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 172, in post
    return await self._post_impl(request, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 179, in _post_impl
    return await super()._post_impl(request, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 84, in _post_impl
    result = await self._flow_mgr.async_init(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1287, in async_init
    flow, result = await self._async_init(flow_id, handler, context, data)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1312, in _async_init
    flow = await self.async_create_flow(handler, context=context, data=data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1499, in async_create_flow
    handler = await _async_get_flow_handler(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 2964, in _async_get_flow_handler
    await _load_integration(hass, domain, hass_config)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 2941, in _load_integration
    await async_process_deps_reqs(hass, hass_config, integration)
  File "/usr/src/homeassistant/homeassistant/setup.py", line 575, in async_process_deps_reqs
    await requirements.async_get_integration_with_requirements(
  File "/usr/src/homeassistant/homeassistant/requirements.py", line 54, in async_get_integration_with_requirements
    return await manager.async_get_integration_with_requirements(domain)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/requirements.py", line 170, in async_get_integration_with_requirements
    await future
  File "/usr/src/homeassistant/homeassistant/requirements.py", line 159, in async_get_integration_with_requirements
    await self._async_process_integration(integration, done)
  File "/usr/src/homeassistant/homeassistant/requirements.py", line 182, in _async_process_integration
    await self.async_process_requirements(
  File "/usr/src/homeassistant/homeassistant/requirements.py", line 269, in async_process_requirements
    await self._async_process_requirements(name, missing)
  File "/usr/src/homeassistant/homeassistant/requirements.py", line 307, in _async_process_requirements
    raise RequirementsNotFound(name, list(failures))
homeassistant.requirements.RequirementsNotFound: Requirements for meshtastic not found: ['meshtastic==2.5.4'].

Diagnostics dump

...

[broglep]

Can you please report what home assistant version you are using?
If you are familiar with python, can you try to manually to install the dependency and report any errors?
Use pip install meshtastic==2.5.4 or pip3 install meshtastic==2.5.4 (you might first need to switch to proper virtual envs when you are using them)

I'm aware of a dependency issue under certain conditions where the protobuf library that meshtastic requires is conflicting with versions from other packages required by home assistant (or integrations)

[metrafonic]

Hey! I installed the requirement by doing a manual exec into the homeassistant docker container and running a pip install of meshtastic. It all works! Im running 2024.10.4 so I probably should have upgraded to the latest HA version first.

Any plans for location parsing? And SNR/RSSI?

This is a very cool project! Im familiar with python, so I could help you with this project.

[metrafonic]

Looking at your code, this looks really nice! Having the ability to send messages to nodes is amazing!

[broglep]

Great to hear that you did get it working. Then I suspect it might have been that issue, I did not yet have a more detailed look into when it exactly happens. pip dependency resolution issues seems to be rather hidden away by home assistant.

I have locally a version with some more metrics exposed (node_hops_away, node_snr, node_last_heard among a few others). I should be able push those changes this weekend.

Location parsing / integrating it into home assistants location concept (tracker) is kind of planned, but rather low on my priority list at the moment. If that is something you would be interest in, would be great if you can help with the project.

I'm currently still focused on some foundational work to get reliable connection to nodes, the current meshtastic python client has some shortcomings (e.g. meshtastic/python#703, but also seems to be unable to recover after a broken connection). I'm currently working on an async meshtastic client that will replace MeshtasticApiClient in this home assistant integration. I have a working prototype that now also supports connection via serial & bluetooth (even with BLE proxy) and have added some support for home assistant auto discovery.

[broglep]

As part of another dev setup, I came across this issue one again as well. With python 3.11 and homeassistant==2024.3.3, this can be reproduced:

2024-11-23 22:02:34.511 ERROR (SyncWorker_0) [homeassistant.util.package] Unable to install package meshtastic==2.5.4: /home/dev/.virtualenvs/homeassistant-meshtastic-root/lib/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
ERROR: Cannot install meshtastic==2.5.4 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
[notice] A new release of pip is available: 23.2.1 -> 24.3.1
[notice] To update, run: /home/dev/.virtualenvs/homeassistant-meshtastic-root/bin/python -m pip install --upgrade pip
2024-11-23 22:02:37.294 ERROR (MainThread) [homeassistant.setup] Setup failed for custom integration 'meshtastic': Requirements for meshtastic not found: ['meshtastic==2.5.4'].

Protobuf version that is conflicting with meshtastic is protobuf==4.25.1 in this case. Home assistant is hiding the details about dependency resolution conflicts.

Note that manual installation of meshtastic dependency with pip will uninstall older protobuf version and install new protobuf-5.28.3, this could have other unwanted side effects an breaking some parts of home assistant.

What I know is that meshtastic python lib code meshtastic.util.message_to_json does not work with older protobuf version due to changes in https://protobuf.dev/news/v26/ (including_default_value_fields vs always_print_fields_with_no_presence), and that code is currently being used by this integration. I have a workaround for this in place, but still unclear if there are some other unforeseen side effects when running meshtastic code with older protobuf version. If you are using this custom component in a bigger home assistant installation, I'd recommend for the time being to try run with the older protobuf version instead the new one

[metrafonic]

Ive been playing around with the integration for the last day. I really like how integrated it is into the HA ecosystem, while not relying on MQTT.

Ive been experiencing dropouts in the connection like you mentioned, however the diagnostic information around this is a bit limited. Sometimes HA logging shows the traceback from a broken pipe. However sometimes there is no log, but I see that no new data being added (eg uptime stays static on the device with the serial connection).

Im running this integration against a RAK4631 connected via POE (ethernet). I assume one of the reasons for the broken pipe is when a different client makes a connection to the API (eg Meshtastic cli from a pc).

I saw how you patched the our_exit function to keep the library from exiting. It's unfortunate that the python library isn't more resilient.

Ill try set up a dev environment for properly debugging the integration as its currently running on my home "production" HA system. This has a lot of potential, especially if BLE connections are added.

[metrafonic]

I made a mistake regarding my observation on the connection being dead without a log entry, it just seems like sometimes the uptime metric is delayed and the connection is intact.

[broglep]

Good news, the protobuf compatibility issue as been addressed upstream in meshtastic/python@426795f and the required protobuf version has also been relaxed, so switching to the new version 2.5.5 will resolve compatibility issues with home assistant

[metrafonic]

Great to hear! I see you added changes to the dev branch. Is there a simple way to switch to this branch in HACS? I tried using the URL https://github.com/broglep/homeassistant-meshtastic?ref=dev but it seems to stay on main

[broglep]

The first alpha version is now in main, with that this issues here regarding meshtastic requirement should be resolved. If not, please re-open.

For non-issue related discussions I have enabled discussions on this repo.

If you encounter bugs with the new version, feel free to report them, pretty sure that whole integration is still far from stable and I don't have this new version running yet for an extended period of time. But should be a bit more robust now as heartbeat mechanism is in place and I have spend quite some time to get dropped connection detection & reconnection handling working

[broglep]

Unfortunately a new instance of the problem can occur:

 Because meshtastic==2.5.5 depends on bleak>=0.22.3,<0.23.0 and
      bleak==0.22.2, we can conclude that meshtastic==2.5.5 cannot be used.