[zigpy-integration] - not able to run 2 instances of zigpy-radio module in Domoticz #1117

pipiche38 · 2022-03-27T13:40:16Z

if trying to run 2 instances of zigpy-radio module, then the 2nd instance doesn't work !

pipiche38 · 2022-03-27T14:41:59Z

looks like the issue is located around the asyncio loop().
In the case of Domoticz, we have only one python interpreter running and all threads run underneath.

Question is how to manage the create_task() in that context.

The 1st instance runs and create the event loop, but when the 2nd instance come (the loop already exist), can we get a new loop for the asyncio ?

here is the code

    loop = asyncio.events.new_event_loop()
    asyncio.events.set_event_loop(loop)
    task = loop.create_task(
        radio_start(self, self._radiomodule, self._serialPort, set_channel=channel, set_extendedPanId=extendedPANID)
        )
    loop.run_until_complete(task)
    loop.run_until_complete(asyncio.sleep(1))
    loop.close()

pipiche38 · 2022-03-27T14:54:50Z

seems to be the right way

    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        
        loop = asyncio.events.new_event_loop()
        asyncio.events.set_event_loop(loop)
        
    task = loop.create_task(
        radio_start(self, self._radiomodule, self._serialPort, set_channel=channel, set_extendedPanId=ex
        )
    loop.run_until_complete(task)
    loop.run_until_complete(asyncio.sleep(1))
    loop.close()

pipiche38 · 2022-03-27T15:03:20Z

edd8b6b

pipiche38 · 2022-03-27T16:12:06Z

Time to time it crashs here loop.run_until_complete(task)

Mar 27 18:11:35 rasp domoticz[12497]: Exception in thread ZigpyCom_2:
Mar 27 18:11:35 rasp domoticz[12497]: Traceback (most recent call last):
Mar 27 18:11:35 rasp domoticz[12497]:   File "/usr/lib/python3.9/threading.py", line 973, in _bootstrap_inner
Mar 27 18:11:35 rasp domoticz[12497]: 2022-03-27 18:11:35.410  Status: Zigbee ZigateV2: ZigpyTransport: thread_processing_and_sending Thread start.
Mar 27 18:11:35 rasp domoticz[12497]:     self.run()
Mar 27 18:11:35 rasp domoticz[12497]:   File "/usr/lib/python3.9/threading.py", line 910, in run
Mar 27 18:11:35 rasp domoticz[12497]:     self._target(*self._args, **self._kwargs)
Mar 27 18:11:35 rasp domoticz[12497]:   File "/var/lib/domoticz/plugins/Domoticz-Zigbee/Classes/ZigpyTransport/zigpyThread.py", line 85, in zigpy_thread
Mar 27 18:11:35 rasp domoticz[12497]: 2022-03-27 18:11:35.445  Status: Zigbee ZigateV2: Start Web Server connection
Mar 27 18:11:35 rasp domoticz[12497]:     loop.run_until_complete(task)
Mar 27 18:11:35 rasp domoticz[12497]:   File "/usr/lib/python3.9/asyncio/base_events.py", line 618, in run_until_complete
Mar 27 18:11:35 rasp domoticz[12497]: 2022-03-27 18:11:35.450  Status: Zigbee ZigateV2: Web backend for Web User Interface started on port: 9440
Mar 27 18:11:35 rasp domoticz[12497]:     self._check_running()
Mar 27 18:11:35 rasp domoticz[12497]:   File "/usr/lib/python3.9/asyncio/base_events.py", line 578, in _check_running
Mar 27 18:11:35 rasp domoticz[12497]:     raise RuntimeError('This event loop is already running')
Mar 27 18:11:35 rasp domoticz[12497]: RuntimeError: This event loop is already running

puddly · 2022-03-27T16:25:01Z

Mixing asyncio and threads is tricky.

What was not working with the first way you were doing it? Also, you don't need the loop.create_task() call at all.

pipiche38 · 2022-03-27T16:47:02Z

I think that I'm in several issues

1/ embedded python with C++.
2/ every plugin instance is launched in one non-blocking thread
3/ the non-blocking thread is spawning threads for it own purpose and one of them is dedicated zigpy

In regards to you comment, if I do not do loop.create_task() how am I going to start asyncio

puddly · 2022-03-27T16:51:14Z

In regards to you comment, if I do not do loop.create_task() how am I going to start asyncio

The same way you run asyncio.sleep in a blocking manner:

radio_coro = radio_start(self, self._radiomodule, self._serialPort, set_channel=channel, set_extendedPanId=extendedPANID)
loop.run_until_complete(radio_coro)

pipiche38 · 2022-03-27T17:17:51Z

so it make the tric. however when starting then 2nd instance with the following code

    try:
        loop = asyncio.get_running_loop()

    except RuntimeError:  # no event loop running:
        loop = asyncio.events.new_event_loop()
        asyncio.events.set_event_loop(loop)

    task = radio_start(self, self._radiomodule, self._serialPort, set_channel=channel, set_extendedPanId=extendedPANID)
 
    loop.run_until_complete(task)
    loop.run_until_complete(asyncio.sleep(1))
    loop.close()

Mar 27 19:10:02 rasp domoticz[14062]: Traceback (most recent call last):
Mar 27 19:10:02 rasp domoticz[14062]: File "/usr/lib/python3.9/threading.py", line 973, in _bootstrap_inner
Mar 27 19:10:02 rasp domoticz[14062]: self.run()
Mar 27 19:10:02 rasp domoticz[14062]: File "/usr/lib/python3.9/threading.py", line 910, in run
Mar 27 19:10:02 rasp domoticz[14062]: self._target(*self._args, **self._kwargs)
Mar 27 19:10:02 rasp domoticz[14062]: File "/var/lib/domoticz/plugins/Domoticz-Zigbee/Classes/ZigpyTransport/zigpyThread.py", line 85, in zigpy_thread
Mar 27 19:10:02 rasp domoticz[14062]: 2022-03-27 19:10:02.039 Status: Zigpy SonOff: ZigpyTransport: thread_processing_and_sending Thread start.
Mar 27 19:10:02 rasp domoticz[14062]: loop.run_until_complete(task)
Mar 27 19:10:02 rasp domoticz[14062]: File "/usr/lib/python3.9/asyncio/base_events.py", line 618, in run_until_complete
Mar 27 19:10:02 rasp domoticz[14062]: self._check_running()
Mar 27 19:10:02 rasp domoticz[14062]: File "/usr/lib/python3.9/asyncio/base_events.py", line 578, in _check_running
Mar 27 19:10:02 rasp domoticz[14062]: raise RuntimeError('This event loop is already running')
Mar 27 19:10:02 rasp domoticz[14062]: RuntimeError: This event loop is already running

And with the following code

    loop = asyncio.events.new_event_loop()
    asyncio.events.set_event_loop(loop)

    task = radio_start(self, self._radiomodule, self._serialPort, set_channel=channel, set_extendedPanId=extendedPANID)
 
    loop.run_until_complete(task)
    loop.run_until_complete(asyncio.sleep(1))
    loop.close()

we are getting this error while starting the 2nd instance

Mar 27 19:16:49 rasp domoticz[14219]: Traceback (most recent call last):
Mar 27 19:16:49 rasp domoticz[14219]: File "/usr/lib/python3.9/threading.py", line 973, in _bootstrap_inner
Mar 27 19:16:49 rasp domoticz[14219]: self.run()
Mar 27 19:16:49 rasp domoticz[14219]: File "/usr/lib/python3.9/threading.py", line 910, in run
Mar 27 19:16:49 rasp domoticz[14219]: 2022-03-27 19:16:49.661 Status: Zigpy SonOff: ZigpyTransport: thread_processing_and_sending Thread start.
Mar 27 19:16:49 rasp domoticz[14219]: self._target(*self._args, **self._kwargs)
Mar 27 19:16:49 rasp domoticz[14219]: File "/var/lib/domoticz/plugins/Domoticz-Zigbee/Classes/ZigpyTransport/zigpyThread.py", line 81, in zigpy_thread
Mar 27 19:16:49 rasp domoticz[14219]: loop.run_until_complete(task)
Mar 27 19:16:49 rasp domoticz[14219]: File "/usr/lib/python3.9/asyncio/base_events.py", line 618, in run_until_complete
Mar 27 19:16:49 rasp domoticz[14219]: self._check_running()
Mar 27 19:16:49 rasp domoticz[14219]: File "/usr/lib/python3.9/asyncio/base_events.py", line 580, in _check_running
Mar 27 19:16:49 rasp domoticz[14219]: raise RuntimeError(
Mar 27 19:16:49 rasp domoticz[14219]: RuntimeError: Cannot run the event loop while another loop is running

pipiche38 · 2022-03-27T17:51:18Z

Latest code

    loop = get_or_create_eventloop()
        
    task = radio_start(self, self._radiomodule, self._serialPort, set_channel=channel, set_extendedPanId=extendedPANID)
 
    loop.run_until_complete(task)
    loop.run_until_complete(asyncio.sleep(1))
    loop.close()

    self.log.logging("TransportZigpy", "Debug", "zigpy_thread - exiting zigpy thread")

def get_or_create_eventloop():
    try:
        return asyncio.get_event_loop()
    except RuntimeError as ex:
        if "There is no current event loop in thread" in str(ex):
            asyncio.set_event_loop(asyncio.new_event_loop())
            return asyncio.get_event_loop()

Does work perfectly for one occurence of a Ziggy instance, but do not work when starting a 2nd instance.

Mar 27 19:44:23 rasp domoticz[14891]: Exception in thread ZigpyCom_2:
Mar 27 19:44:23 rasp domoticz[14891]: Traceback (most recent call last):
Mar 27 19:44:23 rasp domoticz[14891]: File "/usr/lib/python3.9/threading.py", line 973, in _bootstrap_inner
Mar 27 19:44:23 rasp domoticz[14891]: 2022-03-27 19:44:23.497 Status: Zigbee ZigateV2: ZigpyTransport: thread_processing_and_sending Thread start.
Mar 27 19:44:23 rasp domoticz[14891]: self.run()
Mar 27 19:44:23 rasp domoticz[14891]: File "/usr/lib/python3.9/threading.py", line 910, in run
Mar 27 19:44:23 rasp domoticz[14891]: self._target(*self._args, **self._kwargs)
Mar 27 19:44:23 rasp domoticz[14891]: File "/var/lib/domoticz/plugins/Domoticz-Zigbee/Classes/ZigpyTransport/zigpyThread.py", line 80, in zigpy_thread
Mar 27 19:44:23 rasp domoticz[14891]: loop.run_until_complete(task)
Mar 27 19:44:23 rasp domoticz[14891]: File "/usr/lib/python3.9/asyncio/base_events.py", line 618, in run_until_complete
Mar 27 19:44:23 rasp domoticz[14891]: 2022-03-27 19:44:23.518 Status: Zigbee ZigateV2: Start Web Server connection
Mar 27 19:44:23 rasp domoticz[14891]: self._check_running()
Mar 27 19:44:23 rasp domoticz[14891]: File "/usr/lib/python3.9/asyncio/base_events.py", line 578, in _check_running
Mar 27 19:44:23 rasp domoticz[14891]: 2022-03-27 19:44:23.525 Status: Zigbee ZigateV2: Web backend for Web User Interface started on port: 9440
Mar 27 19:44:23 rasp domoticz[14891]: raise RuntimeError('This event loop is already running')
Mar 27 19:44:23 rasp domoticz[14891]: RuntimeError: This event loop is already running
Mar 27 19:44:23 rasp domoticz[14891]: /usr/lib/python3.9/threading.py:975: RuntimeWarning: coroutine 'radio_start' was never awaited
Mar 27 19:44:23 rasp domoticz[14891]: self._invoke_excepthook(self)
Mar 27 19:44:23 rasp domoticz[14891]: RuntimeWarning: Enable tracemalloc to get the object allocation traceback

pipiche38 · 2022-03-27T18:12:27Z

reporting to domoticz
https://www.domoticz.com/forum/viewtopic.php?p=288399#p288399

puddly · 2022-03-27T18:21:39Z

How does Domoticz start a plugin multiple times if onStart and onStop are global functions that take no parameters? _plugin = BasePlugin() is also a global.

pipiche38 · 2022-03-27T18:28:41Z

@dnpwwo if you see this once, your help would be more than welcome. I have created a small plugin.py file to duplicate the issue, which is that I cannot start 2 instances of a plugin running an asyncio.loop(). I'm convinced that it is not about the act that is 2 instances, but more 2 plugins which requires asyncio.loop()

pipiche38 · 2022-03-27T20:53:49Z

How does Domoticz start a plugin multiple times if onStart and onStop are global functions that take no parameters? _plugin = BasePlugin() is also a global.

I don't think they are global to the all Domoticz environment. My understanding is that Domoticz spawn a thread with plugin.py with a given context. When spawning a new thread/plugin it is a total different context.
However both are under the same python interpreter (GIL)

@puddly, @dnpwwo my understanding is when Domoticz start a plugin in a thread , that one is running with its own context, so when a second plugin is started with a new thread , that one is having its own context.
The issue I suspect is that when creating the asyncio.loop instead of been shared between the 2 threads , this is not happening, however Python Interpreter is probably confused , and I think this why what ever I create the loop, or I try to get it , it fails.

dnpwwo · 2022-03-29T10:36:43Z

@pipiche38,

You are correct, each plugin has both its own thread and own Python interpreter so running copies of a plugin is not a problem in most circumstances. That said the Python documentation describes an interpreter as:

This is an (almost) totally separate environment for the execution of Python code

The 'almost' is not fully described but alludes to things external to Python (such as file systems).

Modules are a grey area. The doco states:

In particular, the new interpreter has separate, independent versions of all imported modules, including the fundamental modules builtins, main and sys. The table of loaded modules (sys.modules) and the module search path (sys.path) are also separate.

But a lot of lower level modules are actually compiled C and can therefore do things that the interpreter can't see such as maintain global state and variables which could be the problem here.

pipiche38 · 2022-03-29T12:31:09Z

@dnpwwo really appreciate your valuable feedbacks.
I have posted here the issue I'm facing which I think will be the case if you try to run 2 plugins using asyncio/loop

badzz · 2022-04-02T20:23:20Z

But a lot of lower level modules are actually compiled C and can therefore do things that the interpreter can't see such as maintain global state and variables which could be the problem here.

Hi,
There seems to be a problem with this indeed. Asyncio lib is using c code that seems to have shared variables between the different instance of interpreters.
When running the plugin python file with my own main.c that init one interpreter , it is ok. the asyncio lib is instantiate once for both instance of the plugin as it expects to be running like this.
thx

CC: @dnpwwo

badzz · 2022-04-04T12:50:21Z

@dnpwwo
ok issue seems to be that we get the same loop id across interpreter instance
https://github.com/python/cpython/blob/ff2cf1d7d5fb25224f3ff2e0c678d36f78e1f3cb/Modules/_asynciomodule.c#L238
The running loop id is stored in a static variable and the threadstate id is the same as it is just an iterator that will have the same value for first thread in each interpretor. Hence both thread get the same loop id.
If you remove the cached_running_holder management, everything works fine.
I would like to do a simple test case for the python people, would you have a simple example of c code instantiating two interpreters ? I have a sample code launching thread with one interpreter .. but two interpreters , I am not sure where to start (my sample came from https://gist.github.com/sterin/e8090d0451ab781a4e22)

PS : I found some sample code to have two interpreter .. I have my test case for the python folks .. it seems two interpreters is risky and might have bugs : https://peps.python.org/pep-0554/#interpreter-isolation

pipiche38 · 2022-04-29T13:23:18Z

python/cpython#91375

https://bugs.python.org/issue47219

pipiche38 · 2022-05-24T08:29:43Z

pipiche38/multiinstances/no_asyncio ( ba6d729 )

pipiche38 · 2022-05-25T07:14:34Z

workaround in place

pipiche38 added zigpy Zigpy-Prio1 Blocking labels Mar 27, 2022

pipiche38 closed this as completed Mar 27, 2022

pipiche38 reopened this Mar 27, 2022

pipiche38 removed the Zigpy-Prio1 Blocking label Apr 4, 2022

pipiche38 closed this as completed May 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[zigpy-integration] - not able to run 2 instances of zigpy-radio module in Domoticz #1117

[zigpy-integration] - not able to run 2 instances of zigpy-radio module in Domoticz #1117

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022 •

edited

Loading

puddly commented Mar 27, 2022

pipiche38 commented Mar 27, 2022

puddly commented Mar 27, 2022 •

edited

Loading

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022

puddly commented Mar 27, 2022 •

edited

Loading

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022 •

edited

Loading

dnpwwo commented Mar 29, 2022

pipiche38 commented Mar 29, 2022

badzz commented Apr 2, 2022 •

edited by pipiche38

Loading

badzz commented Apr 4, 2022 •

edited

Loading

pipiche38 commented Apr 29, 2022 •

edited

Loading

pipiche38 commented May 24, 2022

pipiche38 commented May 25, 2022

[zigpy-integration] - not able to run 2 instances of zigpy-radio module in Domoticz #1117

[zigpy-integration] - not able to run 2 instances of zigpy-radio module in Domoticz #1117

Comments

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022 • edited Loading

puddly commented Mar 27, 2022

pipiche38 commented Mar 27, 2022

puddly commented Mar 27, 2022 • edited Loading

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022

puddly commented Mar 27, 2022 • edited Loading

pipiche38 commented Mar 27, 2022

pipiche38 commented Mar 27, 2022 • edited Loading

dnpwwo commented Mar 29, 2022

pipiche38 commented Mar 29, 2022

badzz commented Apr 2, 2022 • edited by pipiche38 Loading

badzz commented Apr 4, 2022 • edited Loading

pipiche38 commented Apr 29, 2022 • edited Loading

pipiche38 commented May 24, 2022

pipiche38 commented May 25, 2022

pipiche38 commented Mar 27, 2022 •

edited

Loading

puddly commented Mar 27, 2022 •

edited

Loading

puddly commented Mar 27, 2022 •

edited

Loading

pipiche38 commented Mar 27, 2022 •

edited

Loading

badzz commented Apr 2, 2022 •

edited by pipiche38

Loading

badzz commented Apr 4, 2022 •

edited

Loading

pipiche38 commented Apr 29, 2022 •

edited

Loading