OSC plug in crashes GUI

[cspurandare]

Hi all, we obtained our first neuropixel recordings (using open ephys gui) last week, where the stimulus was created and presented with bonsai. To sync these two, I tried using the OSC messaging, or ttls from bonsai routed to the 385th channel or both simultaneously. Although it did work sometimes, I noticed -

If both the TTL and osc were communicating, the open ephys gui would sometimes hang/crash when bonsai started. This crash was not accompanied by any error message in the log as far as I can tell. The crash was actually more of a "hang", where LFP and AP band signal just flattened out in the display and nothing was recorded anymore, but the gui showed that it's still recording (red bar at the top, timer still increasing). The CPU requirement tab was below 25-30% so I don't think it was being overwhelmed. I am recording on a 20 thread 10 core computer, using an SSD card, but looking at the SSD card performance, it sometimes does peak close to 100%.

On other occasions, the system worked, and I could see ttls from bonsai appearing in the last channel (385) in open ephys gui. Checking later the osc messages were stored ok too. But after running for 2 to 3 hours, the open ephys gui would randomly crash (flatlined, but gui shows the recording is still going on, as above).

I tried with 0.6.1 and 0.6.4 and saw these issues in both.
Has anyone else experienced this behavior and know of any solutions/fixes? Thanks in advance!

[jsiegle]

Hi Chinmay – we are still looking into a fix for this; since the hanging happens randomly, it's been somewhat difficult to troubleshoot.

Given that the TTL outputs are being recorded by the Neuropixels event channel, do you actually need to send the OSC messages to Open Ephys? The OSC messages can be saved on the Bonsai side, and aligned to their respective TTL events offline.

The OSC Plugin is meant to facilitate closed-loop feedback based on analysis done in Bonsai – for example, gating the output of the Phase Detector when the subject enters a certain section of the environment. If this isn't what you're doing, the OSC Plugin is just sending redundant information.

[jsiegle]

Also, if you can share the Bonsai workflow you've been using, it would be helpful for debugging.

[cspurandare]

https://drive.google.com/file/d/1ewIk8jjGQ3wikIkw-W6MXnFwFff9PxJX/view?usp=drive_link, https://drive.google.com/file/d/1vUKrtjB7YLfVvkIFSM1lXZWYw7LcsGrV/view?usp=drive_link

I was not able to upload them here, so I am sharing the links above. Also, since the OP, I am now getting the issue every single time, in terms of the start of Bonsai crashes OpenEphys when the OSCEvents node is in the workflow. I have tried different ports (apart from 27020), different addresses (like/number, /ttl, /num etc), tried to create manual exceptions in the firewall for the ports, and 3 different versions of OpenEphys 6.0 6.1 6.5.
Thanks and do let me know if any other information might help get to the bottom of the issue.

[jsiegle]

Hi Chinmay -- looking at the Bonsai workflow, I can see that the "OSCGenerator" group workflow is waiting 30 seconds, then sending a message every 200 ms containing the seconds value of the current software timestamp (which cycles through values between 0 and 59). On the Open Ephys side, this translates to 5 TTL "ON" events per second, with each second having events on a different TTL line.

This definitely falls outside of the intended usage of the OSC Events plugin. It's not meant to synchronize processes, because you can get much more precise synchronization by using hardware TTLs. Instead, it's designed to allow closed-loop feedback that combines analysis done in Bonsai (e.g. extracting ROIs from video) with analysis done in Open Ephys (e.g. detecting hippocampal ripples).

Given that you already have the ability to translate hardware TTLs into software times registered by Bonsai (using the "TTLGenerator" group workflow), the OSC Events is just adding redundant (and less precise) information. So I would recommend disabling this part of the workflow and removing the OSC Events plugin on the Open Ephys side.

We still want to determine what's causing these crashes, of course, so I am currently running this workflow + OSC Events plugin in debug mode to see what happens after a few hours. I will let you know what I find.

[jsiegle]

The workflow + OSC Plugin are still running after ~5 days. Can you send the Open Ephys settings.xml file for your experiment, so we can try to replicate your exact configuration?

[cspurandare]

0.6.0 8 6 Aug 2023 19:54:38 Windows 10

Hi sorry about the delay in response. Find inserted above

[cspurandare]

settingsFile_Copied.txt

[jsiegle]

Thanks for sending! I ran this config for 6+ hours without observing any crashes. But we will keep trying to see if we can replicate the problem.

Some advice for now:

• Just perform synchronization with the hardware TTL line, which will be more precise than using OSC messages

For future use of the OSC plug-in for synchronization (in cases where hardware synchronization is not possible):

• Dedicate one TTL line for software synchronization, so this doesn't interfere with events on other lines
• Only send messages at longer intervals (e.g., 30-60 seconds). The potential clock drift on shorter timescales will be negligible.
• Tag each message with a software timestamp on the Bonsai side, and save these to a CSV

I'm going to close this issue, since the problem is also documented here: open-ephys-plugins/osc-io#1