-
Notifications
You must be signed in to change notification settings - Fork 1
Lambda Time Out: IOC Says 'Too Many Open Files' #146
Comments
Error message on Lambda IOC:
|
I pressed 'Stop' on the EPICS panel, but same error shows up again when I rerun the plan. As a result I rebooted Bluesky and Lambda and everything seems to be running, however it would be critical to resolve this problem |
Thanks for writing this up. I'll work with the group to identify and
resolve the issue. Might take some patience.
…On Thu, Feb 6, 2020, 10:10 PM qzhang234 ***@***.***> wrote:
Assigned #146 <#146>
to @prjemian <https://github.com/prjemian>.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#146?email_source=notifications&email_token=AARMUMF2LGIEJQPA62XSYKLRBTNMNA5CNFSM4KRIAJBKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOWPEVI7Q#event-3016316030>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARMUMBTY2B2A3WCTU3MT2DRBTNMNANCNFSM4KRIAJBA>
.
|
Is it the OS version (or OS updates)?Suspecting that updates to the operating system might be the cause, we booted with the OS kernel that was used in mid-December. Stlil, we observe many open files. Answer: No, not the OS version |
What about SPEC? Does it see so many open files?Restart the IOC before making this test, run similar acquisition from SPEC. For a short time, we see a couple dozen open .imm files at any time, but shortly after acquisition is complete, that number drops to zero. Answer: No problem with SPEC. Conclusion: our attention is focused on what is different between how we use this detector from SPEC and bluesky. |
Using bluesky, we see that 21 files .imm remain open on the EPICS IOC console ( |
Later today. we'll ask IT to reboot with the most recent kernel. |
Comparing the GUI screens when SPEC uses the detector and bluesky, we see no differences (aside from the file name and path). |
@keenanlang : What can we do as an EPICS client that would compel the IMMout plugin to keep a file open on the IOC? @mrakitin, @danielballan, @tacaswell: What can we do from bluesky that would compel the IMMout plugin to keep a file open on the IOC? |
Install various updates to bluesky environment including
|
With updates in the framework, see if we still have a problem from bluesky acquisition. |
Run 1 acquisition loop from bluesky:
Analysis from Now, exit bluesky, and check lsof again. Once bluesky is exited, the |
Being more systematic about diagnosing this, have written new function to print out all the EpicsSignal values in a single ophyd device. Here is an example:
|
To be clear, the files we finding open after bluesky acquisitions are the image files from the IMMout plugin, the |
These are the files acquired from the bluesky plan. |
Interesting...
|
Bluesky session console from today: details
|
From the most recent observation, focus on differences in the configuration of the IMMout plugin from SPEC and Bluesky. |
Talked with @keenanlang about the new support he is developing. Since its arrival is imminent, let's suspend further work on this issue since that new support will require a (at least partial) rewrite of In the interim, I'm working on #148 which should make the enormous number of bluesky support components easier to navigate. |
@prjemian Has this issue been solved? |
Make sure that this issue does not happen with #150 then can close here. |
The IOC has been updated. Closing the issue for now and commence Bluesky/EPICS testing. |
As in #132...
Error message on Bluesky:
The text was updated successfully, but these errors were encountered: