lambda det IOC, autosave: too many open files

[prjemian]

Cam plugin says no (0) images complete. Looking at the lambda IOC's console and find this trouble:

2020/01/29 11:43:56.532 Lambda:writeInt32 Setting TriggerMode 0
2020/01/29 11:43:56.534 Lambda:writeInt32 Setting TriggerMode 0
2020/01/29 11:43:56.535 Entering Lambda:writeFloat64
2020/01/29 11:43:56.537 Entering Lambda:writeFloat64
Setting Acquire Period
mkdir umask 777 chmod ret 0
 Created Dir /home/8-id-i/2020-1/bluesky/A1020
IMM Recurse path: statis = 0
Error- Could not open IMM File
2020/01/29 11:43:56.784 NDPluginFile::openFileBase Error opening file /home/8-id-i/2020-1/bluesky/A1020/A1020.imm, status=3
save_restore:write_save_file: Backup file (./autosave/auto_settings.savB) bad or not found.  Writing a new one. [200129-114356]
save_restore:write_it - unable to open file './autosave/auto_settings.savB' [200129-114356]
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
save_restore:write_save_file: Can't write new backup file. [200129-114356]
../save_restore.c(1704): [0x18]=write_it:Too many open files

Will an IOC reboot fix this?

Originally posted by @prjemian in #127 (comment)

[prjemian]

Yes, problem was gone after IOC reboot. But still wonder why too many open files from autosave?

[prjemian]

@JPHammonds, @timmmooney, @keenanlang, @Engbretson, @MarkRivers - Any ideas why this area detector IOC for the Lambda 750 ended up with autosave reporting too many open files? It is running on RHEL workstation bronze. We restarted the IOC and were then able to continue scans. Our scans had stopped when we could not write a 1 to the IMMout plugin's Capture PV in prep for image acquisition.

Thread starts here: #127 (comment)

[prjemian]

note: ran df -HT on workstation bronze but no file systems were full.

[Engbretson]

Don’t suppose you know if this Detector was known to be working and stable before and this is new behavior (i.e., ‘bit rot’) or if this was recently recompiled against epics base 7? Or if the computer in question only started to do this after the latest RHEL7 Linux patches were deployed this shutdown? From: Pete R Jemian <notifications@github.com> Sent: Wednesday, January 29, 2020 1:16 PM To: aps-8id-dys/ipython-8idiuser <ipython-8idiuser@noreply.github.com> Cc: Engbretson <Engbretson@anl.gov>; Mention <mention@noreply.github.com> Subject: Re: [aps-8id-dys/ipython-8idiuser] lambda det IOC, autosave: too many open files (#132) note: ran df -HT on workstation bronze but no file systems were full. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#132?email_source=notifications&email_token=ADHLVC5AJKIT4COLCBW6EEDRAHIXTA5CNFSM4KNJ5QA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKIMWWI#issuecomment-579914585>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADHLVCYQSYV3D64UA7SFPWTRAHIXTANCNFSM4KNJ5QAQ>.

[JPHammonds]

I do not remember seeing an error like this. From: Pete R Jemian <notifications@github.com> Reply-To: aps-8id-dys/ipython-8idiuser <reply@reply.github.com> Date: Wednesday, January 29, 2020 at 1:15 PM To: aps-8id-dys/ipython-8idiuser <ipython-8idiuser@noreply.github.com> Cc: "JPHammonds@anl.gov" <jphammonds@anl.gov>, Mention <mention@noreply.github.com> Subject: Re: [aps-8id-dys/ipython-8idiuser] lambda det IOC, autosave: too many open files (#132) @JPHammonds<https://github.com/JPHammonds>, @timmmooney<https://github.com/timmmooney>, @keenanlang<https://github.com/keenanlang>, @Engbretson<https://github.com/Engbretson>, @MarkRivers<https://github.com/MarkRivers> - Any ideas why this area detector IOC for the Lambda 750 ended up with autosave reporting too many open files? It is running on RHEL workstation bronze. We restarted the IOC and were then able to continue scans. Our scans had stopped when we could not write a 1 to the IMMout plugin's Capture PV in prep for image acquisition. Thread starts here: #127 (comment)<#127 (comment)> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#132?email_source=notifications&email_token=ACE7IMBB7G57AISV4HJ6O5TRAHIS3A5CNFSM4KNJ5QA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKIMSFQ#issuecomment-579914006>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACE7IMBCS5OIAYRV5OTDZXDRAHIS3ANCNFSM4KNJ5QAQ>.

[MarkRivers]

I don't think we know that autosave had too many open files. There are actually 2 error messages:

Error- Could not open IMM File

../save_restore.c(1704): [0x18]=write_it:Too many open files

Those errors just indicate that there are too many files open, but we don't know what software opened them, e.g. the detector driver, autosave, or something else. I think it is more likely that the driver is the culprit than autosave.

[prjemian]

Don’t suppose you know if this Detector was known to be working and stable before and this is new behavior (i.e., ‘bit rot’) or if this was recently recompiled against epics base 7? Or if the computer in question only started to do this after the latest RHEL7 Linux patches were deployed this shutdown?

Detector was being used in a a long scanning procedure involving multiple image acquisitions, typical of regular operations with this specific detector and IOC. Workstation uptime was since before the experiment started and IOC was working.

[prjemian]

There are actually 2 error messages:

Good catch. I agree that autosave was not the likely root cause, but rather a symptom.

[sureshnaps]

Pete, Is this a new Lambda issue due to RHEL update that I should be looking into or is this problem related to something else? Suresh From: Pete R Jemian <notifications@github.com> Reply-To: aps-8id-dys/ipython-8idiuser <reply@reply.github.com> Date: Wednesday, January 29, 2020 at 3:34 PM To: aps-8id-dys/ipython-8idiuser <ipython-8idiuser@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: Re: [aps-8id-dys/ipython-8idiuser] lambda det IOC, autosave: too many open files (#132) There are actually 2 error messages: Good catch. I agree that autosave was not the likely root cause, but rather a symptom. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#132?email_source=notifications&email_token=AAYSAQZWLEZKHEPTFCSPLUTRAHY5TA5CNFSM4KNJ5QA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKI2KII#issuecomment-579970337>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAYSAQ2KQF7WE3NMQ3VI2LLRAHY5TANCNFSM4KNJ5QAQ>.

[prjemian]

Good question, but it is not clear what action to take now. What's new with the workstation?

…

On Wed, Jan 29, 2020, 5:42 PM Suresh N. ***@***.***> wrote: Pete, Is this a new Lambda issue due to RHEL update that I should be looking into or is this problem related to something else? Suresh From: Pete R Jemian ***@***.***> Reply-To: aps-8id-dys/ipython-8idiuser ***@***.***> Date: Wednesday, January 29, 2020 at 3:34 PM To: aps-8id-dys/ipython-8idiuser ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [aps-8id-dys/ipython-8idiuser] lambda det IOC, autosave: too many open files (#132) There are actually 2 error messages: Good catch. I agree that autosave was not the likely root cause, but rather a symptom. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub< #132?email_source=notifications&email_token=AAYSAQZWLEZKHEPTFCSPLUTRAHY5TA5CNFSM4KNJ5QA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKI2KII#issuecomment-579970337>, or unsubscribe< https://github.com/notifications/unsubscribe-auth/AAYSAQ2KQF7WE3NMQ3VI2LLRAHY5TANCNFSM4KNJ5QAQ>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#132?email_source=notifications&email_token=AARMUMEOW22GC52UZEFOHELRAIH7NA5CNFSM4KNJ5QA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKJFO2Y#issuecomment-580015979>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARMUMH6WXW3GEN6KYU2HJTRAIH7NANCNFSM4KNJ5QAQ> .

[prjemian]

I believe now this is related to #127 (comment). With the changes in e0d4731 and e6a18a4 (and restarting the IOC), data acquisition continues.

Summary: the IMMout file path must start with /data/ for the XPCS image data, while the data management workflow is written to a paht that starts with /home/8-id-i/. These are the same shared filesystem but mounted in different places on the different hosts.