-
Notifications
You must be signed in to change notification settings - Fork 1
lambda det IOC, autosave: too many open files #132
Comments
Yes, problem was gone after IOC reboot. But still wonder why too many open files from autosave? |
@JPHammonds, @timmmooney, @keenanlang, @Engbretson, @MarkRivers - Any ideas why this area detector IOC for the Lambda 750 ended up with autosave reporting too many open files? It is running on RHEL workstation bronze. We restarted the IOC and were then able to continue scans. Our scans had stopped when we could not write a 1 to the IMMout plugin's Capture PV in prep for image acquisition. Thread starts here: #127 (comment) |
note: ran |
Don’t suppose you know if this Detector was known to be working and stable before and this is new behavior (i.e., ‘bit rot’) or if this was recently recompiled against epics base 7?
Or if the computer in question only started to do this after the latest RHEL7 Linux patches were deployed this shutdown?
From: Pete R Jemian <notifications@github.com>
Sent: Wednesday, January 29, 2020 1:16 PM
To: aps-8id-dys/ipython-8idiuser <ipython-8idiuser@noreply.github.com>
Cc: Engbretson <Engbretson@anl.gov>; Mention <mention@noreply.github.com>
Subject: Re: [aps-8id-dys/ipython-8idiuser] lambda det IOC, autosave: too many open files (#132)
note: ran df -HT on workstation bronze but no file systems were full.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#132?email_source=notifications&email_token=ADHLVC5AJKIT4COLCBW6EEDRAHIXTA5CNFSM4KNJ5QA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKIMWWI#issuecomment-579914585>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADHLVCYQSYV3D64UA7SFPWTRAHIXTANCNFSM4KNJ5QAQ>.
|
I do not remember seeing an error like this.
From: Pete R Jemian <notifications@github.com>
Reply-To: aps-8id-dys/ipython-8idiuser <reply@reply.github.com>
Date: Wednesday, January 29, 2020 at 1:15 PM
To: aps-8id-dys/ipython-8idiuser <ipython-8idiuser@noreply.github.com>
Cc: "JPHammonds@anl.gov" <jphammonds@anl.gov>, Mention <mention@noreply.github.com>
Subject: Re: [aps-8id-dys/ipython-8idiuser] lambda det IOC, autosave: too many open files (#132)
@JPHammonds<https://github.com/JPHammonds>, @timmmooney<https://github.com/timmmooney>, @keenanlang<https://github.com/keenanlang>, @Engbretson<https://github.com/Engbretson>, @MarkRivers<https://github.com/MarkRivers> - Any ideas why this area detector IOC for the Lambda 750 ended up with autosave reporting too many open files? It is running on RHEL workstation bronze. We restarted the IOC and were then able to continue scans. Our scans had stopped when we could not write a 1 to the IMMout plugin's Capture PV in prep for image acquisition.
Thread starts here: #127 (comment)<#127 (comment)>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#132?email_source=notifications&email_token=ACE7IMBB7G57AISV4HJ6O5TRAHIS3A5CNFSM4KNJ5QA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKIMSFQ#issuecomment-579914006>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACE7IMBCS5OIAYRV5OTDZXDRAHIS3ANCNFSM4KNJ5QAQ>.
|
I don't think we know that autosave had too many open files. There are actually 2 error messages:
Those errors just indicate that there are too many files open, but we don't know what software opened them, e.g. the detector driver, autosave, or something else. I think it is more likely that the driver is the culprit than autosave. |
Detector was being used in a a long scanning procedure involving multiple image acquisitions, typical of regular operations with this specific detector and IOC. Workstation uptime was since before the experiment started and IOC was working. |
Good catch. I agree that autosave was not the likely root cause, but rather a symptom. |
Pete,
Is this a new Lambda issue due to RHEL update that I should be looking into or is this problem related to something else?
Suresh
From: Pete R Jemian <notifications@github.com>
Reply-To: aps-8id-dys/ipython-8idiuser <reply@reply.github.com>
Date: Wednesday, January 29, 2020 at 3:34 PM
To: aps-8id-dys/ipython-8idiuser <ipython-8idiuser@noreply.github.com>
Cc: Subscribed <subscribed@noreply.github.com>
Subject: Re: [aps-8id-dys/ipython-8idiuser] lambda det IOC, autosave: too many open files (#132)
There are actually 2 error messages:
Good catch. I agree that autosave was not the likely root cause, but rather a symptom.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#132?email_source=notifications&email_token=AAYSAQZWLEZKHEPTFCSPLUTRAHY5TA5CNFSM4KNJ5QA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKI2KII#issuecomment-579970337>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAYSAQ2KQF7WE3NMQ3VI2LLRAHY5TANCNFSM4KNJ5QAQ>.
|
Good question, but it is not clear what action to take now. What's new
with the workstation?
…On Wed, Jan 29, 2020, 5:42 PM Suresh N. ***@***.***> wrote:
Pete,
Is this a new Lambda issue due to RHEL update that I should be looking
into or is this problem related to something else?
Suresh
From: Pete R Jemian ***@***.***>
Reply-To: aps-8id-dys/ipython-8idiuser ***@***.***>
Date: Wednesday, January 29, 2020 at 3:34 PM
To: aps-8id-dys/ipython-8idiuser ***@***.***>
Cc: Subscribed ***@***.***>
Subject: Re: [aps-8id-dys/ipython-8idiuser] lambda det IOC, autosave: too
many open files (#132)
There are actually 2 error messages:
Good catch. I agree that autosave was not the likely root cause, but
rather a symptom.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<
#132?email_source=notifications&email_token=AAYSAQZWLEZKHEPTFCSPLUTRAHY5TA5CNFSM4KNJ5QA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKI2KII#issuecomment-579970337>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/AAYSAQ2KQF7WE3NMQ3VI2LLRAHY5TANCNFSM4KNJ5QAQ>.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#132?email_source=notifications&email_token=AARMUMEOW22GC52UZEFOHELRAIH7NA5CNFSM4KNJ5QA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKJFO2Y#issuecomment-580015979>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARMUMH6WXW3GEN6KYU2HJTRAIH7NANCNFSM4KNJ5QAQ>
.
|
I believe now this is related to #127 (comment). With the changes in e0d4731 and e6a18a4 (and restarting the IOC), data acquisition continues. Summary: the IMMout file path must start with |
Cam plugin says no (0) images complete. Looking at the lambda IOC's console and find this trouble:
Will an IOC reboot fix this?
Originally posted by @prjemian in #127 (comment)
The text was updated successfully, but these errors were encountered: