TimeoutError from EpicsSignal #776
Labels
Comments
|
And, another example during a scan: |
|
I suspect this is a bug in ophyd, pyepics, or more specifically the integration between the two (which lives in ophyd). I notice you are running on an ophyd prerelease, |
|
Yes. It's time to document this and catch the root cause
…On Mon, Nov 25, 2019, 12:45 PM Dan Allan ***@***.***> wrote:
I suspect this is a bug in ophyd, pyepics, or more specifically the
integration between the two (which lives in ophyd). I notice you are
running on an ophyd prerelease, 1.4.0rc1. Have you ever seen this on a
version in the 1.3.x series?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#776?email_source=notifications&email_token=AARMUMDUW4EATHSYCCTPR2DQVQMK5A5CNFSM4JRLGCSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFDMYQI#issuecomment-558287937>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARMUMBXE7HFB6K2AJZYPADQVQMK5ANCNFSM4JRLGCSA>
.
|
|
Timeout=None sounds like it comes from ophyd, agree?
…On Mon, Nov 25, 2019, 12:53 PM Pete Jemian ***@***.***> wrote:
Yes. It's time to document this and catch the root cause
On Mon, Nov 25, 2019, 12:45 PM Dan Allan ***@***.***> wrote:
> I suspect this is a bug in ophyd, pyepics, or more specifically the
> integration between the two (which lives in ophyd). I notice you are
> running on an ophyd prerelease, 1.4.0rc1. Have you ever seen this on a
> version in the 1.3.x series?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#776?email_source=notifications&email_token=AARMUMDUW4EATHSYCCTPR2DQVQMK5A5CNFSM4JRLGCSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFDMYQI#issuecomment-558287937>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AARMUMBXE7HFB6K2AJZYPADQVQMK5ANCNFSM4JRLGCSA>
> .
>
|
|
BUT, this line in the error dump: calls Within PyEpics, there are PV objects and Motor objects. This is a PV object. The call chain appears to be:
Here, and then watches for a timeout. Is there code in ophyd that raises the TimeoutError exception? Not seeing that exception from PyEpics. see this table:
Time to look for the relevant TimeoutError in ophyd. |
|
Lines 824 to 831 in 9da7884 In the wrong set of circumstances that trigger our problem, the The value of the timeout is not the real problem, though, only a symptom. Why do we get a timeout at all? First off, though, we don't know how long we have waited. |
|
Actually, why is The |
|
Here's the simple example from SO: |
|
Using
|
|
Perhaps converting these warnings to code: output: |
|
One possible revision of Lines 824 to 831 in 9da7884 self.wait_for_connection(timeout=connection_timeout)
try:
warnings.filterwarnings("error")
t0 = time.time()
info = self._read_pv.get_with_metadata(
as_string=as_string, form=form, **kwargs)
except UserWarning as exc:
print(f"while getting from {self._read_pvname}, caught '{exc}'")
finally:
dt = time.time() - t0
warnings.filterwarnings("default")
if info is None:
timeout = kwargs.get('timeout', None)
raise TimeoutError(f'Failed to read {self._read_pvname} in '
f'{dt:.3f} sec (with timeout={timeout})') |
|
As commented for APS 8-ID-I XPCS, a ...
|
|
@kmpeters reported similar experience with PyEpics (not bluesky):
|
|
@kmpeters - Can you post a snippet of the safer |
|
It looks like the latest version of my fly-scan code only has a safer #
# Debugging
#
def debugPut(self, pv, val, wait=False, use_complete=False, timeout=30.0):
#
exitFlag = False
while ( not exitFlag ):
try:
pv.put(val, wait=wait, use_complete=use_complete, timeout=timeout)
except WindowsError as e:
print
print "EPICS put FAILED:"
print "\tPV =", pv.pvname
print "\tval =", val
print "\twait =", wait, "; use_complete =", use_complete, "; timeout =", timeout
print "\tconnected =", pv.connected, "; host =", pv.host
print
print "A WindowsError occurred:"
# The following line results in an AttributeError. Apparently e.type doesn't exist
#!print "\t", e.type
print "\t", e
print "\t", e.args
print
print "Backtrace:"
traceback.print_stack()
print
time.sleep(15.00)
else:
exitFlag = TrueIt also looks like it wasn't a |
|
Thanks!
…On Tue, Dec 10, 2019, 1:13 PM Kevin Peterson ***@***.***> wrote:
It looks like the latest version of my fly-scan code only has a safer
put():
#
# Debugging
#
def debugPut(self, pv, val, wait=False, use_complete=False, timeout=30.0):
#
exitFlag = False
while ( not exitFlag ):
try:
pv.put(val, wait=wait, use_complete=use_complete, timeout=timeout)
except WindowsError as e:
print
print "EPICS put FAILED:"
print "\tPV =", pv.pvname
print "\tval =", val
print "\twait =", wait, "; use_complete =", use_complete, "; timeout =", timeout
print "\tconnected =", pv.connected, "; host =", pv.host
print
print "A WindowsError occurred:"
# The following line results in an AttributeError. Apparently e.type doesn't exist
#!print "\t", e.type
print "\t", e
print "\t", e.args
print
print "Backtrace:"
traceback.print_stack()
print
time.sleep(15.00)
else:
exitFlag = True
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#776?email_source=notifications&email_token=AARMUMFCVRDHZAMON6EJNZ3QX7S5LA5CNFSM4JRLGCSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGQNICY#issuecomment-564188171>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARMUMAJVPHCSMYNYULWQ4LQX7S5LANCNFSM4JRLGCSA>
.
|
|
The safer |
|
@kmpeters : Looking over the code you supplied to make a safer I'll begin testing the idea of trapping a |
|
@kmpeters: Trapping the |
|
I think |
|
The first item to resolve in this issue is to capture the |
prjemian
added a commit
that referenced
this issue
Dec 20, 2019
|
note: not seeing any test code: https://github.com/prjemian/bstesting/blob/master/test4.py test database when not at APS 8-ID-I: https://github.com/prjemian/bstesting/blob/master/8idi.db |
prjemian
added a commit
that referenced
this issue
Dec 20, 2019
prjemian
added a commit
that referenced
this issue
Dec 26, 2019
prjemian
added a commit
that referenced
this issue
Dec 27, 2019
prjemian
added a commit
that referenced
this issue
Dec 27, 2019
prjemian
added a commit
that referenced
this issue
Dec 27, 2019
prjemian
added a commit
that referenced
this issue
Dec 27, 2019
prjemian
added a commit
that referenced
this issue
Dec 27, 2019
This was referenced Jan 6, 2020
|
With #788, I'm declaring victory here and closing the issue as SOLVED. |
This was referenced Jan 27, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
One of the chronic problems that plague the APS beam lines from relying on Bluesky for user operations is
TimeoutError: Failed to read {some PV} within None secthat happens, seemingly at random. When this happens, the exception stops the running plans completely. Keep in mind, such timeouts are not on initial connection but happen after innumerable previous successful EPICS CA monitor updates on the same. For example, sometimes it fails on the engineering units of an EPICS motor (motor.EGU) that, on point number 47 of a step scan, raises a TimeoutError. The previous steps did not raise this error.Here is an example from interactive use where the .
TDIR(motor target direction changed sign, went the other way) raised a TimeoutError. ironically, this field is of no use to the bluesky user so this failure is unnecessary, at best (that's an ophyd device configuration issue). Of concern is the timeoutwith None secand why a timeout at all.By the way,
The text was updated successfully, but these errors were encountered: