-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TimeoutError from EpicsSignal #776
Comments
And, another example during a scan:
|
I suspect this is a bug in ophyd, pyepics, or more specifically the integration between the two (which lives in ophyd). I notice you are running on an ophyd prerelease, |
Yes. It's time to document this and catch the root cause
…On Mon, Nov 25, 2019, 12:45 PM Dan Allan ***@***.***> wrote:
I suspect this is a bug in ophyd, pyepics, or more specifically the
integration between the two (which lives in ophyd). I notice you are
running on an ophyd prerelease, 1.4.0rc1. Have you ever seen this on a
version in the 1.3.x series?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#776?email_source=notifications&email_token=AARMUMDUW4EATHSYCCTPR2DQVQMK5A5CNFSM4JRLGCSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFDMYQI#issuecomment-558287937>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARMUMBXE7HFB6K2AJZYPADQVQMK5ANCNFSM4JRLGCSA>
.
|
Timeout=None sounds like it comes from ophyd, agree?
…On Mon, Nov 25, 2019, 12:53 PM Pete Jemian ***@***.***> wrote:
Yes. It's time to document this and catch the root cause
On Mon, Nov 25, 2019, 12:45 PM Dan Allan ***@***.***> wrote:
> I suspect this is a bug in ophyd, pyepics, or more specifically the
> integration between the two (which lives in ophyd). I notice you are
> running on an ophyd prerelease, 1.4.0rc1. Have you ever seen this on a
> version in the 1.3.x series?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#776?email_source=notifications&email_token=AARMUMDUW4EATHSYCCTPR2DQVQMK5A5CNFSM4JRLGCSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFDMYQI#issuecomment-558287937>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AARMUMBXE7HFB6K2AJZYPADQVQMK5ANCNFSM4JRLGCSA>
> .
>
|
BUT, this line in the error dump:
calls Within PyEpics, there are PV objects and Motor objects. This is a PV object. The call chain appears to be:
Here,
and then watches for a timeout. Is there code in ophyd that raises the TimeoutError exception? Not seeing that exception from PyEpics. see this table:
Time to look for the relevant TimeoutError in ophyd. |
Lines 824 to 831 in 9da7884
In the wrong set of circumstances that trigger our problem, the The value of the timeout is not the real problem, though, only a symptom. Why do we get a timeout at all? First off, though, we don't know how long we have waited. |
Actually, why is
The |
Here's the simple example from SO:
|
Using
|
Perhaps converting these warnings to code:
output:
|
One possible revision of Lines 824 to 831 in 9da7884
self.wait_for_connection(timeout=connection_timeout)
try:
warnings.filterwarnings("error")
t0 = time.time()
info = self._read_pv.get_with_metadata(
as_string=as_string, form=form, **kwargs)
except UserWarning as exc:
print(f"while getting from {self._read_pvname}, caught '{exc}'")
finally:
dt = time.time() - t0
warnings.filterwarnings("default")
if info is None:
timeout = kwargs.get('timeout', None)
raise TimeoutError(f'Failed to read {self._read_pvname} in '
f'{dt:.3f} sec (with timeout={timeout})') |
As commented for APS 8-ID-I XPCS, a ...
|
@kmpeters reported similar experience with PyEpics (not bluesky):
|
@kmpeters - Can you post a snippet of the safer |
It looks like the latest version of my fly-scan code only has a safer #
# Debugging
#
def debugPut(self, pv, val, wait=False, use_complete=False, timeout=30.0):
#
exitFlag = False
while ( not exitFlag ):
try:
pv.put(val, wait=wait, use_complete=use_complete, timeout=timeout)
except WindowsError as e:
print
print "EPICS put FAILED:"
print "\tPV =", pv.pvname
print "\tval =", val
print "\twait =", wait, "; use_complete =", use_complete, "; timeout =", timeout
print "\tconnected =", pv.connected, "; host =", pv.host
print
print "A WindowsError occurred:"
# The following line results in an AttributeError. Apparently e.type doesn't exist
#!print "\t", e.type
print "\t", e
print "\t", e.args
print
print "Backtrace:"
traceback.print_stack()
print
time.sleep(15.00)
else:
exitFlag = True It also looks like it wasn't a |
Thanks!
…On Tue, Dec 10, 2019, 1:13 PM Kevin Peterson ***@***.***> wrote:
It looks like the latest version of my fly-scan code only has a safer
put():
#
# Debugging
#
def debugPut(self, pv, val, wait=False, use_complete=False, timeout=30.0):
#
exitFlag = False
while ( not exitFlag ):
try:
pv.put(val, wait=wait, use_complete=use_complete, timeout=timeout)
except WindowsError as e:
print
print "EPICS put FAILED:"
print "\tPV =", pv.pvname
print "\tval =", val
print "\twait =", wait, "; use_complete =", use_complete, "; timeout =", timeout
print "\tconnected =", pv.connected, "; host =", pv.host
print
print "A WindowsError occurred:"
# The following line results in an AttributeError. Apparently e.type doesn't exist
#!print "\t", e.type
print "\t", e
print "\t", e.args
print
print "Backtrace:"
traceback.print_stack()
print
time.sleep(15.00)
else:
exitFlag = True
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#776?email_source=notifications&email_token=AARMUMFCVRDHZAMON6EJNZ3QX7S5LA5CNFSM4JRLGCSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGQNICY#issuecomment-564188171>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARMUMAJVPHCSMYNYULWQ4LQX7S5LANCNFSM4JRLGCSA>
.
|
The safer |
@kmpeters : Looking over the code you supplied to make a safer I'll begin testing the idea of trapping a |
@kmpeters: Trapping the |
I think |
The first item to resolve in this issue is to capture the |
note: not seeing any test code: https://github.com/prjemian/bstesting/blob/master/test4.py test database when not at APS 8-ID-I: https://github.com/prjemian/bstesting/blob/master/8idi.db
|
With #788, I'm declaring victory here and closing the issue as SOLVED. |
One of the chronic problems that plague the APS beam lines from relying on Bluesky for user operations is
TimeoutError: Failed to read {some PV} within None sec
that happens, seemingly at random. When this happens, the exception stops the running plans completely. Keep in mind, such timeouts are not on initial connection but happen after innumerable previous successful EPICS CA monitor updates on the same. For example, sometimes it fails on the engineering units of an EPICS motor (motor.EGU) that, on point number 47 of a step scan, raises a TimeoutError. The previous steps did not raise this error.Here is an example from interactive use where the .
TDIR
(motor target direction changed sign, went the other way) raised a TimeoutError. ironically, this field is of no use to the bluesky user so this failure is unnecessary, at best (that's an ophyd device configuration issue). Of concern is the timeoutwith None sec
and why a timeout at all.By the way,
The text was updated successfully, but these errors were encountered: