This repository has been archived by the owner on Apr 19, 2024. It is now read-only.
guard slit motors sometimes stuck in MOVING state #253
Comments
|
Are these the only motors that have this issue? If so, the problem is likely to be in the 4slit support rather than the motor support. |
|
What is the state of the following PVs when the motors get stuck in the moving state? |
|
These are the only motors we are having troubles with. I agree that the problem is in the EPICS IOC layer, not bluesky, since we also see the problem using a GUI. Your suggestion of the 4slit support seems a good place to look. It will take a bit to run the diagnostics since users are measuring data now. We'll get to it but with some delay. |
|
If the problem is with the 4slit database, this can be run offline in a softIOC that simulates the guard slits. See the documentation. We are using the 2slit_soft database that uses soft motors rather than ao records. |
|
I've got a softIOC ready to test this:
Note that there are later updates to this support that might need an IOC update to apply:
|
|
correction:
|
|
The softIOC is not ready for testing now due to IT maintenance. The complete IOC software stack must be rebuilt after the workstation has moved to a new subnet. |
|
What are the symptoms that tell you the complete IOC software stack must be rebuilt? |
|
My desktop computer was relocated to the new subnet and none of the files
exist.
…On Wed, Sep 18, 2019, 10:27 AM Kevin Peterson ***@***.***> wrote:
What are the symptoms that tell you the complete IOC software stack must
be rebuilt?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#253>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AARMUMHYHLVEIDDEMHPZCXLQKJCHJANCNFSM4HZEN6QQ>
.
|
prjemian
modified the milestones:
2020-01 shutdown,
2020-02 operations,
2020-04 operations
prjemian
added a commit
that referenced
this issue
Feb 24, 2020
|
Restarting this:
|
|
|
stopping the current retest, since the motor stuck problem is not found with this configuration |
|
For the record, started scanning again with a revised algorithm to prune old data from the plots. The new algorithm is working but needs more testing. No observations of the original problem (motor stuck in moving state). Still, the retest here stopped (during iteration 129) with the same timeout error when trying to read console output from just before error through to command line promptIt's not obvious why a timeout of 2.0 is used since the session is setup with a default timeout of 10 s: # set default timeout for all EpicsSignal connections & communications
import ophyd
ophyd.EpicsSignal.set_default_timeout(timeout=10, connection_timeout=5)Will it help to run this with a source code debugger? |
|
note:
most recentearlierIn this summary plot, the orange points show the time (seconds) for each tune run, the blue points show the time (minutes) for a set of six tune runs (one iteration). The horizontal axis is the iteration counter before the current (so older runs to the left, newer increasing to the right). The start of iterations is on the left.
Scan times start increasing noticeably about 4.5 hours after start. script to get start time of each run: Note: the |
|
Another observation: When exiting the ipython console session after this error, lots of error output on the console. Suspect this is directly related to the timeout and the increasing scan time. 512 instances of |
|
Error comes from PyEpics: |
|
called from ophyd.signal, this is one of the error reports: |
|
Regarding the slow search as noted above, this is much faster (using the PyMongo API directly) but also might break in future versions of databroker: |
|
@jilavsky : Can we run this kind of testing on the actual guard slit hardware now? |
|
Sorry for missing this. We can run real hardware now. It is available. Well, motors are probably off at this time and no X-rays. |
3 tasks
|
I'll try tomorrow when not in other meetings. Should I contact you
first? If motors are off, I see little problem.
…On 9/2/2020 9:56 PM, Jan Ilavsky wrote:
Sorry for missing this. We can run real hardware now. It is available.
Well, motors are probably off at this time and no X-rays.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#253 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AARMUMGHPUHIRBI4DYWQZLTSD4AXBANCNFSM4HZEN6QQ>.
|
|
test code installed at USAXS instrument (in data directory """
work on GitHub issue #253
%run -im issue253
uids = RE(issue253_plan(times=10))
* guard slit motors sometimes stuck in MOVING state
* https://github.com/APS-USAXS/ipython-usaxs/issues/253
Scan the guard slit motors repeatedly.
Ignore all the other activities associated when
guard slits are "tuned". Use the "noisy" detector
from ophyd's simulators.
``
In [4]: wa gslit
gslit
Positioner Value Low Limit High Limit Offset
guard_slit_bot -0.5 -11.3694 12.6306 -7.0664
guard_slit_inb -0.5 -10.3912 13.6088 -33.0632
guard_slit_outb 0.5 -9.403 14.597 507.34
guard_slit_top 0.5 -13.9196 10.0804 -1.8856
guard_slit_x -1.105 -73.88 28.585 -42.65
guard_slit_y 14.6773 -17.92 59.00621 10.44885
``
"""
from instrument.session_logs import logger
logger.info(__file__)
from bluesky import plan_stubs as bps
from bluesky import plans as bp
from instrument.devices import guard_slit
from instrument.framework import bec, RE
import matplotlib.pyplot as plt
from ophyd.sim import noisy_det
_total_tunes = 0
_tune_number = 0
def scan_xy(detectors, md=None):
_md = dict()
_md.update(md or {})
yield from bp.rel_scan(detectors, guard_slit.y, .1, -.1, 21, md=_md)
yield from bp.rel_scan(detectors, guard_slit.x, .1, -.1, 21, md=_md)
def scan_opening(detectors, md=None):
_md = dict()
_md.update(md or {})
yield from bp.rel_scan(detectors, guard_slit.top, .1, -.1, 21, md=_md)
yield from bp.rel_scan(detectors, guard_slit.bot, -.1, .1, 21, md=_md)
yield from bp.rel_scan(detectors, guard_slit.outb, .1, -.1, 21, md=_md)
yield from bp.rel_scan(detectors, guard_slit.inb, -.1, .1, 21, md=_md)
def select_figure(x, y):
figure_name = f"{y.name} vs {x.name}"
if figure_name in plt.get_figlabels():
print(figure_name)
return plt.figure(figure_name)
def select_live_plot(bec, signal):
for live_plot_dict in bec._live_plots.values():
live_plot = live_plot_dict.get(signal.name)
if live_plot is not None:
return live_plot
def trim_plot(bec, n, x, y):
liveplot = select_live_plot(bec, y)
if liveplot is None:
return
fig = select_figure(x, y)
if fig is None:
return
ax = fig.axes[0]
while len(ax.lines) > n:
try:
ax.lines.pop(0).remove()
except ValueError as exc:
logger.warning(
"%s vs %s: mpl remove() error: %s",
y.name, x.name, str(exc))
ax.legend()
liveplot.update_plot()
def trim_all(n=3):
for axis in "y x top bot outb inb".split(): # in order of tune
trim_plot(bec, n, getattr(guard_slit, axis), noisy_det)
def scan_guard_slit(md=None):
global _tune_number, _total_tunes
_tune_number += 1
_md = dict(
purpose="test if guard slit motors sometimes stuck in MOVING state",
URL="https://github.com/APS-USAXS/ipython-usaxs/issues/253",
tune_number=_tune_number,
tune=f"tune_Gslits() {_tune_number} of {_total_tunes}",
)
_md.update(md or {})
logger.info("# tune number: %d", _tune_number)
detectors = [noisy_det,]
yield from scan_xy(detectors, md=_md)
yield from scan_opening(detectors, md=_md)
yield from bps.sleep(3.0) # allow all plots to finish drawing
trim_all()
def issue253_plan(times=1):
"""
repeat the scans of the guard slit axes
uids = RE(issue253_plan(times=2))
"""
global _tune_number, _total_tunes
logger.info("# ------- Testing for issue #253 with %d iterations", times)
_tune_number = 0
_total_tunes = times
yield from bps.repeat(scan_guard_slit, num=times) |
|
short test in progress to verify code operation:
|
|
@jilavsky - I'll queue up 1000 repetitions of the six-motor "pseudo-tune" of the guard slits, looking for the original problem where one of the motors gets stuck in MOVING state. This should be run from the Jump session so you, too, can see what is happening. |
|
Test of 1000 iterations started at ~18:28
|
|
Note that old lines from each plot are discarded. At most, 3 or 4 lines on each. |
|
172 iterations at 9:45 pm and no stuck MOVING (~52 iterations/hr, 1000 iterations should be complete by ~1:30 pm tomorrow) |
|
664 iterations at 7:15 AM and no stuck MOVING |
|
RunEngine stopped with an exception after 732 iterations, 8:34 AM with a A second exception |
|
Ouch. Stopped our test cold. But, @jilavsky @kmpeters would you both conclude that we are not seeing the original problem (guard slit motors sometimes stuck in MOVING state) now? Could it be the problem is resolved (by other software changes) or is there something to be changed in the test method here? |
|
I agree, that we are not seeing motor hanging. I do not see (or cannot find) the code being run. Tuning GSlits involves multiple steps. Scanning up to 6 motors. Obviously, that works, tested lots of times. Finding center positions (just code, cannot hang motors). And setting motors to proper positions. For x and y this is simple motor move to calculated center. But for slit blades this is using the epics soft slits record or whatever the name is. That last step was what we suspected from failing. Was that tested in these scans? I am not sure how that is used in BS, would need to look at the code. |
|
@jilavsky, @kmpeters : Thanks for your reviews. The test code is in the Details dropdown of this message (above): #253 (comment) The code was modeled after exactly the operations taken by the Tune_GuardSlits code. It scans all six motors in the same sequence (y, x, top, bottom, outboard, inboard). Since the result of each tune is unimportant, that was not done in the test code above. If the tunes had scanned (tuned) either slit's gap or center, that would have used part of the 2slit.db code. However, operation of any of the real motors exercises the 2slit.db code as it evaluates the effects on the virtual motors. |
|
Leave this issue open for now. If problem is not seen again, close automatically at next milestone. Moving to that milestone now. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
@kmpeters We are still having troubles with the guard slit motors getting stuck in MOVING state. Sometimes, we find this while running bluesky, sometimes while moving them from caQtDM. Always, this involves a session where we are using both tools.
Tried adding an explicit
caput MOTOR.STUP 1(status update) for each motor9idcLAX:max:c0:m{3,4,5,6]but that did not help us. (EDIT: Actually, the set to.STUPprovoked an additional problem. Seems to be a distraction.)The text was updated successfully, but these errors were encountered: