Read timeout stopped measurements. #233

prjemian · 2020-10-03T14:39:10Z

@prjemian C034 crashed at measurement 08539 at 4:21 am on 10/03

Error from terminal


C034_CPMV_att00_08524
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08524/C034_CPMV_att00_08524_0001-100000.hdf
C034_CPMV_att00_08525
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08525/C034_CPMV_att00_08525_0001-100000.hdf
C034_CPMV_att00_08526
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08526/C034_CPMV_att00_08526_0001-100000.hdf
C034_CPMV_att00_08527                                                                               
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08527/C034_CPMV_att00_08527_0001-100000.hdf
C034_CPMV_att00_08528
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08528/C034_CPMV_att00_08528_0001-100000.hdf
C034_CPMV_att00_08529
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08529/C034_CPMV_att00_08529_0001-100000.hdf
C034_CPMV_att00_08530
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08530/C034_CPMV_att00_08530_0001-100000.hdf
C034_CPMV_att00_08531
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08531/C034_CPMV_att00_08531_0001-100000.hdf
C034_CPMV_att00_08532
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08532/C034_CPMV_att00_08532_0001-100000.hdf
C034_CPMV_att00_08533
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08533/C034_CPMV_att00_08533_0001-100000.hdf
C034_CPMV_att00_08534
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08534/C034_CPMV_att00_08534_0001-100000.hdf
C034_CPMV_att00_08535
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08535/C034_CPMV_att00_08535_0001-100000.hdf
C034_CPMV_att00_08536
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08536/C034_CPMV_att00_08536_0001-100000.hdf
C034_CPMV_att00_08537
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08537/C034_CPMV_att00_08537_0001-100000.hdf
C034_CPMV_att00_08538
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/C034_CPMV_att00_08538/C034_CPMV_att00_08538_0001-100000.hdf
C034_CPMV_att00_08539
/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/epics/ca.py:1524: UserWarning: ca.get('S:FillNumber') timed out after 2.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
---------------------------------------------------------------------------
ReadTimeoutError                          Traceback (most recent call last)
<ipython-input-2-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in multi_capil()
     94     z0 = 0.5
     95 
---> 96     yield from repeat_acq(sample_name='CPMV',att_value=0,acq_rep=40000,samx_cen=2.2,samz_cen=z0)
     97 
     98     bec.enable_plots()

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
     69         print(acq_name)
     70 
---> 71         yield from AD_Acquire(
     72             rigaku,
     73             acquire_time=19.00e-6,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    317 
    318     logger.info("calling full_acquire_procedure()")
--> 319     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md)
    285         logger.debug("before count()")
    286         # yield from bp.count([areadet], md=md)
--> 287         yield from inner_count([areadet], md=_md)
    288         logger.debug("after count()")
    289 

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in inner_count(devices, md)
    236 
    237     def inner_count(devices, md={}):
--> 238         yield from bps.open_run(md=md)
    239         for obj in devices:
    240             yield from bps.stage(obj)

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in open_run(md)
    821     :func:`bluesky.plans_stubs.close_run`
    822     """
--> 823     return (yield Msg('open_run', **(md or {})))
    824 
    825 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    139         the input message
    140     '''
--> 141     return (yield msg)
    142 
    143 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    139         the input message
    140     '''
--> 141     return (yield msg)
    142 
    143 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    139         the input message
    140     '''
--> 141     return (yield msg)
    142 
    143 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in trigger_and_read(devices, name)
    909         return ret
    910     from .preprocessors import rewindable_wrapper
--> 911     return (yield from rewindable_wrapper(inner_trigger_and_read(),
    912                                           rewindable))
    913 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in rewindable_wrapper(plan, rewindable)
    691                                             restore_rewindable()))
    692     else:
--> 693         return (yield from plan)
    694 
    695 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in inner_trigger_and_read()
    903         ret = {}  # collect and return readings to give plan access to them
    904         for obj in devices:
--> 905             reading = (yield from read(obj))
    906             if reading is not None:
    907                 ret.update(reading)

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in read(obj)
     93         Msg('read', obj)
     94     """
---> 95     return (yield Msg('read', obj))
     96 
     97 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1423                         # exceptions (coming in via throw) can be
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426 
   1427                     # special case `CancelledError` and let the outer

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _read(self, msg)
   1668         obj = msg.obj
   1669         # actually _read_ the object
-> 1670         ret = obj.read(*msg.args, **msg.kwargs)
   1671 
   1672         if ret is None:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/device.py in read(self)
   1235 
   1236         for _, component in self._get_components_of_kind(Kind.normal):
-> 1237             res.update(component.read())
   1238         return res
   1239 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/utils/epics_pvs.py in wrapper(self, *args, **kwargs)
    196     def wrapper(self, *args, **kwargs):
    197         if self.connected:
--> 198             return fcn(self, *args, **kwargs)
    199         else:
    200             raise DisconnectedError('{} is not connected'.format(self.name))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in read(self)
    364             dict
    365         '''
--> 366         value = self.get()
    367         return {self.name: {'value': value,
    368                             'timestamp': self.timestamp}}

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in get(self, as_string, timeout, connection_timeout, form, **kwargs)
   1093             as_string = self._string
   1094 
-> 1095         info = self._get_with_timeout(
   1096             self._read_pv, timeout, connection_timeout, as_string, form)
   1097 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in _get_with_timeout(self, pv, timeout, connection_timeout, as_string, form)
   1056 
   1057         if info is None:
-> 1058             raise ReadTimeoutError(
   1059                 f"Failed to read {pv.pvname} "
   1060                 f"within {timeout:.2} sec")

ReadTimeoutError: Failed to read S:FillNumber within 2.0 sec

Originally posted by @Kacperswitalski in #231 (comment)

The text was updated successfully, but these errors were encountered:

prjemian · 2020-10-03T14:40:54Z

As stated in #231 (comment):

Ok, that's a different failure (read timeout after 2.0 seconds). Ironic that the read failed while recording metadata (the storage ring fill number) that is not essential to the measurement or analysis, only useful for diagnostics later. ☹️

Thought we had a longer timeout than 2 seconds. Needs investigating why this had only a 2 second timeout.

prjemian · 2020-10-03T14:42:55Z

The default (communications) timeout is set to 10 s for all EPICS PVs:

ipython-8idiuser/profile_bluesky/startup/instrument/framework/initialize.py

Lines 123 to 125 in 7d5ee99

    
           # set default timeout for all EpicsSignal connections & communications 
        
           import ophyd 
        
           ophyd.EpicsSignal.set_default_timeout(timeout=10, connection_timeout=5)

prjemian · 2020-10-03T14:57:28Z

At APS, empirical evidence shows that beam lines all around the ring experience random slow response times for EPICS caget. The usual remedy has been to either increase the timeout permitted or to retry. After discussion, a configurable default timeout was added and that is the code shown above. This comment compares retries v. longer timeouts.

The USAXS instrument had a very similar issue with this last week. Looks like the same thing to me.

So the question is:
Why does a baseline signal (ophyd object: aps.fill_number, PV: S:FillNumber) still have a timeout of 2 seconds while we have called ophyd.EpicsSignal.set_default_timeout(timeout=10, connection_timeout=5) before any signals were connected to PVs?

@danielballan, @tcaswell, @MikeHart85: Your comments and suggestions are most welcome.

prjemian · 2020-10-03T15:09:18Z

Here all the places that timeout (upper or lower case) appears in the instrument package (spoiler: it's not set to 2 s anywhere in this code):

profile_bluesky/startup/instrument/devices/epid.py:    def wait_until_settled(self, timeout=None, timeout_fail=False):
profile_bluesky/startup/instrument/devices/epid.py:                if timeout is not None and elapsed > timeout:
profile_bluesky/startup/instrument/devices/epid.py:                    msg = f"PID loop '{self.name}' Timeout after {elapsed:.2f}s"
profile_bluesky/startup/instrument/devices/epid.py:                    if timeout_fail:
profile_bluesky/startup/instrument/devices/epid.py:                        raise TimeoutError(msg)
profile_bluesky/startup/instrument/devices/lakeshore.py:    def wait_until_settled(self, timeout=None, timeout_fail=False):
profile_bluesky/startup/instrument/devices/lakeshore.py:                if timeout is not None and elapsed > timeout:
profile_bluesky/startup/instrument/devices/lakeshore.py:                    msg = f"{self.controller_name} Timeout after {elapsed:.2f}s"
profile_bluesky/startup/instrument/devices/lakeshore.py:                    if timeout_fail:
profile_bluesky/startup/instrument/devices/lakeshore.py:                        raise TimeoutError(msg)
profile_bluesky/startup/instrument/devices/lambda_750k.py:            #         raise TimeoutError(emsg)
profile_bluesky/startup/instrument/devices/lambda_750k.py:except TimeoutError:
profile_bluesky/startup/instrument/devices/rigaku_ufxc.py:except TimeoutError:
profile_bluesky/startup/instrument/devices/scaler.py:    except TimeoutError:
profile_bluesky/startup/instrument/devices/scaler.py:        # no need to sleep since each connection attempt has 1 second timeout
profile_bluesky/startup/instrument/devices/scaler.py:    raise TimeoutError(emsg)
profile_bluesky/startup/instrument/framework/initialize.py:# set default timeout for all EpicsSignal connections & communications
profile_bluesky/startup/instrument/framework/initialize.py:ophyd.EpicsSignal.set_default_timeout(timeout=10, connection_timeout=5)
profile_bluesky/startup/instrument/plans/xpcs_acquire.py:        except ophyd.signal.ReadTimeoutError as exc:
profile_bluesky/startup/instrument/plans/xpcs_acquire.py:            logger.warn("EPICS ReadTimeoutError from scaler (ignoring): %s", str(exc))
profile_bluesky/startup/instrument/utils/explorer.py:    except TimeoutError:
profile_bluesky/startup/instrument/utils/explorer.py:        print(f"timeout: {obj.name}_{nm}")
profile_bluesky/startup/instrument/utils/explorer.py:        return "TIMEOUT"
profile_bluesky/startup/instrument/utils/explorer.py:            if child in (None, "TIMEOUT"):
profile_bluesky/startup/spec_support/spec_DM_support.py:    def put(self, value, wait=False, timeout=30):
profile_bluesky/startup/spec_support/spec_DM_support.py:                response = self.pv.put(value, wait=wait, timeout=timeout)
profile_bluesky/startup/spec_support/spec_DM_support.py:                    self.registers.workflow_start.put(0, wait=True, timeout=0.1)

prjemian · 2020-10-03T15:14:22Z

Looking through the ophyd code (current github master branch), here is where timeout and 2 appear:

(base) jemian@wow ~/.../Bluesky/ophyd $ git grep -i timeout | grep 2
ophyd/_pyepics_shim.py:            timeout = 315569520  # ten years
ophyd/areadetector/plugins.py:def get_areadetector_plugin_class(prefix, timeout=2.0):
ophyd/device.py:    def wait_for_connection(self, all_signals=False, timeout=2.0):
ophyd/signal.py:    __default_timeout = 2.0
ophyd/signal.py:    def set_default_timeout(cls, *, timeout=2.0, connection_timeout=1.0):
ophyd/signal.py:                f"within {connection_timeout:.2f} sec") from err
ophyd/signal.py:                f"within {timeout:.2f} sec")
ophyd/tests/conftest.py:            sig.wait_for_connection(timeout=2)
ophyd/tests/test_epicsmotor.py:    motor.timeout = 20.0
ophyd/tests/test_epicsmotor.py:    assert motor.timeout == 20.0
ophyd/tests/test_epicsmotor.py:    motor.wait_for_connection(all_signals=True, timeout=2)
ophyd/tests/test_epicsmotor.py:    res = motor.home("forward", timeout=2, wait=False)
ophyd/tests/test_epicsmotor.py:    res = motor.home("reverse", timeout=2, wait=False)
ophyd/tests/test_epicsmotor.py:        res = motor.home("foobar", timeout=2, wait=False)
ophyd/tests/test_positioner.py:    p.timeout = 20.0
ophyd/tests/test_positioner.py:    assert p.timeout == 20.0
ophyd/tests/test_positioner.py:    p.move(target_pos, timeout=2, wait=True)
ophyd/tests/test_pseudopos.py:        "", name="mypseudo", concurrent=True, settle_time=0.1, timeout=25.0
ophyd/tests/test_pseudopos.py:    assert pseudo.timeout == 25.0
ophyd/tests/test_pseudopos.py:        ((1, 2, 3), {"timeout": 1}, (1, 2, 3), {"timeout": 1}),
ophyd/tests/test_pseudopos.py:        (((1, 2, 3),), {"timeout": 1}, (1, 2, 3), {"timeout": 1}),
ophyd/tests/test_pvpositioner.py:                       settle_time=0.1, timeout=25.0)
ophyd/tests/test_pvpositioner.py:    assert pos.timeout == 25.0
ophyd/tests/test_signal.py:    # move back to -0.2, forcing a timeout with a low value

prjemian · 2020-10-03T15:18:16Z

Specifically:

ophyd/signal.py:    __default_timeout = 2.0
ophyd/signal.py:    def set_default_timeout(cls, *, timeout=2.0, connection_timeout=1.0):

prjemian · 2020-10-03T15:19:05Z

Is this a local problem or an ophyd problem?

qzhang234 · 2020-10-03T17:31:26Z

@prjemian Is it better at the current stage to simply ignore the time out and make sure the scan continues? I have a hard time imaging that the users would need the storage ring signal for data analysis.

Kacperswitalski · 2020-10-04T22:41:24Z

@prjemian E038 crashed at measurement 14145 at 5:27 pm on 10/4

It looks like another timeout issue I believe

Error from terminal


E038_CPMV_6C_att00_14141
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14141/E038_CPMV_6C_att00_14141_0001-100000.hdf                
E038_CPMV_6C_att00_14142
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14142/E038_CPMV_6C_att00_14142_0001-100000.hdf                
E038_CPMV_6C_att00_14143
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14143/E038_CPMV_6C_att00_14143_0001-100000.hdf                
E038_CPMV_6C_att00_14144
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14144/E038_CPMV_6C_att00_14144_0001-100000.hdf                
E038_CPMV_6C_att00_14145
/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/epics/ca.py:1524: UserWarning: ca.get('8idi:Reg102') timed out after 10.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/epics/ca.py:1524: UserWarning: ca.get('8idi:Reg103') timed out after 10.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/epics/ca.py:1524: UserWarning: ca.get('8idi:Reg121') timed out after 10.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/epics/ca.py:1524: UserWarning: ca.get('8idi:Reg104') timed out after 10.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/epics/ca.py:1524: UserWarning: ca.get('8idi:Reg101') timed out after 10.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
---------------------------------------------------------------------------
FailedStatus                              Traceback (most recent call last)
<ipython-input-10-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1339                         # throw the exception at the current plan
   1340                         try:
-> 1341                             msg = self._plan_stack[-1].throw(
   1342                                 stashed_exception or resp)
   1343                         except Exception as e:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in multi_capil()
     94     z0 = 1.5
     95 
---> 96     yield from repeat_acq(sample_name='CPMV_6C',att_value=0,acq_rep=50000,samx_cen=-1.8,samz_cen=z0)
     97 
     98     bec.enable_plots()

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
     69         print(acq_name)
     70 
---> 71         yield from AD_Acquire(
     72             rigaku,
     73             acquire_time=19.00e-6,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    317 
    318     logger.info("calling full_acquire_procedure()")
--> 319     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    526         # https://docs.python.org/3/reference/expressions.html?#generator.close
    527         if cleanup:
--> 528             yield from ensure_generator(final_plan_instance)
    529     return ret
    530 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in unstage_devices()
    944     def unstage_devices():
    945         for d in reversed(devices):
--> 946             yield Msg('unstage', d)
    947 
    948     def inner():

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

FailedStatus: Status(obj=EpicsSignal(read_pv='8idi:Reg101', name='dm_pars_roi_x1', parent='dm_pars', value=0, timestamp=1601850460.1213074, auto_monitor=False, string=False, write_pv='8idi:Reg101', limits=False, put_complete=False), done=True, success=False)

Logger


|2020-10-04 17:26:56.824|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14137/
|2020-10-04 17:26:56.875|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-04 17:26:56.887|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-04 17:26:56.924|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-04 17:26:56.928|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1311937| - DM workflow kickoff done
|2020-10-04 17:26:56.928|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1311937| - b'id=ae03de23-17f1-473a-81d3-bb31f5eefcc5 owner=8idiuser status=pending startTime=1601850416.91 startTimestamp=2020/10/04 17:26:56 CDT \n'
|2020-10-04 17:26:56.982|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-04 17:26:57.195|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:26:57.195|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E038_CPMV_6C_att00_14137 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19647, info=psutil.Process(pid=19647, name='sh', started='17:26:56')
|2020-10-04 17:26:57.765|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:26:57.765|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19653, info=psutil.Process(pid=19653, name='sh', started='17:26:56')
|2020-10-04 17:27:01.680|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-1312029| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14137/E038_CPMV_6C_att00_14137_0001-100000.hdf
|2020-10-04 17:27:01.972|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E038_CPMV_6C_att00_14138
|2020-10-04 17:27:01.972|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-04 17:27:01.972|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14138/
|2020-10-04 17:27:01.992|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-04 17:27:02.002|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-04 17:27:02.069|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-04 17:27:02.084|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1312029| - DM workflow kickoff done
|2020-10-04 17:27:02.085|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1312029| - b'id=6d798821-d6eb-405f-9e17-189e7fd17147 owner=8idiuser status=pending startTime=1601850422.07 startTimestamp=2020/10/04 17:27:02 CDT \n'
|2020-10-04 17:27:02.160|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-04 17:27:02.391|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:02.391|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E038_CPMV_6C_att00_14138 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19781, info=psutil.Process(pid=19781, name='sh', started='17:27:01')
|2020-10-04 17:27:02.962|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:02.962|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19787, info=psutil.Process(pid=19787, name='sh', started='17:27:02')
|2020-10-04 17:27:07.220|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-1312121| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14138/E038_CPMV_6C_att00_14138_0001-100000.hdf
|2020-10-04 17:27:07.506|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E038_CPMV_6C_att00_14139
|2020-10-04 17:27:07.506|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-04 17:27:07.506|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14139/
|2020-10-04 17:27:07.528|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-04 17:27:07.571|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-04 17:27:07.607|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1312121| - DM workflow kickoff done
|2020-10-04 17:27:07.607|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1312121| - b'id=3140ead8-bcf5-4975-9a1b-0797da9dcec6 owner=8idiuser status=pending startTime=1601850427.59 startTimestamp=2020/10/04 17:27:07 CDT \n'
|2020-10-04 17:27:07.610|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-04 17:27:07.669|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-04 17:27:07.881|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:07.881|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E038_CPMV_6C_att00_14139 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19924, info=psutil.Process(pid=19924, name='sh', started='17:27:07')
|2020-10-04 17:27:08.427|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:08.427|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19930, info=psutil.Process(pid=19930, name='sh', started='17:27:07')
|2020-10-04 17:27:12.492|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-1312213| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14139/E038_CPMV_6C_att00_14139_0001-100000.hdf
|2020-10-04 17:27:12.774|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E038_CPMV_6C_att00_14140
|2020-10-04 17:27:12.774|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-04 17:27:12.774|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14140/
|2020-10-04 17:27:12.797|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-04 17:27:12.809|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-04 17:27:12.878|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-04 17:27:12.894|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1312213| - DM workflow kickoff done
|2020-10-04 17:27:12.895|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1312213| - b'id=02660907-ec40-4c48-87e2-479e2bbd78d0 owner=8idiuser status=pending startTime=1601850432.88 startTimestamp=2020/10/04 17:27:12 CDT \n'
|2020-10-04 17:27:12.960|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-04 17:27:13.178|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:13.179|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E038_CPMV_6C_att00_14140 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20058, info=psutil.Process(pid=20058, name='sh', started='17:27:12')
|2020-10-04 17:27:13.720|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:13.721|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20064, info=psutil.Process(pid=20064, name='sh', started='17:27:12')
|2020-10-04 17:27:18.114|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-1312305| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14140/E038_CPMV_6C_att00_14140_0001-100000.hdf
|2020-10-04 17:27:18.391|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E038_CPMV_6C_att00_14141
|2020-10-04 17:27:18.392|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-04 17:27:18.392|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14141/
|2020-10-04 17:27:18.410|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-04 17:27:18.422|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-04 17:27:18.486|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-04 17:27:18.501|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1312305| - DM workflow kickoff done
|2020-10-04 17:27:18.501|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1312305| - b'id=afc4cf18-3358-49b4-8338-72c2ab77ce89 owner=8idiuser status=pending startTime=1601850438.48 startTimestamp=2020/10/04 17:27:18 CDT \n'
|2020-10-04 17:27:18.561|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-04 17:27:18.782|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:18.782|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E038_CPMV_6C_att00_14141 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20191, info=psutil.Process(pid=20191, name='sh', started='17:27:18')
|2020-10-04 17:27:19.243|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:19.243|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20201, info=psutil.Process(pid=20201, name='sh', started='17:27:18')
|2020-10-04 17:27:23.476|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-1312397| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14141/E038_CPMV_6C_att00_14141_0001-100000.hdf
|2020-10-04 17:27:23.768|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E038_CPMV_6C_att00_14142
|2020-10-04 17:27:23.768|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-04 17:27:23.768|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14142/
|2020-10-04 17:27:23.791|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-04 17:27:23.804|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-04 17:27:23.858|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1312397| - DM workflow kickoff done
|2020-10-04 17:27:23.858|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1312397| - b'id=930b3c70-1358-4a4a-9262-c09f8a8b1b49 owner=8idiuser status=pending startTime=1601850443.84 startTimestamp=2020/10/04 17:27:23 CDT \n'
|2020-10-04 17:27:23.871|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-04 17:27:23.983|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-04 17:27:24.222|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:24.223|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E038_CPMV_6C_att00_14142 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20328, info=psutil.Process(pid=20328, name='sh', started='17:27:23')
|2020-10-04 17:27:24.769|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:24.769|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20334, info=psutil.Process(pid=20334, name='sh', started='17:27:24')
|2020-10-04 17:27:28.876|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-1312489| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14142/E038_CPMV_6C_att00_14142_0001-100000.hdf
|2020-10-04 17:27:29.172|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E038_CPMV_6C_att00_14143
|2020-10-04 17:27:29.172|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-04 17:27:29.172|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14143/
|2020-10-04 17:27:29.195|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-04 17:27:29.206|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-04 17:27:29.261|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1312489| - DM workflow kickoff done
|2020-10-04 17:27:29.261|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1312489| - b'id=ddc4bba9-cd02-40d1-ab27-7988a2a03ebd owner=8idiuser status=pending startTime=1601850449.25 startTimestamp=2020/10/04 17:27:29 CDT \n'
|2020-10-04 17:27:29.272|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-04 17:27:29.327|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-04 17:27:29.541|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:29.542|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E038_CPMV_6C_att00_14143 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20461, info=psutil.Process(pid=20461, name='sh', started='17:27:28')
|2020-10-04 17:27:30.017|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:30.017|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20470, info=psutil.Process(pid=20470, name='sh', started='17:27:29')
|2020-10-04 17:27:34.331|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-1312581| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14143/E038_CPMV_6C_att00_14143_0001-100000.hdf
|2020-10-04 17:27:34.623|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E038_CPMV_6C_att00_14144
|2020-10-04 17:27:34.623|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-04 17:27:34.623|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14144/
|2020-10-04 17:27:34.668|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-04 17:27:34.681|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-04 17:27:34.719|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-04 17:27:34.735|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1312581| - DM workflow kickoff done
|2020-10-04 17:27:34.735|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1312581| - b'id=747bc4b6-3d78-4f85-bdaf-c207e5940169 owner=8idiuser status=pending startTime=1601850454.72 startTimestamp=2020/10/04 17:27:34 CDT \n'
|2020-10-04 17:27:34.776|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-04 17:27:34.992|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:34.993|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E038_CPMV_6C_att00_14144 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20597, info=psutil.Process(pid=20597, name='sh', started='17:27:34')
|2020-10-04 17:27:35.586|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 17:27:35.586|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20603, info=psutil.Process(pid=20603, name='sh', started='17:27:34')
|2020-10-04 17:27:39.725|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-1312673| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14144/E038_CPMV_6C_att00_14144_0001-100000.hdf
|2020-10-04 17:27:40.029|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E038_CPMV_6C_att00_14145
|2020-10-04 17:27:40.029|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-04 17:27:40.029|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E038_CPMV_6C_att00_14145/
|2020-10-04 17:27:40.083|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-04 17:27:40.097|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-04 17:27:50.969|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1312673| - DM workflow kickoff done
|2020-10-04 17:27:50.970|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1312673| - b'id=2c107702-2067-4b47-8bb3-1de6259eac40 owner=8idiuser status=pending startTime=1601850460.13 startTimestamp=2020/10/04 17:27:40 CDT \n'

edit: Shortly after I restarted I saw this come up in the terminal, not sure if unusual or not

Terminal message


39_CPMV_6C_att00_00214
rigaku [In progress. No progress bar available.]                                                                                             
CA.Client.Exception...............................................
    Warning: "Virtual circuit disconnect"
    Context: "lambda.xray.aps.anl.gov:5064"
    Source File: ../cac.cpp line 1236
    Current Time: Sun Oct 04 2020 18:02:04.379065452
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E039_CPMV_6C_att00_00214/E039_CPMV_6C_att00_00214_0001-100000.hdf                
E039_CPMV_6C_att00_00215

qzhang234 · 2020-10-05T04:09:37Z

E039 crashed at measurement 2242 at 21:06, 10/04:

ReadTimeoutError: Failed to read S:SRlifeTimeHrsCC within 2.0 sec

Error from terminal


E039_CPMV_6C_att00_02240
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E039_CPMV_6C_att00_02240/E039_CPMV_6C_att00_02240_0001-100000.hdf                
E039_CPMV_6C_att00_02241
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E039_CPMV_6C_att00_02241/E039_CPMV_6C_att00_02241_0001-100000.hdf                
E039_CPMV_6C_att00_02242
/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/epics/ca.py:1524: UserWarning: ca.get('S:SRlifeTimeHrsCC') timed out after 2.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
---------------------------------------------------------------------------
ReadTimeoutError                          Traceback (most recent call last)
<ipython-input-13-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in multi_capil()
     94     z0 = 1.5
     95 
---> 96     yield from repeat_acq(sample_name='CPMV_6C',att_value=0,acq_rep=50000,samx_cen=-1.8,samz_cen=z0)
     97 
     98     bec.enable_plots()

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
     69         print(acq_name)
     70 
---> 71         yield from AD_Acquire(
     72             rigaku,
     73             acquire_time=19.00e-6,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    317 
    318     logger.info("calling full_acquire_procedure()")
--> 319     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md)
    285         logger.debug("before count()")
    286         # yield from bp.count([areadet], md=md)
--> 287         yield from inner_count([areadet], md=_md)
    288         logger.debug("after count()")
    289 

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in inner_count(devices, md)
    236 
    237     def inner_count(devices, md={}):
--> 238         yield from bps.open_run(md=md)
    239         for obj in devices:
    240             yield from bps.stage(obj)

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in open_run(md)
    821     :func:`bluesky.plans_stubs.close_run`
    822     """
--> 823     return (yield Msg('open_run', **(md or {})))
    824 
    825 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    139         the input message
    140     '''
--> 141     return (yield msg)
    142 
    143 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    139         the input message
    140     '''
--> 141     return (yield msg)
    142 
    143 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    139         the input message
    140     '''
--> 141     return (yield msg)
    142 
    143 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in trigger_and_read(devices, name)
    909         return ret
    910     from .preprocessors import rewindable_wrapper
--> 911     return (yield from rewindable_wrapper(inner_trigger_and_read(),
    912                                           rewindable))
    913 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in rewindable_wrapper(plan, rewindable)
    691                                             restore_rewindable()))
    692     else:
--> 693         return (yield from plan)
    694 
    695 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in inner_trigger_and_read()
    903         ret = {}  # collect and return readings to give plan access to them
    904         for obj in devices:
--> 905             reading = (yield from read(obj))
    906             if reading is not None:
    907                 ret.update(reading)

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in read(obj)
     93         Msg('read', obj)
     94     """
---> 95     return (yield Msg('read', obj))
     96 
     97 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1423                         # exceptions (coming in via throw) can be
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426 
   1427                     # special case `CancelledError` and let the outer

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _read(self, msg)
   1668         obj = msg.obj
   1669         # actually _read_ the object
-> 1670         ret = obj.read(*msg.args, **msg.kwargs)
   1671 
   1672         if ret is None:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/device.py in read(self)
   1235 
   1236         for _, component in self._get_components_of_kind(Kind.normal):
-> 1237             res.update(component.read())
   1238         return res
   1239 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/utils/epics_pvs.py in wrapper(self, *args, **kwargs)
    196     def wrapper(self, *args, **kwargs):
    197         if self.connected:
--> 198             return fcn(self, *args, **kwargs)
    199         else:
    200             raise DisconnectedError('{} is not connected'.format(self.name))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in read(self)
    364             dict
    365         '''
--> 366         value = self.get()
    367         return {self.name: {'value': value,
    368                             'timestamp': self.timestamp}}

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in get(self, as_string, timeout, connection_timeout, form, **kwargs)
   1093             as_string = self._string
   1094 
-> 1095         info = self._get_with_timeout(
   1096             self._read_pv, timeout, connection_timeout, as_string, form)
   1097 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in _get_with_timeout(self, pv, timeout, connection_timeout, as_string, form)
   1056 
   1057         if info is None:
-> 1058             raise ReadTimeoutError(
   1059                 f"Failed to read {pv.pvname} "
   1060                 f"within {timeout:.2} sec")

ReadTimeoutError: Failed to read S:SRlifeTimeHrsCC within 2.0 sec

In [14]:

Error from logger file


|2020-10-04 21:06:17.456|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1518639| - DM workflow kickoff done
|2020-10-04 21:06:17.456|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1518639| - b'id=ce7ba1fc-fb56-4b53-88c8-ee09c8ba777d owner=8idiuser status=pending startTime=1601863577.44 startTimestamp=2020/10/04 21:06:17 CDT \n'
|2020-10-04 21:06:17.471|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-04 21:06:17.721|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 21:06:17.722|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E039_CPMV_6C_att00_02239 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=27885, info=psutil.Process(pid=27885, name='sh', started='21:06:16')
|2020-10-04 21:06:18.410|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 21:06:18.410|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=27891, info=psutil.Process(pid=27891, name='sh', started='21:06:17')
|2020-10-04 21:06:22.471|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-1518731| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E039_CPMV_6C_att00_02239/E039_CPMV_6C_att00_02239_0001-100000.hdf
|2020-10-04 21:06:22.769|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E039_CPMV_6C_att00_02240
|2020-10-04 21:06:22.769|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-04 21:06:22.769|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E039_CPMV_6C_att00_02240/
|2020-10-04 21:06:22.793|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-04 21:06:22.805|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-04 21:06:22.876|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-04 21:06:22.906|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1518731| - DM workflow kickoff done
|2020-10-04 21:06:22.906|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1518731| - b'id=fcae672e-d777-4719-98e5-7e8d0b064e53 owner=8idiuser status=pending startTime=1601863582.89 startTimestamp=2020/10/04 21:06:22 CDT \n'
|2020-10-04 21:06:22.958|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-04 21:06:23.204|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 21:06:23.204|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E039_CPMV_6C_att00_02240 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=28025, info=psutil.Process(pid=28025, name='sh', started='21:06:22')
|2020-10-04 21:06:23.794|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 21:06:23.794|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=28031, info=psutil.Process(pid=28031, name='sh', started='21:06:22')
|2020-10-04 21:06:27.900|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-1518823| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E039_CPMV_6C_att00_02240/E039_CPMV_6C_att00_02240_0001-100000.hdf
|2020-10-04 21:06:28.187|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E039_CPMV_6C_att00_02241
|2020-10-04 21:06:28.187|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-04 21:06:28.187|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E039_CPMV_6C_att00_02241/
|2020-10-04 21:06:28.208|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-04 21:06:28.218|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-04 21:06:28.300|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-04 21:06:28.350|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1518823| - DM workflow kickoff done
|2020-10-04 21:06:28.350|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1518823| - b'id=50c7cc9c-b8dd-47d7-8723-78ba12f81dc8 owner=8idiuser status=pending startTime=1601863588.33 startTimestamp=2020/10/04 21:06:28 CDT \n'
|2020-10-04 21:06:28.366|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-04 21:06:28.610|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 21:06:28.610|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E039_CPMV_6C_att00_02241 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=28165, info=psutil.Process(pid=28165, name='sh', started='21:06:27')
|2020-10-04 21:06:29.214|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-04 21:06:29.215|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=28172, info=psutil.Process(pid=28172, name='sh', started='21:06:28')
|2020-10-04 21:06:33.381|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-1518915| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E039_CPMV_6C_att00_02241/E039_CPMV_6C_att00_02241_0001-100000.hdf
|2020-10-04 21:06:33.670|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E039_CPMV_6C_att00_02242
|2020-10-04 21:06:33.670|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-04 21:06:33.670|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E039_CPMV_6C_att00_02242/
|2020-10-04 21:06:33.696|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-04 21:06:33.706|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-04 21:06:33.781|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-04 21:06:43.976|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-1518915| - DM workflow kickoff done
|2020-10-04 21:06:43.997|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-1518915| - b'id=c6912c20-5a7c-4f1f-8afd-722958e5ced6 owner=8idiuser status=pending startTime=1601863593.79 startTimestamp=2020/10/04 21:06:33 CDT \n'

qzhang234 · 2020-10-05T04:14:34Z

Scan restarted at 23:14, 10/04

prjemian · 2020-10-05T14:58:21Z

Reports of random caget timeouts from a variety of EPICS clients are common yet not frequent. See this issue for some discussion: bluesky/ophyd#776

See this issue comment for comparison of longer timeout v. retry: bluesky/ophyd#788 (comment) (summary: the preference depends on the EPICS client)

Kacperswitalski · 2020-10-05T15:23:53Z

@prjemian E040 crashed at measurement 06615 at 9:25 am on 10/5

Error from terminal


E040_CPMV_6C_att00_06601
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06601/E040_CPMV_6C_att00_06601_0001-100000.hdf                
E040_CPMV_6C_att00_06602
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06602/E040_CPMV_6C_att00_06602_0001-100000.hdf                
E040_CPMV_6C_att00_06603
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06603/E040_CPMV_6C_att00_06603_0001-100000.hdf                
E040_CPMV_6C_att00_06604
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06604/E040_CPMV_6C_att00_06604_0001-100000.hdf                
E040_CPMV_6C_att00_06605
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06605/E040_CPMV_6C_att00_06605_0001-100000.hdf                
E040_CPMV_6C_att00_06606
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06606/E040_CPMV_6C_att00_06606_0001-100000.hdf                
E040_CPMV_6C_att00_06607
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06607/E040_CPMV_6C_att00_06607_0001-100000.hdf                
E040_CPMV_6C_att00_06608
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06608/E040_CPMV_6C_att00_06608_0001-100000.hdf                
E040_CPMV_6C_att00_06609
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06609/E040_CPMV_6C_att00_06609_0001-100000.hdf                
E040_CPMV_6C_att00_06610
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06610/E040_CPMV_6C_att00_06610_0001-100000.hdf                
E040_CPMV_6C_att00_06611
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06611/E040_CPMV_6C_att00_06611_0001-100000.hdf                
E040_CPMV_6C_att00_06612
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06612/E040_CPMV_6C_att00_06612_0001-100000.hdf                
E040_CPMV_6C_att00_06613
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06613/E040_CPMV_6C_att00_06613_0001-100000.hdf                
E040_CPMV_6C_att00_06614
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06614/E040_CPMV_6C_att00_06614_0001-100000.hdf                
E040_CPMV_6C_att00_06615
/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/epics/ca.py:1524: UserWarning: ca.get('S:ActualMode') timed out after 2.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
---------------------------------------------------------------------------
ReadTimeoutError                          Traceback (most recent call last)
<ipython-input-16-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in multi_capil()
     94     z0 = 1.5
     95 
---> 96     yield from repeat_acq(sample_name='CPMV_6C',att_value=0,acq_rep=30000,samx_cen=-1.8,samz_cen=z0)
     97 
     98     bec.enable_plots()

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
     69         print(acq_name)
     70 
---> 71         yield from AD_Acquire(
     72             rigaku,
     73             acquire_time=19.00e-6,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    317 
    318     logger.info("calling full_acquire_procedure()")
--> 319     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md)
    285         logger.debug("before count()")
    286         # yield from bp.count([areadet], md=md)
--> 287         yield from inner_count([areadet], md=_md)
    288         logger.debug("after count()")
    289 

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in inner_count(devices, md)
    236 
    237     def inner_count(devices, md={}):
--> 238         yield from bps.open_run(md=md)
    239         for obj in devices:
    240             yield from bps.stage(obj)

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in open_run(md)
    821     :func:`bluesky.plans_stubs.close_run`
    822     """
--> 823     return (yield Msg('open_run', **(md or {})))
    824 
    825 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    139         the input message
    140     '''
--> 141     return (yield msg)
    142 
    143 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    139         the input message
    140     '''
--> 141     return (yield msg)
    142 
    143 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    139         the input message
    140     '''
--> 141     return (yield msg)
    142 
    143 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in trigger_and_read(devices, name)
    909         return ret
    910     from .preprocessors import rewindable_wrapper
--> 911     return (yield from rewindable_wrapper(inner_trigger_and_read(),
    912                                           rewindable))
    913 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in rewindable_wrapper(plan, rewindable)
    691                                             restore_rewindable()))
    692     else:
--> 693         return (yield from plan)
    694 
    695 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in inner_trigger_and_read()
    903         ret = {}  # collect and return readings to give plan access to them
    904         for obj in devices:
--> 905             reading = (yield from read(obj))
    906             if reading is not None:
    907                 ret.update(reading)

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in read(obj)
     93         Msg('read', obj)
     94     """
---> 95     return (yield Msg('read', obj))
     96 
     97 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1423                         # exceptions (coming in via throw) can be
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426 
   1427                     # special case `CancelledError` and let the outer

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _read(self, msg)
   1668         obj = msg.obj
   1669         # actually _read_ the object
-> 1670         ret = obj.read(*msg.args, **msg.kwargs)
   1671 
   1672         if ret is None:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/device.py in read(self)
   1235 
   1236         for _, component in self._get_components_of_kind(Kind.normal):
-> 1237             res.update(component.read())
   1238         return res
   1239 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/utils/epics_pvs.py in wrapper(self, *args, **kwargs)
    196     def wrapper(self, *args, **kwargs):
    197         if self.connected:
--> 198             return fcn(self, *args, **kwargs)
    199         else:
    200             raise DisconnectedError('{} is not connected'.format(self.name))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in read(self)
    364             dict
    365         '''
--> 366         value = self.get()
    367         return {self.name: {'value': value,
    368                             'timestamp': self.timestamp}}

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in get(self, as_string, timeout, connection_timeout, form, **kwargs)
   1093             as_string = self._string
   1094 
-> 1095         info = self._get_with_timeout(
   1096             self._read_pv, timeout, connection_timeout, as_string, form)
   1097 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in _get_with_timeout(self, pv, timeout, connection_timeout, as_string, form)
   1056 
   1057         if info is None:
-> 1058             raise ReadTimeoutError(
   1059                 f"Failed to read {pv.pvname} "
   1060                 f"within {timeout:.2} sec")

ReadTimeoutError: Failed to read S:ActualMode within 2.0 sec

Logger


|2020-10-05 09:24:32.906|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-05 09:24:32.976|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-05 09:24:32.982|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-2127081| - DM workflow kickoff done
|2020-10-05 09:24:32.982|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-2127081| - b'id=e9151676-dd2a-4c02-9547-6aaab0d52eaa owner=8idiuser status=pending startTime=1601907872.97 startTimestamp=2020/10/05 09:24:32 CDT \n'
|2020-10-05 09:24:33.060|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-05 09:24:33.301|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-05 09:24:33.301|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E040_CPMV_6C_att00_06609 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=21491, info=psutil.Process(pid=21491, name='sh', started='09:24:32')
|2020-10-05 09:24:33.884|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-05 09:24:33.884|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=21498, info=psutil.Process(pid=21498, name='sh', started='09:24:33')
|2020-10-05 09:24:38.201|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-2127173| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06609/E040_CPMV_6C_att00_06609_0001-100000.hdf
|2020-10-05 09:24:38.524|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E040_CPMV_6C_att00_06610
|2020-10-05 09:24:38.524|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-05 09:24:38.524|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06610/
|2020-10-05 09:24:38.569|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-05 09:24:38.578|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-05 09:24:38.614|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-05 09:24:38.631|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-2127173| - DM workflow kickoff done
|2020-10-05 09:24:38.632|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-2127173| - b'id=a1915a93-bff9-46bc-a5a0-4233f4b3157d owner=8idiuser status=pending startTime=1601907878.61 startTimestamp=2020/10/05 09:24:38 CDT \n'
|2020-10-05 09:24:38.673|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-05 09:24:38.909|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-05 09:24:38.909|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E040_CPMV_6C_att00_06610 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=21632, info=psutil.Process(pid=21632, name='sh', started='09:24:38')
|2020-10-05 09:24:39.477|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-05 09:24:39.478|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=21641, info=psutil.Process(pid=21641, name='sh', started='09:24:38')
|2020-10-05 09:24:43.691|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-2127265| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06610/E040_CPMV_6C_att00_06610_0001-100000.hdf
|2020-10-05 09:24:43.974|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E040_CPMV_6C_att00_06611
|2020-10-05 09:24:43.974|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-05 09:24:43.974|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06611/
|2020-10-05 09:24:43.993|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-05 09:24:44.005|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-05 09:24:44.069|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-05 09:24:44.125|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-05 09:24:44.234|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-2127265| - DM workflow kickoff done
|2020-10-05 09:24:44.235|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-2127265| - b'id=e6388c6c-82ce-4c50-a284-7a8117e11f5d owner=8idiuser status=pending startTime=1601907884.12 startTimestamp=2020/10/05 09:24:44 CDT \n'
|2020-10-05 09:24:44.362|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-05 09:24:44.363|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E040_CPMV_6C_att00_06611 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=21773, info=psutil.Process(pid=21773, name='sh', started='09:24:43')
|2020-10-05 09:24:44.929|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-05 09:24:44.929|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=21780, info=psutil.Process(pid=21780, name='sh', started='09:24:44')
|2020-10-05 09:24:49.079|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-2127357| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06611/E040_CPMV_6C_att00_06611_0001-100000.hdf
|2020-10-05 09:24:49.375|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E040_CPMV_6C_att00_06612
|2020-10-05 09:24:49.375|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-05 09:24:49.375|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06612/
|2020-10-05 09:24:49.396|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-05 09:24:49.407|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-05 09:24:49.473|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-05 09:24:49.519|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-2127357| - DM workflow kickoff done
|2020-10-05 09:24:49.519|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-2127357| - b'id=8b8d6ed4-4542-4dd7-9d5a-b6ff262cb612 owner=8idiuser status=pending startTime=1601907889.5 startTimestamp=2020/10/05 09:24:49 CDT \n'
|2020-10-05 09:24:49.562|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-05 09:24:49.814|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-05 09:24:49.815|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E040_CPMV_6C_att00_06612 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=21911, info=psutil.Process(pid=21911, name='sh', started='09:24:49')
|2020-10-05 09:24:50.402|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-05 09:24:50.402|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=21917, info=psutil.Process(pid=21917, name='sh', started='09:24:49')
|2020-10-05 09:24:54.685|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-2127449| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06612/E040_CPMV_6C_att00_06612_0001-100000.hdf
|2020-10-05 09:24:54.975|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E040_CPMV_6C_att00_06613
|2020-10-05 09:24:54.975|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-05 09:24:54.975|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06613/
|2020-10-05 09:24:54.999|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-05 09:24:55.014|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-05 09:24:55.089|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-05 09:24:55.092|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-2127449| - DM workflow kickoff done
|2020-10-05 09:24:55.092|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-2127449| - b'id=9db7f510-fe08-42ac-afbf-1f66a330d030 owner=8idiuser status=pending startTime=1601907895.08 startTimestamp=2020/10/05 09:24:55 CDT \n'
|2020-10-05 09:24:55.167|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-05 09:24:55.411|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-05 09:24:55.412|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E040_CPMV_6C_att00_06613 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=22048, info=psutil.Process(pid=22048, name='sh', started='09:24:54')
|2020-10-05 09:24:56.001|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-05 09:24:56.002|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=22055, info=psutil.Process(pid=22055, name='sh', started='09:24:55')
|2020-10-05 09:25:00.197|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-2127541| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06613/E040_CPMV_6C_att00_06613_0001-100000.hdf
|2020-10-05 09:25:00.493|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E040_CPMV_6C_att00_06614
|2020-10-05 09:25:00.494|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-05 09:25:00.494|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06614/
|2020-10-05 09:25:00.512|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-05 09:25:00.523|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-05 09:25:00.586|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-05 09:25:00.623|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-2127541| - DM workflow kickoff done
|2020-10-05 09:25:00.623|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-2127541| - b'id=036facfc-3c5d-427a-b789-a13e6403fdbe owner=8idiuser status=pending startTime=1601907900.6 startTimestamp=2020/10/05 09:25:00 CDT \n'
|2020-10-05 09:25:00.667|INFO|11782|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-05 09:25:00.922|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-05 09:25:00.922|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:E040_CPMV_6C_att00_06614 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=22197, info=psutil.Process(pid=22197, name='sh', started='09:25:00')
|2020-10-05 09:25:01.510|INFO|11782|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-05 09:25:01.510|INFO|11782|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=22203, info=psutil.Process(pid=22203, name='sh', started='09:25:00')
|2020-10-05 09:25:05.763|INFO|11782|bluesky-session|xpcs_acquire|308|Thread-2127633| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06614/E040_CPMV_6C_att00_06614_0001-100000.hdf
|2020-10-05 09:25:06.067|INFO|11782|bluesky-session|bp_Test_Bluesky|63|bluesky-run-engine| - E040_CPMV_6C_att00_06615
|2020-10-05 09:25:06.067|INFO|11782|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-05 09:25:06.067|INFO|11782|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/E040_CPMV_6C_att00_06615/
|2020-10-05 09:25:06.089|INFO|11782|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-05 09:25:06.099|INFO|11782|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-05 09:25:06.167|INFO|11782|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-05 09:25:14.336|INFO|11782|bluesky-session|xpcs_acquire|313|Thread-2127633| - DM workflow kickoff done
|2020-10-05 09:25:14.338|INFO|11782|bluesky-session|xpcs_acquire|314|Thread-2127633| - b'id=adf86c20-2ef9-4634-843c-815c59abca96 owner=8idiuser status=pending startTime=1601907906.16 startTimestamp=2020/10/05 09:25:06 CDT \n'

prjemian · 2020-10-05T15:37:20Z

Yikes! Monday operations? I've got a branch open instead of master.

prjemian · 2020-10-05T15:42:16Z

Confirmed this is the active issue on which to report.

qzhang234 · 2020-10-06T02:48:56Z

Scan aborted on 9:48 pm, 41 min after the initial jam. No update from the terminal since then.

prjemian · 2020-10-06T18:22:18Z

Instead of using a countdown timer (see #238), the countdown timer is one aspect of the ophyd.status.Status() object. Refactor to use the Status() object and wait() for it to complete.

prjemian · 2020-10-06T18:24:35Z

NOTE: ophyd.status.wait() raises StatusTimeoutError.

prjemian · 2020-10-06T18:35:55Z

code snippet:

from ophyd.status import Status
import time

def thing(attempts=5, timeout_s=1, wait_s=2):
    success = False
    for attempt in range(attempts):
        status = Status(timeout=timeout_s)
        print("starting to wait...")
        expired = time.time() + wait_s
        while not status.done and time.time() < expired:
            time.sleep(0.01)
        if not status.done:
            status.set_finished()   # successfully
            time.sleep(1e-6)    # brief pause for status internals
        print(status)
        if status.success:
            print("Timer finished.")
            success = True
            break
        else:
            print(attempt+1, "time expired")
    if not success:
        print(f"did not succeed after {attempts} attempts")

examples

In [8]: thing(3, timeout_s=1, wait_s=0.5)                                                                                          
starting to wait...
Status(obj=None, done=True, success=True)
Timer finished.

In [9]: thing(3, timeout_s=1, wait_s=1.5)                                                                                          
starting to wait...
Status(obj=None, done=False, success=False) has timed out
Status(obj=None, done=True, success=False)
1 time expired
starting to wait...
Status(obj=None, done=False, success=False) has timed out
Status(obj=None, done=True, success=False)
2 time expired
starting to wait...
Status(obj=None, done=False, success=False) has timed out
Status(obj=None, done=True, success=False)
3 time expired
did not succeed after 3 attempts

prjemian · 2020-10-06T18:36:39Z

In a plan, use yield from bps.sleep(0.01) instead of time.sleep(0.01)

Kacperswitalski · 2020-10-06T19:00:37Z

@prjemian H053 crashed at measurement 10396 at 1:43 pm on 10/6

Error from terminal


H053_10391_CPMV_6C_Stock_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/H053_10391_CPMV_6C_Stock_att00_Try1/H053_10391_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
H053_10392_CPMV_6C_Stock_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/H053_10392_CPMV_6C_Stock_att00_Try1/H053_10392_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
H053_10393_CPMV_6C_Stock_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/H053_10393_CPMV_6C_Stock_att00_Try1/H053_10393_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
H053_10394_CPMV_6C_Stock_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/H053_10394_CPMV_6C_Stock_att00_Try1/H053_10394_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
H053_10395_CPMV_6C_Stock_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/H053_10395_CPMV_6C_Stock_att00_Try1/H053_10395_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
H053_10396_CPMV_6C_Stock_att00_Try1
                                                                                                                                             
                                                                                                                                             
/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/epics/ca.py:1524: UserWarning: ca.get('OPS:message8') timed out a---------------------------------------------------------------------------
ReadTimeoutError                          Traceback (most recent call last)
/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
    115                 # acquire_result = (
--> 116                 yield from _one_acquire(
    117                     areadet, acq_name, ii, FLY_YES_NO, samx_cen, samz_cen)

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in _one_acquire(areadet, acq_name, ii, FLY_YES_NO, samx_cen, samz_cen)
     55 
---> 56     yield from AD_Acquire(
     57         areadet,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    318     logger.info("calling full_acquire_procedure()")
--> 319     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md)
    286         # yield from bp.count([areadet], md=md)
--> 287         yield from inner_count([areadet], md=_md)
    288         logger.debug("after count()")

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in inner_count(devices, md)
    237     def inner_count(devices, md={}):
--> 238         yield from bps.open_run(md=md)
    239         for obj in devices:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in open_run(md)
    822     """
--> 823     return (yield Msg('open_run', **(md or {})))
    824 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    140     '''
--> 141     return (yield msg)
    142 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    140     '''
--> 141     return (yield msg)
    142 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in single_gen(msg)
    140     '''
--> 141     return (yield msg)
    142 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in trigger_and_read(devices, name)
    910     from .preprocessors import rewindable_wrapper
--> 911     return (yield from rewindable_wrapper(inner_trigger_and_read(),
    912                                           rewindable))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in rewindable_wrapper(plan, rewindable)
    692     else:
--> 693         return (yield from plan)
    694 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in inner_trigger_and_read()
    904         for obj in devices:
--> 905             reading = (yield from read(obj))
    906             if reading is not None:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in read(obj)
     94     """
---> 95     return (yield Msg('read', obj))
     96 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1424                         # raised
-> 1425                         new_response = await coro(msg)
   1426 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _read(self, msg)
   1669         # actually _read_ the object
-> 1670         ret = obj.read(*msg.args, **msg.kwargs)
   1671 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/device.py in read(self)
   1236         for _, component in self._get_components_of_kind(Kind.normal):
-> 1237             res.update(component.read())
   1238         return res

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/device.py in read(self)
   1236         for _, component in self._get_components_of_kind(Kind.normal):
-> 1237             res.update(component.read())
   1238         return res

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/utils/epics_pvs.py in wrapper(self, *args, **kwargs)
    197         if self.connected:
--> 198             return fcn(self, *args, **kwargs)
    199         else:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in read(self)
    365         '''
--> 366         value = self.get()
    367         return {self.name: {'value': value,

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in get(self, as_string, timeout, connection_timeout, form, **kwargs)
   1094 
-> 1095         info = self._get_with_timeout(
   1096             self._read_pv, timeout, connection_timeout, as_string, form)

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/ophyd/signal.py in _get_with_timeout(self, pv, timeout, connection_timeout, as_string, form)
   1057         if info is None:
-> 1058             raise ReadTimeoutError(
   1059                 f"Failed to read {pv.pvname} "

ReadTimeoutError: Failed to read OPS:message8 within 2.0 sec

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
<ipython-input-18-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in multi_capil()
    178     z0 = 1.2
    179 
--> 180     yield from repeat_acq(
    181         sample_name='CPMV_6C_Stock',
    182         att_value=0,

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
    123                 logger.error(
    124                     "%s while collecting %s (attempt %d of %d): %s",
--> 125                     exc.__name__,
    126                     acq_name,
    127                     attempt+1,

AttributeError: 'ReadTimeoutError' object has no attribute '__name__'

Exception in thread Thread-983412:
Traceback (most recent call last):
  File "/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/utils/countdown.py", line 50, in countdown_timer
    raise CountdownExpired(
instrument.utils.countdown.CountdownExpired: waited 10.0s for process to complete

Logger


|2020-10-06 13:42:52.564|INFO|26943|bluesky-session|xpcs_acquire|308|Thread-982760| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/H053_10388_CPMV_6C_Stock_att00_Try1/H053_10388_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
|2020-10-06 13:42:52.673|INFO|26943|bluesky-session|bp_Test_Bluesky|37|bluesky-run-engine| - H053_10389_CPMV_6C_Stock_att00_Try1
|2020-10-06 13:42:52.827|INFO|26943|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-06 13:42:52.827|INFO|26943|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/H053_10389_CPMV_6C_Stock_att00_Try1/
|2020-10-06 13:42:52.873|INFO|26943|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-06 13:42:52.882|INFO|26943|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-06 13:42:52.933|INFO|26943|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-06 13:42:52.940|INFO|26943|bluesky-session|xpcs_acquire|313|Thread-982760| - DM workflow kickoff done
|2020-10-06 13:42:52.940|INFO|26943|bluesky-session|xpcs_acquire|314|Thread-982760| - b'id=28b0b21c-4f6d-4c35-a3b3-ce1b2d465539 owner=8idiuser status=pending startTime=1602009772.92 startTimestamp=2020/10/06 13:42:52 CDT \n'
|2020-10-06 13:42:52.995|INFO|26943|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-06 13:42:53.179|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:42:53.179|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:H053_10389_CPMV_6C_Stock_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19363, info=psutil.Process(pid=19363, name='sh', started='13:42:52')
|2020-10-06 13:42:53.581|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:42:53.581|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19370, info=psutil.Process(pid=19370, name='sh', started='13:42:52')
|2020-10-06 13:42:58.267|INFO|26943|bluesky-session|xpcs_acquire|308|Thread-982853| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/H053_10389_CPMV_6C_Stock_att00_Try1/H053_10389_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
|2020-10-06 13:42:58.368|INFO|26943|bluesky-session|bp_Test_Bluesky|37|bluesky-run-engine| - H053_10390_CPMV_6C_Stock_att00_Try1
|2020-10-06 13:42:58.527|INFO|26943|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-06 13:42:58.527|INFO|26943|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/H053_10390_CPMV_6C_Stock_att00_Try1/
|2020-10-06 13:42:58.575|INFO|26943|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-06 13:42:58.585|INFO|26943|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-06 13:42:58.625|INFO|26943|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-06 13:42:58.644|INFO|26943|bluesky-session|xpcs_acquire|313|Thread-982853| - DM workflow kickoff done
|2020-10-06 13:42:58.644|INFO|26943|bluesky-session|xpcs_acquire|314|Thread-982853| - b'id=66f02510-c1d3-435b-8e04-cad9328a302c owner=8idiuser status=pending startTime=1602009778.63 startTimestamp=2020/10/06 13:42:58 CDT \n'
|2020-10-06 13:42:58.717|INFO|26943|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-06 13:42:58.886|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:42:58.887|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:H053_10390_CPMV_6C_Stock_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19502, info=psutil.Process(pid=19502, name='sh', started='13:42:58')
|2020-10-06 13:42:59.300|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:42:59.300|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19509, info=psutil.Process(pid=19509, name='sh', started='13:42:58')
|2020-10-06 13:43:03.732|INFO|26943|bluesky-session|xpcs_acquire|308|Thread-982946| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/H053_10390_CPMV_6C_Stock_att00_Try1/H053_10390_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
|2020-10-06 13:43:03.813|INFO|26943|bluesky-session|bp_Test_Bluesky|37|bluesky-run-engine| - H053_10391_CPMV_6C_Stock_att00_Try1
|2020-10-06 13:43:03.997|INFO|26943|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-06 13:43:03.997|INFO|26943|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/H053_10391_CPMV_6C_Stock_att00_Try1/
|2020-10-06 13:43:04.028|INFO|26943|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-06 13:43:04.037|INFO|26943|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-06 13:43:04.101|INFO|26943|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-06 13:43:04.157|INFO|26943|bluesky-session|xpcs_acquire|313|Thread-982946| - DM workflow kickoff done
|2020-10-06 13:43:04.157|INFO|26943|bluesky-session|xpcs_acquire|314|Thread-982946| - b'id=e239f7cf-504f-40e0-b25e-3ce9ae66b6eb owner=8idiuser status=pending startTime=1602009784.13 startTimestamp=2020/10/06 13:43:04 CDT \n'
|2020-10-06 13:43:04.175|INFO|26943|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-06 13:43:04.346|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:43:04.347|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:H053_10391_CPMV_6C_Stock_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19642, info=psutil.Process(pid=19642, name='sh', started='13:43:03')
|2020-10-06 13:43:04.742|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:43:04.742|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19648, info=psutil.Process(pid=19648, name='sh', started='13:43:04')
|2020-10-06 13:43:09.338|INFO|26943|bluesky-session|xpcs_acquire|308|Thread-983039| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/H053_10391_CPMV_6C_Stock_att00_Try1/H053_10391_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
|2020-10-06 13:43:09.429|INFO|26943|bluesky-session|bp_Test_Bluesky|37|bluesky-run-engine| - H053_10392_CPMV_6C_Stock_att00_Try1
|2020-10-06 13:43:09.611|INFO|26943|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-06 13:43:09.611|INFO|26943|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/H053_10392_CPMV_6C_Stock_att00_Try1/
|2020-10-06 13:43:09.631|INFO|26943|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-06 13:43:09.671|INFO|26943|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-06 13:43:09.713|INFO|26943|bluesky-session|xpcs_acquire|313|Thread-983039| - DM workflow kickoff done
|2020-10-06 13:43:09.713|INFO|26943|bluesky-session|xpcs_acquire|314|Thread-983039| - b'id=e58130f8-c15a-4599-85f7-cbca3e9f8705 owner=8idiuser status=pending startTime=1602009789.7 startTimestamp=2020/10/06 13:43:09 CDT \n'
|2020-10-06 13:43:09.720|INFO|26943|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-06 13:43:09.781|INFO|26943|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-06 13:43:09.949|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:43:09.950|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:H053_10392_CPMV_6C_Stock_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19780, info=psutil.Process(pid=19780, name='sh', started='13:43:09')
|2020-10-06 13:43:10.343|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:43:10.344|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19787, info=psutil.Process(pid=19787, name='sh', started='13:43:09')
|2020-10-06 13:43:14.917|INFO|26943|bluesky-session|xpcs_acquire|308|Thread-983132| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/H053_10392_CPMV_6C_Stock_att00_Try1/H053_10392_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
|2020-10-06 13:43:14.999|INFO|26943|bluesky-session|bp_Test_Bluesky|37|bluesky-run-engine| - H053_10393_CPMV_6C_Stock_att00_Try1
|2020-10-06 13:43:15.179|INFO|26943|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-06 13:43:15.179|INFO|26943|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/H053_10393_CPMV_6C_Stock_att00_Try1/
|2020-10-06 13:43:15.204|INFO|26943|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-06 13:43:15.216|INFO|26943|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-06 13:43:15.273|INFO|26943|bluesky-session|xpcs_acquire|313|Thread-983132| - DM workflow kickoff done
|2020-10-06 13:43:15.273|INFO|26943|bluesky-session|xpcs_acquire|314|Thread-983132| - b'id=58cec8a8-74f6-4d77-b8f7-50b840853fe7 owner=8idiuser status=pending startTime=1602009795.26 startTimestamp=2020/10/06 13:43:15 CDT \n'
|2020-10-06 13:43:15.299|INFO|26943|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-06 13:43:15.366|INFO|26943|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-06 13:43:15.535|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:43:15.535|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:H053_10393_CPMV_6C_Stock_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19923, info=psutil.Process(pid=19923, name='sh', started='13:43:14')
|2020-10-06 13:43:16.161|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:43:16.161|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=19929, info=psutil.Process(pid=19929, name='sh', started='13:43:15')
|2020-10-06 13:43:20.416|INFO|26943|bluesky-session|xpcs_acquire|308|Thread-983225| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/H053_10393_CPMV_6C_Stock_att00_Try1/H053_10393_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
|2020-10-06 13:43:20.497|INFO|26943|bluesky-session|bp_Test_Bluesky|37|bluesky-run-engine| - H053_10394_CPMV_6C_Stock_att00_Try1
|2020-10-06 13:43:20.681|INFO|26943|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-06 13:43:20.681|INFO|26943|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/H053_10394_CPMV_6C_Stock_att00_Try1/
|2020-10-06 13:43:20.707|INFO|26943|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-06 13:43:20.718|INFO|26943|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-06 13:43:20.766|INFO|26943|bluesky-session|xpcs_acquire|313|Thread-983225| - DM workflow kickoff done
|2020-10-06 13:43:20.767|INFO|26943|bluesky-session|xpcs_acquire|314|Thread-983225| - b'id=0d574dc1-6e10-473b-9245-4a224876d360 owner=8idiuser status=pending startTime=1602009800.75 startTimestamp=2020/10/06 13:43:20 CDT \n'
|2020-10-06 13:43:20.793|INFO|26943|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-06 13:43:20.865|INFO|26943|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-06 13:43:21.037|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:43:21.037|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:H053_10394_CPMV_6C_Stock_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20062, info=psutil.Process(pid=20062, name='sh', started='13:43:20')
|2020-10-06 13:43:21.554|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:43:21.554|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20069, info=psutil.Process(pid=20069, name='sh', started='13:43:20')
|2020-10-06 13:43:26.016|INFO|26943|bluesky-session|xpcs_acquire|308|Thread-983318| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/H053_10394_CPMV_6C_Stock_att00_Try1/H053_10394_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
|2020-10-06 13:43:26.097|INFO|26943|bluesky-session|bp_Test_Bluesky|37|bluesky-run-engine| - H053_10395_CPMV_6C_Stock_att00_Try1
|2020-10-06 13:43:26.281|INFO|26943|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-06 13:43:26.281|INFO|26943|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/H053_10395_CPMV_6C_Stock_att00_Try1/
|2020-10-06 13:43:26.305|INFO|26943|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-06 13:43:26.315|INFO|26943|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-06 13:43:26.364|INFO|26943|bluesky-session|xpcs_acquire|313|Thread-983318| - DM workflow kickoff done
|2020-10-06 13:43:26.364|INFO|26943|bluesky-session|xpcs_acquire|314|Thread-983318| - b'id=3ebfe6b2-8546-4175-9123-6b28bef12a75 owner=8idiuser status=pending startTime=1602009806.35 startTimestamp=2020/10/06 13:43:26 CDT \n'
|2020-10-06 13:43:26.378|INFO|26943|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-06 13:43:26.466|INFO|26943|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-06 13:43:26.636|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:43:26.636|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:H053_10395_CPMV_6C_Stock_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20203, info=psutil.Process(pid=20203, name='sh', started='13:43:25')
|2020-10-06 13:43:27.156|INFO|26943|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 13:43:27.156|INFO|26943|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=20209, info=psutil.Process(pid=20209, name='sh', started='13:43:26')
|2020-10-06 13:43:31.667|INFO|26943|bluesky-session|xpcs_acquire|308|Thread-983411| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/H053_10395_CPMV_6C_Stock_att00_Try1/H053_10395_CPMV_6C_Stock_att00_Try1_0001-100000.hdf
|2020-10-06 13:43:31.773|INFO|26943|bluesky-session|bp_Test_Bluesky|37|bluesky-run-engine| - H053_10396_CPMV_6C_Stock_att00_Try1
|2020-10-06 13:43:32.054|INFO|26943|bluesky-session|xpcs_acquire|313|Thread-983411| - DM workflow kickoff done
|2020-10-06 13:43:32.054|INFO|26943|bluesky-session|xpcs_acquire|314|Thread-983411| - b'id=4705b5d2-136f-47a6-98a6-b0fa07a245f9 owner=8idiuser status=pending startTime=1602009812.04 startTimestamp=2020/10/06 13:43:32 CDT \n'
|2020-10-06 13:43:32.972|INFO|26943|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-06 13:43:32.973|INFO|26943|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/H053_10396_CPMV_6C_Stock_att00_Try1/
|2020-10-06 13:43:33.001|INFO|26943|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-06 13:43:33.016|INFO|26943|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-06 13:43:33.082|INFO|26943|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now

prjemian · 2020-10-06T19:05:48Z

This error: ReadTimeoutError: Failed to read OPS:message8 within 2.0 sec will be fixed (timeout will change to 10 s) by PR #238

This error: instrument.utils.countdown.CountdownExpired: waited 10.0s for process to complete will be fixed by the change from countdown timer to Status() object (see above).

This error: AttributeError: 'ReadTimeoutError' object has no attribute '__name__' will be changed with the new edit I just finished.

qzhang234 · 2020-10-07T01:12:34Z

@prjemian New Bluesky code jammed after the first measurement. Logger file shows ERROR

Logger file


|2020-10-06 17:07:47.345|INFO|16271|bluesky-session|shutters|16|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/shutters.py
|2020-10-06 17:07:47.354|INFO|16271|bluesky-session|slits|9|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/slits.py
|2020-10-06 17:07:47.537|INFO|16271|bluesky-session|soft_glue_fpga|15|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/soft_glue_fpga.py
|2020-10-06 17:07:47.609|INFO|16271|bluesky-session|tables|9|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/tables.py
|2020-10-06 17:07:47.678|INFO|16271|bluesky-session|lambda_750k|14|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/lambda_750k.py
|2020-10-06 17:07:47.705|INFO|16271|bluesky-session|rigaku_ufxc|14|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py
|2020-10-06 17:07:47.750|INFO|16271|bluesky-session|alignments|16|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/alignments.py
|2020-10-06 17:07:47.752|INFO|16271|bluesky-session|shutters|15|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/shutters.py
|2020-10-06 17:07:47.754|INFO|16271|bluesky-session|flyscan|11|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/flyscan.py
|2020-10-06 17:07:47.756|INFO|16271|bluesky-session|flux_calculations|14|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/flux_calculations.py
|2020-10-06 17:07:47.758|INFO|16271|bluesky-session|functions|9|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/functions.py
|2020-10-06 17:07:47.760|INFO|16271|bluesky-session|lineup_tweak|12|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/lineup_tweak.py
|2020-10-06 17:07:47.762|INFO|16271|bluesky-session|move_diodes|18|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/move_diodes.py
|2020-10-06 17:07:47.763|INFO|16271|bluesky-session|move_sample|19|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/move_sample.py
|2020-10-06 17:07:47.766|INFO|16271|bluesky-session|pv_registers|17|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/pv_registers.py
|2020-10-06 17:07:47.827|INFO|16271|bluesky-session|xpcs_acquire|11|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py
|2020-10-06 17:07:47.833|INFO|16271|bluesky-session|explorer|14|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/utils/explorer.py
|2020-10-06 17:07:47.833|INFO|16271|bluesky-session|collection|23|MainThread| - Disabling log output to the console now. Log output will still be recorded in log files. Re-enable with this console-only command:
    logger.addHandler(console_log_writer)
|2020-10-06 20:04:30.374|INFO|16271|bluesky-session|emails|13|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/utils/emails.py
|2020-10-06 20:04:30.377|DEBUG|16271|bluesky-session|pv_registers|86|bluesky-run-engine| - Backing up RIGAKU500K_NoGap detector Beam Params
|2020-10-06 20:04:30.448|DEBUG|16271|bluesky-session|pv_registers|104|bluesky-run-engine| - Detector RIGAKU500K_NoGap Beam Params are Backed up
================ ========== ===========
value            from       to         
================ ========== ===========
254.1            8idi:Reg11 8idi:Reg141
149.4            8idi:Reg12 8idi:Reg142
13.386           8idi:Reg13 8idi:Reg143
-0.2315          8idi:Reg14 8idi:Reg144
876.71777020394  8idi:Reg15 8idi:Reg145
148.45656772884  8idi:Reg16 8idi:Reg146
250.0            8idi:Reg17 8idi:Reg147
72.0             8idi:Reg18 8idi:Reg148
0.0              8idi:Reg19 8idi:Reg149
0.56746170980549 8idi:Reg20 8idi:Reg150
================ ========== ===========

|2020-10-06 20:04:30.460|INFO|16271|bluesky-session|pv_registers|208|bluesky-run-engine| - ******** detector number 46.0 ************************
|2020-10-06 20:04:30.502|DEBUG|16271|bluesky-session|pv_registers|133|bluesky-run-engine| - Detector RIGAKU500K_NoGap Beam Params are restored
================ =========== ==========
value            from        to        
================ =========== ==========
254.1            8idi:Reg141 8idi:Reg11
149.4            8idi:Reg142 8idi:Reg12
13.386           8idi:Reg143 8idi:Reg13
-0.2315          8idi:Reg144 8idi:Reg14
876.71777020394  8idi:Reg145 8idi:Reg15
148.45656772884  8idi:Reg146 8idi:Reg16
250.0            8idi:Reg147 8idi:Reg17
72.0             8idi:Reg148 8idi:Reg18
0.0              8idi:Reg149 8idi:Reg19
0.56746170980549 8idi:Reg150 8idi:Reg20
================ =========== ==========

|2020-10-06 20:04:31.507|INFO|16271|bluesky-session|pv_registers|213|bluesky-run-engine| - Moving RIGAKU to the direct beam position
|2020-10-06 20:04:43.426|INFO|16271|bluesky-session|bp_Test_Bluesky|36|bluesky-run-engine| - J054_00001_CPMV_6C_7v1_Water_att00_Try1
|2020-10-06 20:04:57.052|INFO|16271|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-06 20:04:57.053|INFO|16271|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/J054_00001_CPMV_6C_7v1_Water_att00_Try1/
|2020-10-06 20:04:57.079|INFO|16271|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-06 20:04:57.090|INFO|16271|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-06 20:04:57.222|INFO|16271|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-06 20:04:57.378|INFO|16271|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-06 20:04:57.412|INFO|16271|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 20:04:57.416|INFO|16271|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:J054_00001_CPMV_6C_7v1_Water_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=11528, info=psutil.Process(pid=11528, name='sh', started='20:04:56')
|2020-10-06 20:04:57.701|INFO|16271|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-06 20:04:57.701|INFO|16271|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=11534, info=psutil.Process(pid=11534, name='sh', started='20:04:57')
|2020-10-06 20:05:02.633|INFO|16271|bluesky-session|xpcs_acquire|308|Thread-697| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/J054_00001_CPMV_6C_7v1_Water_att00_Try1/J054_00001_CPMV_6C_7v1_Water_att00_Try1_0001-100000.hdf
|2020-10-06 20:05:02.645|ERROR|16271|bluesky-session|bp_Test_Bluesky|120|bluesky-run-engine| - timeout while collecting J054_00001_CPMV_6C_7v1_Water_att00_Try1 (attempt 1 of 5)
|2020-10-06 20:05:02.645|INFO|16271|bluesky-session|bp_Test_Bluesky|125|bluesky-run-engine| - waiting 300 seconds before retry or continue ...
|2020-10-06 20:05:03.501|INFO|16271|bluesky-session|xpcs_acquire|313|Thread-697| - DM workflow kickoff done
|2020-10-06 20:05:03.501|INFO|16271|bluesky-session|xpcs_acquire|314|Thread-697| - b'id=4fdeb260-fb67-4323-b509-bccc00f6ac77 owner=8idiuser status=pending startTime=1602032703.46 startTimestamp=2020/10/06 20:05:03 CDT \n'

prjemian · 2020-10-07T01:25:54Z

Anything on the console?

…

On Tue, Oct 6, 2020, 8:12 PM Qingteng Zhang ***@***.***> wrote: New Bluesky code jammed after the first measurement. Logger file shows ERROR Logger file |2020-10-06 17:07:47.345|INFO|16271|bluesky-session|shutters|16|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/shutters.py |2020-10-06 17:07:47.354|INFO|16271|bluesky-session|slits|9|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/slits.py |2020-10-06 17:07:47.537|INFO|16271|bluesky-session|soft_glue_fpga|15|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/soft_glue_fpga.py |2020-10-06 17:07:47.609|INFO|16271|bluesky-session|tables|9|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/tables.py |2020-10-06 17:07:47.678|INFO|16271|bluesky-session|lambda_750k|14|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/lambda_750k.py |2020-10-06 17:07:47.705|INFO|16271|bluesky-session|rigaku_ufxc|14|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/devices/rigaku_ufxc.py |2020-10-06 17:07:47.750|INFO|16271|bluesky-session|alignments|16|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/alignments.py |2020-10-06 17:07:47.752|INFO|16271|bluesky-session|shutters|15|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/shutters.py |2020-10-06 17:07:47.754|INFO|16271|bluesky-session|flyscan|11|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/flyscan.py |2020-10-06 17:07:47.756|INFO|16271|bluesky-session|flux_calculations|14|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/flux_calculations.py |2020-10-06 17:07:47.758|INFO|16271|bluesky-session|functions|9|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/functions.py |2020-10-06 17:07:47.760|INFO|16271|bluesky-session|lineup_tweak|12|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/lineup_tweak.py |2020-10-06 17:07:47.762|INFO|16271|bluesky-session|move_diodes|18|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/move_diodes.py |2020-10-06 17:07:47.763|INFO|16271|bluesky-session|move_sample|19|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/move_sample.py |2020-10-06 17:07:47.766|INFO|16271|bluesky-session|pv_registers|17|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/pv_registers.py |2020-10-06 17:07:47.827|INFO|16271|bluesky-session|xpcs_acquire|11|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py |2020-10-06 17:07:47.833|INFO|16271|bluesky-session|explorer|14|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/utils/explorer.py |2020-10-06 17:07:47.833|INFO|16271|bluesky-session|collection|23|MainThread| - Disabling log output to the console now. Log output will still be recorded in log files. Re-enable with this console-only command: logger.addHandler(console_log_writer) |2020-10-06 20:04:30.374|INFO|16271|bluesky-session|emails|13|MainThread| - /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/instrument/utils/emails.py |2020-10-06 20:04:30.377|DEBUG|16271|bluesky-session|pv_registers|86|bluesky-run-engine| - Backing up RIGAKU500K_NoGap detector Beam Params |2020-10-06 20:04:30.448|DEBUG|16271|bluesky-session|pv_registers|104|bluesky-run-engine| - Detector RIGAKU500K_NoGap Beam Params are Backed up ================ ========== =========== value from to ================ ========== =========== 254.1 8idi:Reg11 8idi:Reg141 149.4 8idi:Reg12 8idi:Reg142 13.386 8idi:Reg13 8idi:Reg143 -0.2315 8idi:Reg14 8idi:Reg144 876.71777020394 8idi:Reg15 8idi:Reg145 148.45656772884 8idi:Reg16 8idi:Reg146 250.0 8idi:Reg17 8idi:Reg147 72.0 8idi:Reg18 8idi:Reg148 0.0 8idi:Reg19 8idi:Reg149 0.56746170980549 8idi:Reg20 8idi:Reg150 ================ ========== =========== |2020-10-06 20:04:30.460|INFO|16271|bluesky-session|pv_registers|208|bluesky-run-engine| - ******** detector number 46.0 ************************ |2020-10-06 20:04:30.502|DEBUG|16271|bluesky-session|pv_registers|133|bluesky-run-engine| - Detector RIGAKU500K_NoGap Beam Params are restored ================ =========== ========== value from to ================ =========== ========== 254.1 8idi:Reg141 8idi:Reg11 149.4 8idi:Reg142 8idi:Reg12 13.386 8idi:Reg143 8idi:Reg13 -0.2315 8idi:Reg144 8idi:Reg14 876.71777020394 8idi:Reg145 8idi:Reg15 148.45656772884 8idi:Reg146 8idi:Reg16 250.0 8idi:Reg147 8idi:Reg17 72.0 8idi:Reg148 8idi:Reg18 0.0 8idi:Reg149 8idi:Reg19 0.56746170980549 8idi:Reg150 8idi:Reg20 ================ =========== ========== |2020-10-06 20:04:31.507|INFO|16271|bluesky-session|pv_registers|213|bluesky-run-engine| - Moving RIGAKU to the direct beam position |2020-10-06 20:04:43.426|INFO|16271|bluesky-session|bp_Test_Bluesky|36|bluesky-run-engine| - J054_00001_CPMV_6C_7v1_Water_att00_Try1 |2020-10-06 20:04:57.052|INFO|16271|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting |2020-10-06 20:04:57.053|INFO|16271|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/J054_00001_CPMV_6C_7v1_Water_att00_Try1/ |2020-10-06 20:04:57.079|INFO|16271|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure() |2020-10-06 20:04:57.090|INFO|16271|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'} |2020-10-06 20:04:57.222|INFO|16271|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now |2020-10-06 20:04:57.378|INFO|16271|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state. |2020-10-06 20:04:57.412|INFO|16271|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running |2020-10-06 20:04:57.416|INFO|16271|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:J054_00001_CPMV_6C_7v1_Water_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=11528, info=psutil.Process(pid=11528, name='sh', started='20:04:56') |2020-10-06 20:04:57.701|INFO|16271|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running |2020-10-06 20:04:57.701|INFO|16271|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=11534, info=psutil.Process(pid=11534, name='sh', started='20:04:57') |2020-10-06 20:05:02.633|INFO|16271|bluesky-session|xpcs_acquire|308|Thread-697| - DM workflow kickoff starting: analysis:True file:/home/8ididata/2020-3/qzhang202010/J054_00001_CPMV_6C_7v1_Water_att00_Try1/J054_00001_CPMV_6C_7v1_Water_att00_Try1_0001-100000.hdf |2020-10-06 20:05:02.645|ERROR|16271|bluesky-session|bp_Test_Bluesky|120|bluesky-run-engine| - timeout while collecting J054_00001_CPMV_6C_7v1_Water_att00_Try1 (attempt 1 of 5) |2020-10-06 20:05:02.645|INFO|16271|bluesky-session|bp_Test_Bluesky|125|bluesky-run-engine| - waiting 300 seconds before retry or continue ... |2020-10-06 20:05:03.501|INFO|16271|bluesky-session|xpcs_acquire|313|Thread-697| - DM workflow kickoff done |2020-10-06 20:05:03.501|INFO|16271|bluesky-session|xpcs_acquire|314|Thread-697| - b'id=4fdeb260-fb67-4323-b509-bccc00f6ac77 owner=8idiuser status=pending startTime=1602032703.46 startTimestamp=2020/10/06 20:05:03 CDT \n' — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#233 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARMUMFDNJZCFTN5P7ARGOTSJO6A5ANCNFSM4SCWH73Q> .

qzhang234 · 2020-10-07T01:53:43Z

@prjemian It stopped updating after the first measurement and Rigaku is idle.

Should I check out the older version or would you like to take a look at it?

Output from Console


In [11]: 

In [11]: _ = RE(multi_capil())
J054_00001_CPMV_6C_7v1_Water_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/J054_00001_CPMV_6C_7v1_Water_att00_Try1/J054_00001_CPMV_6C_7v1_Water_att00_Try1_0001-100000.hdf                                                                                                                                                   



^CA 'deferred pause' has been requested. The RunEngine will pause at the next checkpoint. To pause immediately, hit Ctrl+C again in the next 10 seconds.

qzhang234 · 2020-10-07T01:57:39Z

The sample is already loaded and I'm already home. Will be in standby all night. Please let me know how we should proceed

prjemian · 2020-10-07T04:14:12Z

Adjusted the code to watch the status object after acquire started, now working, started run with 20k measurements. Monitor the log files for Try2 (retries detected) or ERROR (retries failed).

prjemian · 2020-10-07T14:31:42Z

With recent changes, the plan logs timeouts (10 s now), waits the prescribed time (5 minutes now) and then continues.

(bluesky_2020_9) jemian@wow .../2020-3/.logs $ grep ERROR *.log*
ipython_logger.log.1:|2020-10-07 09:02:22.097|ERROR|16271|bluesky-session|bp_Test_Bluesky|122|bluesky-run-engine| - timeout while collecting J058_06364_CPMV_6C_7v1_Water_att00_Try1 (attempt 1 of 5)
ipython_logger.log.13:|2020-10-07 01:59:16.353|ERROR|16271|bluesky-session|bp_Test_Bluesky|122|bluesky-run-engine| - timeout while collecting J058_02122_CPMV_6C_7v1_Water_att00_Try1 (attempt 1 of 5)
ipython_logger.log.20:|2020-10-06 20:05:02.645|ERROR|16271|bluesky-session|bp_Test_Bluesky|120|bluesky-run-engine| - timeout while collecting J054_00001_CPMV_6C_7v1_Water_att00_Try1 (attempt 1 of 5)
ipython_logger.log.20:|2020-10-06 22:22:31.035|ERROR|16271|bluesky-session|bp_Test_Bluesky|121|bluesky-run-engine| - timeout while collecting J055_00001_CPMV_6C_7v1_Water_att00_Try1 (attempt 1 of 5)
ipython_logger.log.20:|2020-10-06 22:27:41.093|ERROR|16271|bluesky-session|bp_Test_Bluesky|121|bluesky-run-engine| - timeout while collecting J055_00002_CPMV_6C_7v1_Water_att00_Try1 (attempt 1 of 5)
ipython_logger.log.20:|2020-10-06 22:29:40.048|ERROR|16271|bluesky-session|bp_Test_Bluesky|123|bluesky-run-engine| - timeout while collecting J056_00001_CPMV_6C_7v1_Water_att00_Try1 (attempt 1 of 5)
ipython_logger.log.20:|2020-10-06 22:34:50.152|ERROR|16271|bluesky-session|bp_Test_Bluesky|123|bluesky-run-engine| - timeout while collecting J056_00002_CPMV_6C_7v1_Water_att00_Try1 (attempt 1 of 5)
ipython_logger.log.20:|2020-10-06 22:36:52.639|ERROR|16271|bluesky-session|bp_Test_Bluesky|122|bluesky-run-engine| - timeout while collecting J057_00001_CPMV_6C_7v1_Water_att00_Try1 (attempt 1 of 5)
ipython_logger.log.3:|2020-10-07 08:05:40.253|ERROR|16271|bluesky-session|bp_Test_Bluesky|122|bluesky-run-engine| - timeout while collecting J058_05837_CPMV_6C_7v1_Water_att00_Try1 (attempt 1 of 5)
ipython_logger.log.7:|2020-10-07 05:26:28.143|ERROR|16271|bluesky-session|bp_Test_Bluesky|122|bluesky-run-engine| - timeout while collecting J058_04243_CPMV_6C_7v1_Water_att00_Try1 (attempt 1 of 5)

Doesn't seem to be retrying but that is a lower priority than the priority to keep running.

We could lengthen the timeouts to 60s (1m). Perhaps encounter them even less frequently.

prjemian · 2020-10-07T21:49:19Z

FYI, the code should look a bit simpler and more like other bluesky code that subscribes to a signal with a related status object. The DeviceStatus is not so helpful here since it expects a stop() method that the Rigaku acquisition_complete signal does not have. The Status object is a simpler version that requires not such object.

The example code also uses a userCalc record to simulate the detector acquire and complete signals. The detector completion is manipulated manually in a different client. Here's the revised example:


from apstools.synApps import UserCalcsDevice
from bluesky import plan_stubs as bps
import datetime
from ophyd.status import DeviceStatus, Status, StatusTimeoutError
import pyRestTable

calcs = UserCalcsDevice("sky:", name="calcs")
calc = calcs.calc1
calc.wait_for_connection()
val = calc.calculated_value

def only_val(timeout=20, attempts=5):
    report_table = pyRestTable.Table()
    report_table.labels = "attempt success datetime".split()

    for attempt in range(attempts):
        status = Status(timeout=timeout)

        def watch_calc(value, old_value, **kwargs):
            if value != 0 and old_value == 0:
                status.set_finished()

        yield from bps.mv(
            calc.channels.A.input_value, 0,
            calc.process_record, 1,
        )
        val.subscribe(watch_calc)
        # print(f"before: {val.get()} {status}")
        try:
            status.wait()
            # print(f"after: {val.get()} {status}")
            break
        except StatusTimeoutError as exc:
            # print(f"attempt {attempt+1}, caught exception: {exc}")
            report_table.addRow((attempt+1, status.success, datetime.datetime.now()))
        finally:
            val.clear_sub(watch_calc)
            # print("finally")

    if not status.success:
        print(report_table)

prjemian · 2020-10-07T21:50:16Z

I put the new code in a file next to the existing bp_????.py file.

qzhang234 · 2020-10-08T04:36:47Z

Scan crashed at 14239/20000 due to what appears to be a timeout. Did not go to Try2.

Terminal


J058_14327_CPMV_6C_7v1_Water_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/J058_14327_CPMV_6C_7v1_Water_att00_Try1/J058_14327_CPMV_6C_7v1_Water_att00_Try1_0001-100000.hdf
J058_14328_CPMV_6C_7v1_Water_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/J058_14328_CPMV_6C_7v1_Water_att00_Try1/J058_14328_CPMV_6C_7v1_Water_att00_Try1_0001-100000.hdf
J058_14329_CPMV_6C_7v1_Water_att00_Try1
/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/epics/ca.py:1524: UserWarning: ca.get('8idi:Reg124') timed out after 10.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
/home/beams/8IDIUSER/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/epics/ca.py:1524: UserWarning: ca.get('8idi:Reg123') timed out after 10.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
---------------------------------------------------------------------------
FailedStatus                              Traceback (most recent call last)
<ipython-input-27-e00c4c348150> in <module>
----> 1 _ = RE(multi_capil())

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    805             self._task_fut.add_done_callback(set_blocking_event)
    806 
--> 807         self._resume_task(init_func=_build_task)
    808 
    809         if self._interrupted:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func)
    929                     if (exc is not None
    930                             and not isinstance(exc, _RunEnginePanic)):
--> 931                         raise exc
    932 
    933     def install_suspender(self, suspender):

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1498             exit_reason = str(err)
   1499             self.log.exception("Run aborted")
-> 1500             raise err
   1501         finally:
   1502             if not exit_reason:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/run_engine.py in _run(self)
   1363                     else:
   1364                         try:
-> 1365                             msg = self._plan_stack[-1].send(resp)
   1366                         # We have exhausted the top generator
   1367                         except StopIteration:

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in __call__(self, plan)
   1305         plan = monitor_during_wrapper(plan, self.monitors)
   1306         plan = baseline_wrapper(plan, self.baseline)
-> 1307         return (yield from plan)
   1308 
   1309 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in baseline_wrapper(plan, devices, name)
   1160         return (yield from plan)
   1161     else:
-> 1162         return (yield from plan_mutator(plan, insert_baseline))
   1163 
   1164 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in fly_during_wrapper(plan, flyers)
    859     plan1 = plan_mutator(plan, insert_after_open)
    860     plan2 = plan_mutator(plan1, insert_before_close)
--> 861     return (yield from plan2)
    862 
    863 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    168                     continue
    169                 else:
--> 170                     raise ex
    171         # if inserting / mutating, put new generator on the stack
    172         # and replace the current msg with the first element from the

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    121             ret = result_stack.pop()
    122             try:
--> 123                 msg = plan_stack[-1].send(ret)
    124             except StopIteration as e:
    125                 # discard the exhausted generator

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in multi_capil()
    172     # post_align()
    173 
--> 174     z0 = 0.6
    175 
    176     yield from repeat_acq(

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in repeat_acq(areadet, file_header, acq_rep, sample_name, Temp_list, att_value, FLY_YES_NO, samx_cen, samz_cen)
    107             # logger.info("acquire status object: %s", status)
    108             # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
--> 109             yield from _one_acquire(
    110                 areadet, acq_name, ii, FLY_YES_NO, samx_cen, samz_cen)
    111             # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

/home/beams10/8IDIUSER/bluesky_data/2020/2020-3/bp_Test_Bluesky.py in _one_acquire(areadet, acq_name, ii, FLY_YES_NO, samx_cen, samz_cen)
     53         yield from flyscan_spinner(8, 8+0.04*2.5, 0.1)
     54 
---> 55     yield from AD_Acquire(
     56         areadet,
     57         acquire_time=19.00e-6,

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in AD_Acquire(areadet, file_name, acquire_time, acquire_period, num_images, path, submit_xpcs_job, atten, md)
    317 
    318     logger.info("calling full_acquire_procedure()")
--> 319     return (yield from full_acquire_procedure(md=md))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in stage_wrapper(plan, devices)
    950         return (yield from plan)
    951 
--> 952     return (yield from finalize_wrapper(inner(), unstage_devices()))
    953 
    954 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in finalize_wrapper(plan, final_plan, pause_for_debug)
    507     cleanup = True
    508     try:
--> 509         ret = yield from plan
    510     except GeneratorExit:
    511         cleanup = False

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in inner()
    948     def inner():
    949         yield from stage_devices()
--> 950         return (yield from plan)
    951 
    952     return (yield from finalize_wrapper(inner(), unstage_devices()))

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/utils.py in dec_inner(*inner_args, **inner_kwargs)
   1112                 plan = gen_func(*inner_args, **inner_kwargs)
   1113                 plan = wrapper(plan, *args, **kwargs)
-> 1114                 return (yield from plan)
   1115             return dec_inner
   1116         return dec

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in monitor_during_wrapper(plan, signals)
    801     plan1 = plan_mutator(plan, insert_after_open)
    802     plan2 = plan_mutator(plan1, insert_before_close)
--> 803     return (yield from plan2)
    804 
    805 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
     76             # if we have a stashed exception, pass it along
     77             try:
---> 78                 msg = plan_stack[-1].throw(exception)
     79             except StopIteration as e:
     80                 # discard the exhausted generator

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in full_acquire_procedure(md)
    266     def full_acquire_procedure(md={}):
    267         logger.debug("before update_metadata_prescan()")
--> 268         yield from update_metadata_prescan()
    269         logger.debug("after update_metadata_prescan()")
    270 

~/.ipython-bluesky/profile_bluesky/startup/instrument/plans/xpcs_acquire.py in update_metadata_prescan()
    209 
    210         try:
--> 211             yield from bps.mv(
    212                 # Reg 123-127 in order
    213                 dm_pars.I0mon, I0Mon.get(),

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/plan_stubs.py in mv(group, *args, **kwargs)
    256         ret = yield Msg('set', obj, val, group=group, **kwargs)
    257         status_objects.append(ret)
--> 258     yield Msg('wait', None, group=group)
    259     return tuple(status_objects)
    260 

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

~/.conda/envs/bluesky_2020_9/lib/python3.8/site-packages/bluesky/preprocessors.py in plan_mutator(plan, msg_proc)
    193         try:
    194             # yield out the 'current message' and collect the return
--> 195             inner_ret = yield msg
    196         except GeneratorExit:
    197             # special case GeneratorExit.  We must clean up all of our plans

FailedStatus: Status(obj=EpicsSignal(read_pv='8idi:Reg123', name='dm_pars_I0mon', parent='dm_pars', value=0.0, timestamp=1602130368.1495004, auto_monitor=False, string=False, write_pv='8idi:Reg123', limits=False, put_complete=False), done=True, success=False)

In [28]:

Logger


|2020-10-07 23:12:28.641|INFO|16271|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-07 23:12:28.641|INFO|16271|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/J058_14326_CPMV_6C_7v1_Water_att00_Try1/
|2020-10-07 23:12:28.665|INFO|16271|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-07 23:12:28.675|INFO|16271|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-07 23:12:28.739|INFO|16271|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-07 23:12:28.757|INFO|16271|bluesky-session|xpcs_acquire|313|Thread-1334056| - DM workflow kickoff done
|2020-10-07 23:12:28.757|INFO|16271|bluesky-session|xpcs_acquire|314|Thread-1334056| - b'id=4013b1d4-4358-4e44-8b98-74a52d833962 owner=8idiuser status=pending startTime=1602130348.74 startTimestamp=2020/10/07 23:12:28 CDT \n'
|2020-10-07 23:12:28.828|INFO|16271|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-07 23:12:29.060|INFO|16271|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-07 23:12:29.060|INFO|16271|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:J058_14326_CPMV_6C_7v1_Water_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=15978, info=psutil.Process(pid=15978, name='sh', started='23:12:28')
|2020-10-07 23:12:29.639|INFO|16271|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-07 23:12:29.639|INFO|16271|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=15984, info=psutil.Process(pid=15984, name='sh', started='23:12:28')
|2020-10-07 23:12:35.445|INFO|16271|bluesky-session|xpcs_acquire|308|Thread-1334149| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/J058_14326_CPMV_6C_7v1_Water_att00_Try1/J058_14326_CPMV_6C_7v1_Water_att00_Try1_0001-100000.hdf
|2020-10-07 23:12:35.554|INFO|16271|bluesky-session|bp_Test_Bluesky|112|bluesky-run-engine| - acquire status object: Status(obj=None, done=False, success=False)
|2020-10-07 23:12:35.565|INFO|16271|bluesky-session|bp_Test_Bluesky|118|bluesky-run-engine| - acquire status object: Status(obj=None, done=True, success=True)
|2020-10-07 23:12:35.566|INFO|16271|bluesky-session|bp_Test_Bluesky|107|bluesky-run-engine| - acquire status object: Status(obj=None, done=False, success=False)
|2020-10-07 23:12:35.566|INFO|16271|bluesky-session|bp_Test_Bluesky|36|bluesky-run-engine| - J058_14327_CPMV_6C_7v1_Water_att00_Try1
|2020-10-07 23:12:35.741|INFO|16271|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-07 23:12:35.741|INFO|16271|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/J058_14327_CPMV_6C_7v1_Water_att00_Try1/
|2020-10-07 23:12:35.763|INFO|16271|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-07 23:12:35.774|INFO|16271|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-07 23:12:35.835|INFO|16271|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-07 23:12:35.853|INFO|16271|bluesky-session|xpcs_acquire|313|Thread-1334149| - DM workflow kickoff done
|2020-10-07 23:12:35.854|INFO|16271|bluesky-session|xpcs_acquire|314|Thread-1334149| - b'id=57238a0a-38e7-47d4-82de-cbc58da2e7a9 owner=8idiuser status=pending startTime=1602130355.83 startTimestamp=2020/10/07 23:12:35 CDT \n'
|2020-10-07 23:12:35.931|INFO|16271|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-07 23:12:36.184|INFO|16271|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-07 23:12:36.184|INFO|16271|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:J058_14327_CPMV_6C_7v1_Water_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=16131, info=psutil.Process(pid=16131, name='sh', started='23:12:35')
|2020-10-07 23:12:36.753|INFO|16271|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-07 23:12:36.753|INFO|16271|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=16138, info=psutil.Process(pid=16138, name='sh', started='23:12:35')
|2020-10-07 23:12:41.755|INFO|16271|bluesky-session|xpcs_acquire|308|Thread-1334242| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/J058_14327_CPMV_6C_7v1_Water_att00_Try1/J058_14327_CPMV_6C_7v1_Water_att00_Try1_0001-100000.hdf
|2020-10-07 23:12:41.864|INFO|16271|bluesky-session|bp_Test_Bluesky|112|bluesky-run-engine| - acquire status object: Status(obj=None, done=False, success=False)
|2020-10-07 23:12:41.875|INFO|16271|bluesky-session|bp_Test_Bluesky|118|bluesky-run-engine| - acquire status object: Status(obj=None, done=True, success=True)
|2020-10-07 23:12:41.875|INFO|16271|bluesky-session|bp_Test_Bluesky|107|bluesky-run-engine| - acquire status object: Status(obj=None, done=False, success=False)
|2020-10-07 23:12:41.876|INFO|16271|bluesky-session|bp_Test_Bluesky|36|bluesky-run-engine| - J058_14328_CPMV_6C_7v1_Water_att00_Try1
|2020-10-07 23:12:42.058|INFO|16271|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-07 23:12:42.059|INFO|16271|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/J058_14328_CPMV_6C_7v1_Water_att00_Try1/
|2020-10-07 23:12:42.078|INFO|16271|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-07 23:12:42.089|INFO|16271|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-07 23:12:42.154|INFO|16271|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-07 23:12:42.169|INFO|16271|bluesky-session|xpcs_acquire|313|Thread-1334242| - DM workflow kickoff done
|2020-10-07 23:12:42.169|INFO|16271|bluesky-session|xpcs_acquire|314|Thread-1334242| - b'id=1a08f776-d647-4837-b286-0d37e7baa872 owner=8idiuser status=pending startTime=1602130362.15 startTimestamp=2020/10/07 23:12:42 CDT \n'
|2020-10-07 23:12:42.229|INFO|16271|bluesky-session|rigaku_ufxc|129|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-07 23:12:42.470|INFO|16271|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-07 23:12:42.471|INFO|16271|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo FILE:F:J058_14328_CPMV_6C_7v1_Water_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=16279, info=psutil.Process(pid=16279, name='sh', started='23:12:41')
|2020-10-07 23:12:43.052|INFO|16271|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-07 23:12:43.052|INFO|16271|bluesky-session|rigaku_ufxc|77|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=16298, info=psutil.Process(pid=16298, name='sh', started='23:12:42')
|2020-10-07 23:12:47.755|INFO|16271|bluesky-session|xpcs_acquire|308|Thread-1334335| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/J058_14328_CPMV_6C_7v1_Water_att00_Try1/J058_14328_CPMV_6C_7v1_Water_att00_Try1_0001-100000.hdf
|2020-10-07 23:12:47.866|INFO|16271|bluesky-session|bp_Test_Bluesky|112|bluesky-run-engine| - acquire status object: Status(obj=None, done=False, success=False)
|2020-10-07 23:12:47.878|INFO|16271|bluesky-session|bp_Test_Bluesky|118|bluesky-run-engine| - acquire status object: Status(obj=None, done=True, success=True)
|2020-10-07 23:12:47.878|INFO|16271|bluesky-session|bp_Test_Bluesky|107|bluesky-run-engine| - acquire status object: Status(obj=None, done=False, success=False)
|2020-10-07 23:12:47.878|INFO|16271|bluesky-session|bp_Test_Bluesky|36|bluesky-run-engine| - J058_14329_CPMV_6C_7v1_Water_att00_Try1
|2020-10-07 23:12:48.058|INFO|16271|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-07 23:12:48.059|INFO|16271|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/J058_14329_CPMV_6C_7v1_Water_att00_Try1/
|2020-10-07 23:12:48.080|INFO|16271|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-07 23:12:48.091|INFO|16271|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-07 23:12:59.293|INFO|16271|bluesky-session|xpcs_acquire|313|Thread-1334335| - DM workflow kickoff done
|2020-10-07 23:12:59.294|INFO|16271|bluesky-session|xpcs_acquire|314|Thread-1334335| - b'id=a92d5695-bad8-4745-bb7b-6909a2da5cd7 owner=8idiuser status=pending startTime=1602130368.15 startTimestamp=2020/10/07 23:12:48 CDT \n'

prjemian · 2020-10-08T13:29:15Z

Try the bp_modified_multicap.py file to see if this problem remains.

qzhang234 · 2020-10-10T22:40:19Z

Rigaku interface hung again. The try-catch block did not restart the measurement for roughly 2 hours. Restarting the scan.

Terminal


M071_08049_CPMV_6C_7v1_NH4M_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/M071_08049_CPMV_6C_7v1_NH4M_att00_Try1/M071_08049_CPMV_6C_7v1_NH4M_att00_Try1_0001-100000.hdf
M071_08050_CPMV_6C_7v1_NH4M_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/M071_08050_CPMV_6C_7v1_NH4M_att00_Try1/M071_08050_CPMV_6C_7v1_NH4M_att00_Try1_0001-100000.hdf
M071_08051_CPMV_6C_7v1_NH4M_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/M071_08051_CPMV_6C_7v1_NH4M_att00_Try1/M071_08051_CPMV_6C_7v1_NH4M_att00_Try1_0001-100000.hdf
M071_08052_CPMV_6C_7v1_NH4M_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/M071_08052_CPMV_6C_7v1_NH4M_att00_Try1/M071_08052_CPMV_6C_7v1_NH4M_att00_Try1_0001-100000.hdf
M071_08053_CPMV_6C_7v1_NH4M_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/M071_08053_CPMV_6C_7v1_NH4M_att00_Try1/M071_08053_CPMV_6C_7v1_NH4M_att00_Try1_0001-100000.hdf
M071_08054_CPMV_6C_7v1_NH4M_att00_Try1
HDF5 workflow file name: /home/8ididata/2020-3/qzhang202010/M071_08054_CPMV_6C_7v1_NH4M_att00_Try1/M071_08054_CPMV_6C_7v1_NH4M_att00_Try1_0001-100000.hdf
M071_08055_CPMV_6C_7v1_NH4M_att00_Try1
rigaku [In progress. No progress bar available.]

Logger


|2020-10-10 15:23:09.143|INFO|1376|bluesky-session|bp_modified_multicap|36|bluesky-run-engine| - M071_08051_CPMV_6C_7v1_NH4M_att00_Try1
|2020-10-10 15:23:09.331|INFO|1376|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-10 15:23:09.331|INFO|1376|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/M071_08051_CPMV_6C_7v1_NH4M_att00_Try1/
|2020-10-10 15:23:09.352|INFO|1376|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-10 15:23:09.362|INFO|1376|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-10 15:23:09.399|INFO|1376|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-10 15:23:09.417|INFO|1376|bluesky-session|xpcs_acquire|313|Thread-750402| - DM workflow kickoff done
|2020-10-10 15:23:09.417|INFO|1376|bluesky-session|xpcs_acquire|314|Thread-750402| - b'id=de8c4bf6-3196-45ed-b85c-0bedc9117920 owner=8idiuser status=pending startTime=1602361389.4 startTimestamp=2020/10/10 15:23:09 CDT \n'
|2020-10-10 15:23:09.457|INFO|1376|bluesky-session|rigaku_ufxc|104|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-10 15:23:09.595|INFO|1376|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-10 15:23:09.596|INFO|1376|bluesky-session|rigaku_ufxc|61|bluesky-run-engine| - command: echo FILE:F:M071_08051_CPMV_6C_7v1_NH4M_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=4123, info=psutil.Process(pid=4123, name='sh', started='15:23:08')
|2020-10-10 15:23:10.158|INFO|1376|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-10 15:23:10.158|INFO|1376|bluesky-session|rigaku_ufxc|61|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=4132, info=psutil.Process(pid=4132, name='sh', started='15:23:09')
|2020-10-10 15:23:15.383|INFO|1376|bluesky-session|xpcs_acquire|308|Thread-750495| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/M071_08051_CPMV_6C_7v1_NH4M_att00_Try1/M071_08051_CPMV_6C_7v1_NH4M_att00_Try1_0001-100000.hdf
|2020-10-10 15:23:15.457|INFO|1376|bluesky-session|bp_modified_multicap|36|bluesky-run-engine| - M071_08052_CPMV_6C_7v1_NH4M_att00_Try1
|2020-10-10 15:23:15.638|INFO|1376|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-10 15:23:15.639|INFO|1376|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/M071_08052_CPMV_6C_7v1_NH4M_att00_Try1/
|2020-10-10 15:23:15.666|INFO|1376|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-10 15:23:15.675|INFO|1376|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-10 15:23:15.699|INFO|1376|bluesky-session|xpcs_acquire|313|Thread-750495| - DM workflow kickoff done
|2020-10-10 15:23:15.700|INFO|1376|bluesky-session|xpcs_acquire|314|Thread-750495| - b'id=c0f95bce-6f74-4765-9bc1-49532118da7f owner=8idiuser status=pending startTime=1602361395.68 startTimestamp=2020/10/10 15:23:15 CDT \n'
|2020-10-10 15:23:15.740|INFO|1376|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-10 15:23:15.849|INFO|1376|bluesky-session|rigaku_ufxc|104|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-10 15:23:16.003|INFO|1376|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-10 15:23:16.003|INFO|1376|bluesky-session|rigaku_ufxc|61|bluesky-run-engine| - command: echo FILE:F:M071_08052_CPMV_6C_7v1_NH4M_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=4391, info=psutil.Process(pid=4391, name='sh', started='15:23:15')
|2020-10-10 15:23:16.387|INFO|1376|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-10 15:23:16.387|INFO|1376|bluesky-session|rigaku_ufxc|61|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=4405, info=psutil.Process(pid=4405, name='sh', started='15:23:15')
|2020-10-10 15:23:21.892|INFO|1376|bluesky-session|xpcs_acquire|308|Thread-750588| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/M071_08052_CPMV_6C_7v1_NH4M_att00_Try1/M071_08052_CPMV_6C_7v1_NH4M_att00_Try1_0001-100000.hdf
|2020-10-10 15:23:21.963|INFO|1376|bluesky-session|bp_modified_multicap|36|bluesky-run-engine| - M071_08053_CPMV_6C_7v1_NH4M_att00_Try1
|2020-10-10 15:23:22.139|INFO|1376|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-10 15:23:22.139|INFO|1376|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/M071_08053_CPMV_6C_7v1_NH4M_att00_Try1/
|2020-10-10 15:23:22.163|INFO|1376|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-10 15:23:22.176|INFO|1376|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-10 15:23:22.210|INFO|1376|bluesky-session|xpcs_acquire|313|Thread-750588| - DM workflow kickoff done
|2020-10-10 15:23:22.210|INFO|1376|bluesky-session|xpcs_acquire|314|Thread-750588| - b'id=f4044b26-86f6-4e3a-a6d9-80c1d710eb09 owner=8idiuser status=pending startTime=1602361402.19 startTimestamp=2020/10/10 15:23:22 CDT \n'
|2020-10-10 15:23:22.238|INFO|1376|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-10 15:23:22.327|INFO|1376|bluesky-session|rigaku_ufxc|104|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-10 15:23:22.483|INFO|1376|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-10 15:23:22.483|INFO|1376|bluesky-session|rigaku_ufxc|61|bluesky-run-engine| - command: echo FILE:F:M071_08053_CPMV_6C_7v1_NH4M_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=4630, info=psutil.Process(pid=4630, name='sh', started='15:23:21')
|2020-10-10 15:23:22.896|INFO|1376|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-10 15:23:22.897|INFO|1376|bluesky-session|rigaku_ufxc|61|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=4639, info=psutil.Process(pid=4639, name='sh', started='15:23:22')
|2020-10-10 15:23:29.374|INFO|1376|bluesky-session|xpcs_acquire|308|Thread-750681| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/M071_08053_CPMV_6C_7v1_NH4M_att00_Try1/M071_08053_CPMV_6C_7v1_NH4M_att00_Try1_0001-100000.hdf
|2020-10-10 15:23:29.444|INFO|1376|bluesky-session|bp_modified_multicap|36|bluesky-run-engine| - M071_08054_CPMV_6C_7v1_NH4M_att00_Try1
|2020-10-10 15:23:29.632|INFO|1376|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-10 15:23:29.632|INFO|1376|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/M071_08054_CPMV_6C_7v1_NH4M_att00_Try1/
|2020-10-10 15:23:29.652|INFO|1376|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-10 15:23:29.664|INFO|1376|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-10 15:23:29.719|INFO|1376|bluesky-session|xpcs_acquire|313|Thread-750681| - DM workflow kickoff done
|2020-10-10 15:23:29.719|INFO|1376|bluesky-session|xpcs_acquire|314|Thread-750681| - b'id=1cf3760b-a0b8-4be1-9491-3b70aa50fb82 owner=8idiuser status=pending startTime=1602361409.7 startTimestamp=2020/10/10 15:23:29 CDT \n'
|2020-10-10 15:23:29.730|INFO|1376|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-10 15:23:29.794|INFO|1376|bluesky-session|rigaku_ufxc|104|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-10 15:23:29.936|INFO|1376|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-10 15:23:29.936|INFO|1376|bluesky-session|rigaku_ufxc|61|bluesky-run-engine| - command: echo FILE:F:M071_08054_CPMV_6C_7v1_NH4M_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=4817, info=psutil.Process(pid=4817, name='sh', started='15:23:29')
|2020-10-10 15:23:30.544|INFO|1376|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-10 15:23:30.544|INFO|1376|bluesky-session|rigaku_ufxc|61|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=4827, info=psutil.Process(pid=4827, name='sh', started='15:23:29')
|2020-10-10 15:23:35.939|INFO|1376|bluesky-session|xpcs_acquire|308|Thread-750774| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/qzhang202010/M071_08054_CPMV_6C_7v1_NH4M_att00_Try1/M071_08054_CPMV_6C_7v1_NH4M_att00_Try1_0001-100000.hdf
|2020-10-10 15:23:36.034|INFO|1376|bluesky-session|bp_modified_multicap|36|bluesky-run-engine| - M071_08055_CPMV_6C_7v1_NH4M_att00_Try1
|2020-10-10 15:23:36.190|INFO|1376|bluesky-session|xpcs_acquire|49|bluesky-run-engine| - AD_Acquire starting
|2020-10-10 15:23:36.190|INFO|1376|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/qzhang202010/M071_08055_CPMV_6C_7v1_NH4M_att00_Try1/
|2020-10-10 15:23:36.237|INFO|1376|bluesky-session|xpcs_acquire|318|bluesky-run-engine| - calling full_acquire_procedure()
|2020-10-10 15:23:36.247|INFO|1376|bluesky-session|xpcs_acquire|147|bluesky-run-engine| - detNum=46, det_pars={'ccdHardwareColSize': 512, 'ccdHardwareRowSize': 1024, 'ccdxsense': 1, 'ccdzsense': 1, 'harmonic': 1, 'dpix': 0.076, 'saturation': 3, 'adupphot': 1.0, 'efficiency': 1.0, 'gain': 1, 'blemish': 1, 'flatfield': 0, 'distortion': 0, 'parasitic': 0, 'lld': 0, 'manufacturer': 'RIGAKU500K_NoGap'}
|2020-10-10 15:23:36.272|INFO|1376|bluesky-session|xpcs_acquire|313|Thread-750774| - DM workflow kickoff done
|2020-10-10 15:23:36.273|INFO|1376|bluesky-session|xpcs_acquire|314|Thread-750774| - b'id=2ea37825-56a5-4dbc-975d-56c401b5f434 owner=8idiuser status=pending startTime=1602361416.25 startTimestamp=2020/10/10 15:23:36 CDT \n'
|2020-10-10 15:23:36.283|INFO|1376|bluesky-session|xpcs_acquire|282|bluesky-run-engine| - scaler should be autocounting now
|2020-10-10 15:23:36.345|INFO|1376|bluesky-session|rigaku_ufxc|104|bluesky-run-engine| - Shutter will be controlled by UFXC if shutter is left in the **showbeam** state.
|2020-10-10 15:23:36.485|INFO|1376|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-10 15:23:36.485|INFO|1376|bluesky-session|rigaku_ufxc|61|bluesky-run-engine| - command: echo FILE:F:M071_08055_CPMV_6C_7v1_NH4M_att00_Try1 | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=4991, info=psutil.Process(pid=4991, name='sh', started='15:23:35')
|2020-10-10 15:23:36.899|INFO|1376|bluesky-session|rigaku_ufxc|33|bluesky-run-engine| - process_info status: running
|2020-10-10 15:23:36.899|INFO|1376|bluesky-session|rigaku_ufxc|61|bluesky-run-engine| - command: echo EXPOSURE | nc rigaku1.xray.aps.anl.gov 10000 - process info: pid=4998, info=psutil.Process(pid=4998, name='sh', started='15:23:36')

qzhang234 · 2021-03-04T18:12:31Z

Part of this issue is related to the Rigaku LabView PC, which has since been switched to the dedicated Linux server. Closing the issue for now and will reopen if problem resurfaces.

prjemian self-assigned this Oct 3, 2020

prjemian added the bug Something isn't working label Oct 3, 2020

prjemian modified the milestones: Running Bluesky Continuously for a Week, 2020-10 operations Oct 3, 2020

qzhang234 self-assigned this Oct 3, 2020

qzhang234 mentioned this issue Oct 3, 2020

send emails #232

Closed

qzhang234 assigned Kacperswitalski Oct 5, 2020

qzhang234 mentioned this issue Oct 5, 2020

StrReg connection timeout for Bluesky #235

Closed

qzhang234 added the high priority label Oct 5, 2020

This was referenced Oct 5, 2020

Skipping jammed Rigaku measurement and restart run engine? #230

Closed

handle jammed acquires (retry & continue) #236

Merged

prjemian added a commit that referenced this issue Oct 5, 2020

ENH #232 #233 send email when retries exhausted

bc7f4e9

prjemian closed this as completed in #236 Oct 5, 2020

qzhang234 reopened this Oct 6, 2020

prjemian added a commit that referenced this issue Oct 6, 2020

MNT #233 Set default timeout on EpicsSignalBase

e8110fe

prjemian mentioned this issue Oct 6, 2020

Set default timeout on EpicsSignalBase #238

Merged

prjemian closed this as completed Oct 7, 2020

qzhang234 reopened this Oct 8, 2020

qzhang234 mentioned this issue Oct 11, 2020

Reconstruct Bluesky PV connections #245

Closed

prjemian mentioned this issue Dec 21, 2020

install new bluesky conda environment: 2021_1 #246

Closed

prjemian modified the milestones: 2020-10 operations, 2021-1 operations Dec 21, 2020

qzhang234 closed this as completed Mar 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Read timeout stopped measurements. #233

Read timeout stopped measurements. #233

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

qzhang234 commented Oct 3, 2020

Kacperswitalski commented Oct 4, 2020 •

edited

Loading

qzhang234 commented Oct 5, 2020 •

edited

Loading

qzhang234 commented Oct 5, 2020

prjemian commented Oct 5, 2020

Kacperswitalski commented Oct 5, 2020

prjemian commented Oct 5, 2020

prjemian commented Oct 5, 2020

qzhang234 commented Oct 6, 2020

prjemian commented Oct 6, 2020

prjemian commented Oct 6, 2020

prjemian commented Oct 6, 2020

prjemian commented Oct 6, 2020

Kacperswitalski commented Oct 6, 2020

prjemian commented Oct 6, 2020 •

edited

Loading

qzhang234 commented Oct 7, 2020 •

edited

Loading

prjemian commented Oct 7, 2020 via email

qzhang234 commented Oct 7, 2020 •

edited

Loading

qzhang234 commented Oct 7, 2020

prjemian commented Oct 7, 2020

prjemian commented Oct 7, 2020

prjemian commented Oct 7, 2020

prjemian commented Oct 7, 2020

qzhang234 commented Oct 8, 2020

prjemian commented Oct 8, 2020

qzhang234 commented Oct 10, 2020 •

edited

Loading

qzhang234 commented Mar 4, 2021

Read timeout stopped measurements. #233

Read timeout stopped measurements. #233

Comments

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

prjemian commented Oct 3, 2020

qzhang234 commented Oct 3, 2020

Kacperswitalski commented Oct 4, 2020 • edited Loading

qzhang234 commented Oct 5, 2020 • edited Loading

qzhang234 commented Oct 5, 2020

prjemian commented Oct 5, 2020

Kacperswitalski commented Oct 5, 2020

prjemian commented Oct 5, 2020

prjemian commented Oct 5, 2020

qzhang234 commented Oct 6, 2020

prjemian commented Oct 6, 2020

prjemian commented Oct 6, 2020

prjemian commented Oct 6, 2020

prjemian commented Oct 6, 2020

Kacperswitalski commented Oct 6, 2020

prjemian commented Oct 6, 2020 • edited Loading

qzhang234 commented Oct 7, 2020 • edited Loading

prjemian commented Oct 7, 2020 via email

qzhang234 commented Oct 7, 2020 • edited Loading

qzhang234 commented Oct 7, 2020

prjemian commented Oct 7, 2020

prjemian commented Oct 7, 2020

prjemian commented Oct 7, 2020

prjemian commented Oct 7, 2020

qzhang234 commented Oct 8, 2020

prjemian commented Oct 8, 2020

qzhang234 commented Oct 10, 2020 • edited Loading

qzhang234 commented Mar 4, 2021

Kacperswitalski commented Oct 4, 2020 •

edited

Loading

qzhang234 commented Oct 5, 2020 •

edited

Loading

prjemian commented Oct 6, 2020 •

edited

Loading

qzhang234 commented Oct 7, 2020 •

edited

Loading

qzhang234 commented Oct 7, 2020 •

edited

Loading

qzhang234 commented Oct 10, 2020 •

edited

Loading